War of the Allocators
This post shares an allocator analysis conducted in late 2024 while migrating internal Presto clusters to Velox execution through Prestissimo. Although the decision described here was made at that time, the experiments and trade-offs remain relevant to Velox users choosing between jemalloc and Velox's mmap-based allocator.
Objective
We conducted this investigation to determine which allocator to use as we migrated Presto workloads to Velox through Prestissimo. We compared jemalloc with Velox's custom mmap-based allocator, focusing on performance, memory fragmentation, RSS control, and behavior under high concurrency. This evaluation considered the trade-offs between leveraging the maturity, integration, and support of jemalloc, versus investing significant engineering bandwidth in developing a robust custom allocator that could match the reliability and debugging capabilities of jemalloc.