5 posts tagged with "nimble"

Introduction​

Machine learning training data often stores features in wide maps. A single map column can hold thousands of feature IDs per row, while each model reads only a sparse subset of them. Different models read different subsets, and those subsets overlap. A serving layer that streams this data to trainers has to project the requested features out of every batch it returns.

Nimble's FlatMap layout, called Full FlatMap in this post, makes that projection precise by giving every key its own streams, but thousands of small streams add encoding, framing, and schema overhead that can be a large share of a small batch. Hybrid FlatMap partitions a map's keys into disjoint groups of features that are usually read together. All keys in a group share one set of streams, and the server projects each group on its own by passing its encoded streams through unchanged.

This article explains the Hybrid FlatMap layout and its two-stage projection, then compares it with Full FlatMap and regular maps in the Presto Page format.

Introduction​

Nimble can reduce wall-clock time when a stripe or batch contains enough independent work. It parallelizes encoding across complete streams and decoding across eligible sibling fields. The writer bounds the number of encode tasks using the total stream count, maximum parallelism, and the minimum-streams-per-task setting. Each encode task may process several streams during a flush, encoding one stream at a time. A decode task handles a contiguous group of eligible sibling fields under a row or FlatMap-as-struct reader. The nested schema defines parent-child dependencies: Nimble reads a row's null information before its child values, and array lengths before element values because they provide the element offsets.

Introduction​

On serving paths, indexed reads such as prefix scans often need a small row range from a much larger stripe. Fetching the full stripe creates row-level over-fetch: it wastes network bandwidth in NIC-bound workloads, while decoding it into Velox vectors, selecting rows, and serializing again incurs significant CPU, allocation, and data-copy costs.

Nimble's StreamSlicer leverages white-box knowledge of each stream's encoding structure. It maps the requested top-level row range to the corresponding range in each nested stream and slices encoded bytes directly. For a compressed chunk, it decompresses the chunk before applying encoding-level slicing.

This article explains how StreamSlicer maps nested row ranges and slices encoded streams without decoding and re-encoding the full batch.

TL;DR​

Nimble from Meta has been using OpenZL as the backend compressor for many years. Today, we are open-sourcing our integration, so external contributors (i.e. outside Meta) can use Nimble with OpenZL.

Introduction​

Analytical data lakes excel at full-table scans but struggle with point lookups. Key-value stores handle point lookups efficiently but cannot serve analytical queries. What if a single file format could serve both workloads?

Nimble's Cluster Index bridges this gap. It is a lightweight, hierarchical index structure embedded directly inside Nimble columnar files. It enables O(log n) point lookups and range scans on sorted data — without a separate index file, without an external service, and without sacrificing Nimble's columnar scan performance.

We have integrated the cluster index with Presto for analytical index joins and are actively integrating with ZippyDB for prefix key scans — both powered by the same underlying index structure, served through Velox.