4 posts tagged with "nimble"

Introduction​

Nimble can reduce wall-clock time when a stripe or batch contains enough independent work. It parallelizes encoding across complete streams and decoding across eligible sibling fields. The writer bounds the number of encode tasks using the total stream count, maximum parallelism, and the minimum-streams-per-task setting. Each encode task may process several streams during a flush, encoding one stream at a time. A decode task handles a contiguous group of eligible sibling fields under a row or FlatMap-as-struct reader. The nested schema defines parent-child dependencies: Nimble reads a row's null information before its child values, and array lengths before element values because they provide the element offsets.

Introduction​

On serving paths, indexed reads such as prefix scans often need a small row range from a much larger stripe. Fetching the full stripe creates row-level over-fetch: it wastes network bandwidth in NIC-bound workloads, while decoding it into Velox vectors, selecting rows, and serializing again incurs significant CPU, allocation, and data-copy costs.

Nimble's StreamSlicer leverages white-box knowledge of each stream's encoding structure. It maps the requested top-level row range to the corresponding range in each nested stream and slices encoded bytes directly. For a compressed chunk, it decompresses the chunk before applying encoding-level slicing.

This article explains how StreamSlicer maps nested row ranges and slices encoded streams without decoding and re-encoding the full batch.

TL;DR​

Nimble from Meta has been using OpenZL as the backend compressor for many years. Today, we are open-sourcing our integration, so external contributors (i.e. outside Meta) can use Nimble with OpenZL.

Introduction​

Analytical data lakes excel at full-table scans but struggle with point lookups. Key-value stores handle point lookups efficiently but cannot serve analytical queries. What if a single file format could serve both workloads?

Nimble's Cluster Index bridges this gap. It is a lightweight, hierarchical index structure embedded directly inside Nimble columnar files. It enables O(log n) point lookups and range scans on sorted data — without a separate index file, without an external service, and without sacrificing Nimble's columnar scan performance.

We have integrated the cluster index with Presto for analytical index joins and are actively integrating with ZippyDB for prefix key scans — both powered by the same underlying index structure, served through Velox.