3 posts tagged with "nimble"

Introduction​

On serving paths, indexed reads such as prefix scans often need a small row range from a much larger stripe. Fetching the full stripe creates row-level over-fetch: it wastes network bandwidth in NIC-bound workloads, while decoding it into Velox vectors, selecting rows, and serializing again incurs significant CPU, allocation, and data-copy costs.

Nimble's StreamSlicer leverages white-box knowledge of each stream's encoding structure. It maps the requested top-level row range to the corresponding range in each nested stream and slices encoded bytes directly. For a compressed chunk, it decompresses the chunk before applying encoding-level slicing.

This article explains how StreamSlicer maps nested row ranges and slices encoded streams without decoding and re-encoding the full batch.

TL;DR​

Nimble from Meta has been using OpenZL as the backend compressor for many years. Today, we are open-sourcing our integration, so external contributors (i.e. outside Meta) can use Nimble with OpenZL.

Introduction​

Analytical data lakes excel at full-table scans but struggle with point lookups. Key-value stores handle point lookups efficiently but cannot serve analytical queries. What if a single file format could serve both workloads?

Nimble's Cluster Index bridges this gap. It is a lightweight, hierarchical index structure embedded directly inside Nimble columnar files. It enables O(log n) point lookups and range scans on sorted data — without a separate index file, without an external service, and without sacrificing Nimble's columnar scan performance.

We have integrated the cluster index with Presto for analytical index joins and are actively integrating with ZippyDB for prefix key scans — both powered by the same underlying index structure, served through Velox.