One post tagged with "delta-lake"

Apache Gluten uses Velox to execute Spark workloads natively, including writing Delta Lake tables with Velox's Parquet writer. However, writing the data natively addressed only one part of the write path. Delta Lake also collects per-file statistics, and this statistics path continued to process data one Spark row at a time.

To eliminate this overhead, Gluten now evaluates Delta statistics with a native Velox aggregation task. A Task Barrier marks the boundary between output files, allowing the same task to emit one statistics result, reset its aggregation state, and continue with the next file.