2 posts tagged with "task"

Apache Gluten uses Velox to execute Spark workloads natively, including writing Delta Lake tables with Velox's Parquet writer. However, writing the data natively addressed only one part of the write path. Delta Lake also collects per-file statistics, and this statistics path continued to process data one Spark row at a time.

To eliminate this overhead, Gluten now evaluates Delta statistics with a native Velox aggregation task. A Task Barrier marks the boundary between output files, allowing the same task to emit one statistics result, reset its aggregation state, and continue with the next file.

TL;DR​

Velox Task Barriers provide a synchronization mechanism that not only enables efficient task reuse, important for workloads such as AI training data loading, but also delivers the strict sequencing and checkpointing semantics required for streaming workloads.

By injecting a barrier split, users guarantee that no subsequent data is processed until the entire DAG is flushed and the synchronization signal is unblocked. This capability serves two critical patterns:

1. Task Reuse: Eliminates the overhead of repeated task initialization and teardown by safely reconfiguring warm tasks for new queries. This is a recurring pattern in AI training data loading workloads.

2. Streaming Processing: Enables continuous data handling with consistent checkpoints, allowing stateful operators to maintain context across batches without service interruption.

See the Task Barrier Developer Guide for implementation details.