Extending a data engine used to mean forking the source, vendoring internal types, and hoping the next release doesn't break your patches. Daft v0.7.5 changes that. This release ships a native extension framework via stable C ABI — the kind of plugin system that turned PostgreSQL and DuckDB from databases into ecosystems. Pair that with a live query dashboard and 2-5x faster Parquet reads on nested types, and v0.7.5 is the release where Daft starts becoming infrastructure you build on.

Native Extensions via Stable C ABI

Any language that can produce a C-compatible shared library can now extend Daft's query engine.

The extension framework, designed by @rchowell and modeled after PostgreSQL's extension system, introduces five new packages that define the extension boundary and establish a stable binary interface contract. Data crosses the extension boundary using the Arrow C Data Interface — zero-copy, no serialization overhead.

Here's what it looks like from the Python side:

1import daft
2import hello  # your pip-installable extension package
3
4daft.load_extension(hello)
5
6df = daft.from_pydict({"name": ["John", "Paul"]})
7df = df.select(hello.greet(df["name"]))

The release ships with two example extensions: hello (a tutorial) and dvector — a pgvector clone that brings vector operations to Daft as a native extension. That second example is the proof point: the same pattern that powers PostgreSQL's vector ecosystem now works in a distributed DataFrame engine.

This is 3,774 lines of new code. It's not a prototype.

Live Query Dashboard

If you've ever stared at a progress bar wondering which operator is actually slow, this release is for you. @samstokes shipped three PRs that overhaul the Daft dashboard into a real-time query debugger:

Interactive plan tree — The optimized and unoptimized query plans render as collapsible trees instead of raw JSON. Nodes are color-coded by operator type. Click any node to inspect its predicates, projections, sort keys, or join conditions.

Live operator stats — The physical plan tree shows rows in/out on every node, updated in real-time during execution. Border colors reflect operator status: orange while executing, green when finished, red on failure.

Wall-clock and CPU duration — Every operator card now shows live wall-clock duration that ticks while executing, with CPU time available in the expandable stats section. You can see exactly where your query spends its time without attaching a profiler.

Query result preview — A new Results tab in the dashboard lets you inspect query output directly, so you can verify results alongside the execution plan without switching back to your notebook or terminal (#6327).

Toggle between Tree View, Table, and JSON modes depending on how you prefer to debug. The old progress table is still there, but it's just not the only option anymore.

Parquet Reads: 2-5x Faster on Nested Types

@desmondcheongzx landed late materialization for the arrow-rs Parquet reader, and the benchmark numbers are worth reading twice.

Filter pushdown: Instead of decoding all columns and filtering post-read, Daft now decodes only predicate columns first, evaluates the filter, then decodes remaining columns only for matching rows. At 1% selectivity — the common case for point lookups — filtered reads are 2.1-2.6x faster than the old parquet2 reader.

Nested types: The arrow-rs reader is dramatically faster on structs, maps, and lists:

Page index is enabled for local reads, so a row-level filter can skip entire pages within column chunks based on page-level min/max statistics. Iceberg positional deletes are handled efficiently — deleted rows are skipped during decode rather than masked afterward.

The parallel stream path also landed: per-row-group decode dispatches as async tasks with parallel processing capped at the number of available CPU cores (num_cpus.min(num_row_groups)).

write_sql: The SQL Story Is Complete

You could read from SQL databases before. Now you can write back.

@huleilei implemented DataFrame.write_sql() — a distributed SQL data sink built on SQLAlchemy:

1df.write_sql(
2    table_name="results",
3    conn="postgresql://user:pass@host/db",
4    write_mode="overwrite",  # or "append" or "fail"
5    dtype={"embedding": Vector(384)},  # explicit SQLAlchemy types
6)

The implementation handles the hard parts: driver-side table initialization, worker-side parallel writes with independent connection lifecycles (no socket serialization issues), and an explicit dtype parameter for when you need precise control over the target schema. Supports PostgreSQL, SQLite, and anything else SQLAlchemy can talk to.

The End of parquet2

This is the release where Daft's Parquet reader fully transitions to arrow-rs.

Six refactoring PRs from @desmondcheongzx and @universalmind303 removed the parquet2 read pipeline entirely and made arrow-rs the default reader. The CSV reader, JSON reader, and function serialization layer also migrated. The DaftParquetMetadata adapter now decouples downstream crates from parquet2's types.

What this means in practice: Daft's I/O layer now speaks the same Arrow dialect as the rest of the Rust data ecosystem. Compatibility with arrow-rs, DataFusion, and other Apache Arrow projects gets simpler with every PR in this migration.

Since this is a complete port of a core subsystem, we want to hear from you. If you encounter any breaking behavior changes, performance regressions, or — just as valuable — unexpected speedups on your workloads, please open an issue on the Daft GitHub repository. Real-world feedback directly shapes what gets prioritized next.

Everything Else

read_text API — A new daft.read_text() function for reading text files into DataFrames, complementing the existing read_csv, read_json, and read_parquet APIs (#6111).

Parallelized image operations — Image decoding and transformation operations now use rayon for parallel execution, improving throughput on multi-core machines (#6333).

11 dependencies removed — @universalmind303 inlined small utility crates and removed 11 transitive dependencies, shrinking the build graph and reducing compile times (#6364).

Numeric range expansion in globs — Glob patterns now support numeric ranges (e.g., file_{1..10}.parquet), making it easier to target specific partitions without listing every file (#6127).

max_concurrency fix for async UDFs — The max_concurrency parameter on async UDFs now correctly limits parallelism instead of being silently ignored (#6302).

Zero-copy URL upload — @universalmind303 eliminated per-row memcpy in url_upload by creating bytes::Bytes slices that share ownership of the underlying Arrow buffer. Previously every row was fully copied into a new Vec<u8> before upload.

Lazy Lance import— import daft no longer eagerly loads the Lance library, reducing startup time. Lance users need to trigger the import explicitly.

Human-readable query names — @samstokes replaced UUID-based query identifiers with human-readable names, making logs and the dashboard easier to follow.

write_sql query names — @plotor added Python function data source info to EXPLAIN output, so you can see exactly which Python function generated your data when debugging query plans.

Breaking: stddev defaults to ddof=1 — @aaron-ang aligned Daft's stddev with pandas and numpy conventions. Sample standard deviation is now the default. Pass ddof=0 explicitly if you need population standard deviation.

Community Contributions

This release wouldn't be possible without contributions from the community:

• •

  @singularityDLW — Numeric range expansion in globs, Azure Blob connection pool fix, duplicate CTE rejection in SQL

• •

  @huleilei — write_sql distributed SQL sink, Lance distributed scalar index support

• •

  @plotor — write_sql query names in EXPLAIN output

• •

  @aaron-ang — stddev ddof=1 alignment

• •

  @yewleb — workspace lints inheritance

Upgrade

1uv add "daft>=0.7.5"

Or try the latest nightly:

uv pip install daft --pre --extra-index-url https://nightly.daft.ai

Check the full changelog for the complete list of changes.