From raw multimodal data to training-ready datasets. In one pipeline, at any scale.
The data engine built for AI.
pip install daft
From raw multimodal data to training-ready datasets. In one pipeline, at any scale.
Multimodal-native
Process video, images, audio, and sensor data alongside structured metadata in a single dataframe.
CPU and GPU in one pipeline
Run GPU inference/embeddings alongside CPU decode and filter in one pipeline. Daft handles the scheduling and batching, no glue code required.
Python dataframe API
Same operations you use in Pandas or Spark: filter, transform, aggregate, write. No new framework to learn.
Multimodal-native
Process video, images, audio, and sensor data alongside structured metadata in a single dataframe.
CPU and GPU in one pipeline
Run GPU inference/embeddings alongside CPU decode and filter in one pipeline. Daft handles the scheduling and batching, no glue code required.
Python dataframe API
Same operations you use in Pandas or Spark: filter, transform, aggregate, write. No new framework to learn.
Why Daft
- Open-source data engine (Apache 2.0) with more than 5k GitHub stars, in production at companies like Amazon and Essential AI.
- Bring your own models, storage, and infrastructure. Run pipelines exactly where and how you want.
- If you know Pandas or Spark, you know Daft. Load a dataset, filter, transform, and export to training format in a few lines.
Use Cases
AI Search
- This example shows how using LLMs and embedding models, Daft extracts metadata, generates vectors, and writes them to a vector database.
Use Cases
Data Enrichment
- Pipelines label items, normalize attributes, redact PII, and extract structured fields using schema validation and guardrails.
- Daft handles text, images, and other modalities in one place, producing consistent outputs for downstream systems like search, analytics, and product catalogs.
Use Cases
Multimodal AI ETL
- This example showcases Daft's AI-powered data pipeline capabilities, transforming multimodal data into actionable intelligence.
- The workflow leverages Daft's Rust-based file downloader (40x faster than the competition) to process financial PDFs, extract content from embedded images and documents, and deploy LLMs to identify specific data points like quarterly revenue.
- The end-to-end pipeline demonstrates how Daft bridges multimodal inputs and structured outputs, delivering analysis-ready data in modern table formats like Apache Iceberg—all within a single, cohesive framework.
Native model operators
Embeddings, LLM extraction, and structured outputs as first-class operations. Plug in models from OpenAI, Hugging Face, or your own.
Multimodal column types
Images, video, audio, text, and embeddings as native column types. Decode, transform, and filter them like any other column.
Local to production consistency
Define pipelines once. Run them on your laptop or scale across a cluster. Same code, no rewrites.
Managed UDF runtime
Automatic batching, retries, and error handling for model UDFs. Zero-copy execution powered by Apache Arrow.
Lower memory footprint
Run the same queries with 5x less memory than alternatives. Jobs that would OOM on Spark or Pandas just work.
Built in Rust for speed
Daft's core is written in Rust. Decode video, run transforms, and join multimodal data at TB scale without paying Python overhead.
“Daft was incredible at large volumes of abnormally shaped workloads - I pointed it at 16,000 small Parquet files in a self-hosted S3 service and it just worked! It's the data engine built for the cloud and AI workloads.”
Apache Hudi
Apache Iceberg™
Amazon Web Services
Azure
Delta Lake
Google Cloud
Unity Catalog
Ray
Pandas
PyTorch