Daft turns raw multimodal data into vectors, labels, and structured outputs. No infrastructure management required.
AI Pipelines Made Simple.
Daft turns raw multimodal data into vectors, labels, and structured outputs. No infrastructure management required.
One Engine for any modality
Power AI data pipelines in a single framework that combines ingestion, chunking, embeddings, LLM extraction, and multimodal transforms, with consistent behavior from local development to production.
Model-first by design
Offer first-class operators for embeddings and structured outputs, enabling reliable model-on-data pipelines across millions of processed rows without relying on stitched-together ETL and LLM tools.
Minimal operations
Reduce operational overhead with built-in scaling, orchestration, logging, model execution control, and more, without managing infrastructure or glue code.
One Engine for any modality
Power AI data pipelines in a single framework that combines ingestion, chunking, embeddings, LLM extraction, and multimodal transforms, with consistent behavior from local development to production.
Model-first by design
Offer first-class operators for embeddings and structured outputs, enabling reliable model-on-data pipelines across millions of processed rows without relying on stitched-together ETL and LLM tools.
Minimal operations
Reduce operational overhead with built-in scaling, orchestration, logging, model execution control, and more, without managing infrastructure or glue code.
Unlock the full power of Daft with Daft Cloud
Why Daft OSS
- Open source data engine with more than 5k GitHub stars for defining and running LLM-powered data pipelines.
- Bring your own models, storage, and infrastructure to run pipelines exactly where and how you want.
- Runs on your laptop for fast, iterative development, and in your own cluster when you need more scale.
Why Daft Cloud
- Serverless platform for experimenting, deploying, and operating AI pipelines with zero infrastructure headaches.
- Managed LLMs/VLMs and compute workers that scale automatically as your data grows.
- Production-grade auth, observability, retries, and versioning for minimal overhead.
Use Cases
Large Scale Document Processing
- This example demonstrates Daft's ability to seamlessly integrate with AI models to create text embeddings at scale.
- Using a pre-trained transformer model, Daft processes large document collections stored in cloud storage or Hugging Face, converting text into embeddings using easy-to-use user-defined functions for downstream AI applications like semantic search or clustering.
[1]
Native model operators
Run embeddings, LLM extraction, multimodal transforms, and structured outputs with first-class operators designed for model-driven pipelines.
[2]
Continuous freshness
Process only new or changed data and keep vectors, labels, and structured fields continuously fresh without unnecessary recompute.
[3]
Local to production consistency
Define pipelines once and run them the same way everywhere, from OSS development to production scale in the Cloud.
[4]
Managed model runtime
Get automatic batching, validation, retries, and scaling so pipelines run reliably without maintaining infrastructure or orchestration.
[5]
Integrated orchestration
Run pipelines continuously, on a schedule, or on events with built-in scheduling and lifecycle management.
[6]
Built-in observability
View logs, runs, and performance insights in one place so you can understand and debug pipelines with confidence.
Tony Wang
Data @ Anthropic, PhD @ Stanford
Patrick Ames
Principal Engineer @ Amazon
Ritvik Kapila
ML Research @ Essential AI
Maurice Weber
PhD AI Researcher @ Together AI
Alexander Filipchik
Head Of Infrastructure at City Storage Systems (CloudKitchens)
Apache Hudi
Apache Iceberg™
Amazon Web Services
Azure
Delta Lake
Google Cloud
Unity Catalog
Ray
Pandas
PyTorch
