Data pipelines that don't lose data: orchestration, idempotency, and backfill
Moving millions of documents through ingestion and indexing reliably is its own discipline. Orchestrators, queues, idempotent steps, and careful backfill are what separate a pipeline that heals from one that silently drops data.
Behind any document-heavy product is a pipeline that ingests, parses, embeds, and indexes at scale — and at millions of documents, the interesting failures aren't the code, they're the operations: retries, partial failures, and backfills. Getting this right is what makes the difference between a pipeline you trust and one you babysit.
Orchestrators: DAGs and durable workflows
An orchestrator schedules and tracks the steps. Airflow (DAG-based), Dagster (asset-based, data-aware), and Prefect are the batch options; Temporal is the choice for long-running, stateful workflows, because it makes execution durable — a workflow can run for days, survive process restarts, and resume exactly where it left off. For a pipeline whose steps take minutes to hours and must not restart from scratch on a crash, that durability is the feature.
Decouple ingestion from indexing with a queue
Don't let the thing that receives documents also index them synchronously — a slow index shouldn't back up ingestion. Put a queue or a log (Kafka) between them: ingestion writes to the log and returns; indexing consumers read at their own pace. This decoupling gives you backpressure (consumers pull when ready), independent scaling, and a buffer that absorbs spikes.
Idempotency, retries, and dead letters
In a distributed pipeline everything is retried, so every step must be idempotent — processing the same document twice must produce the same result, not a duplicate. Key on a stable document id and upsert. Wrap steps in bounded retries with backoff, and send messages that keep failing to a dead-letter queue rather than blocking the pipeline or looping forever — a DLQ turns a poison message into a triaging task instead of an outage.
Backfill vs incremental, and watermarking
You run two modes: a one-time backfill of the existing corpus, and an incremental sync of new and changed documents. Keep them from fighting — a backfill that overwrites fresher incremental data is a classic bug. Watermarking tracks how far the incremental stream has progressed (by event time), so you know what's been processed and can resume without gaps or double-work. And bake in data-quality checks and schema-evolution handling: at millions of documents, a small fraction are always malformed, and the pipeline needs to quarantine them, not choke.
A good pipeline isn't one that never fails — it's one that fails a single document into a dead-letter queue instead of taking the whole ingest down with it.