Operations
Running a pipeline
Section titled “Running a pipeline”Pipelines run on the platform’s flow engine. Firing one on a schedule or from an external event is a Jobs trigger rather than a Data Pipes feature — a nightly extract is a pipeline plus a scheduled trigger.
Streaming, not buffering
Section titled “Streaming, not buffering”Records stream between stages. A pipeline’s memory cost is a function of stage width — how much each stage holds at once — not of total volume passing through.
The practical consequence: a pipeline that works on a sample usually works on the full extract, unless a stage buffers. Aggregation is the stage that does, by nature, since it cannot emit until it has seen its group.
Failure modes
Section titled “Failure modes”A source is unavailable. The pipeline fails at the first stage, before any writes. This is the good case — no partial state.
A destination fails mid-run. Records already written stay written. Whether that is safe depends on your destination: an idempotent upsert keyed on a stable id re-runs cleanly, an append does not. Decide this when you design the pipeline, not when you are recovering one.
A record fails to transform. Depending on the stage, either the record is skipped or the pipeline stops. For bulk loads, skipping and reporting is usually right; for financial data it usually is not.
Memory grows through the run. Something is buffering that you did not expect — almost always an aggregation over a higher-cardinality group than you assumed.
Idempotency
Section titled “Idempotency”Design pipelines to be re-runnable. Bulk data movement fails partway often enough that “can I just run it again?” should have an obvious yes.
Keying writes on a stable id from the source, rather than generating one per run, is what makes that true. It costs nothing at design time and saves a manual reconciliation later.
What to watch
Section titled “What to watch”| Signal | Why it matters |
|---|---|
| Records in vs records out | Silent drops in a transform stage |
| Run duration trend | Growing source volume, or a degrading destination |
| Memory profile through the run | A stage buffering when you thought it streamed |
| Failure stage distribution | Which stage is fragile, not just that runs fail |
| Re-run frequency | Pipelines needing manual intervention are a design defect |
Records-in versus records-out is the one to instrument first. A transform that silently drops malformed rows produces a successful run and a quietly incomplete destination, and nothing else will tell you.