Data Pipes

Data Pipes builds ETL and ELT pipelines: read records from a source, transform them stage by stage, write them to a destination. Records stream between stages rather than materialising in full, so a pipeline’s memory cost is a function of its stage width, not of how much data passes through it.

Shape of a pipeline

  source ──▶ transform ──▶ transform ──▶ destination
   csv         compute        morph          db
   db          aggregate                     parquet
   rest                                      csv
   parquet                                   entity
   excel                                     stdout

Any task can be a source, a stage or a sink — csv reads and writes, db queries and executes. A pipeline is a declared sequence of them.

ETL or ELT

Both, and the choice is yours rather than the tool’s. Transform before the load when the destination is expensive to write or you want it clean on arrival; load first and transform in place when the destination is a warehouse better at the work than the pipeline is.

Where it fits

Use	When
Data Pipes	Moving and transforming data in bulk
Jobs	Firing a pipeline on a schedule or trigger
Workflows	Processes that suspend and wait for humans
AI Flow	Task DAGs including model calls, and agentic loops

Data Pipes moves data; Jobs decides when. A nightly extract is a pipeline with a scheduled trigger, not a Data Pipes feature.

Next

• Task Reference — the built-in task catalogue
• Operations — running pipelines and what to watch