What is a data pipeline?
View details →
Data Pipelines & Transforms
When someone says 'we have a data pipeline,' they mean: automated steps that take raw inputs and produce clean, structured outputs. No more copy-paste from Excel into a dashboard.
- Pipeline = extract (get data) → transform (clean, reshape, join) → load (into a usable form)
- Transforms are the rules: standardize units, normalize names, fill gaps, validate ranges
- Why separate transform from load? So you can rerun transforms without re-extracting
- dbt, SQL, Python—different tools, same idea: codified transformation logic
- The goal: raw data in, decision-ready data out. Repeatable. Auditable.
Real-world example
Supplier sends 10 Excel files with different formats
Each file has "recycled %" in a different column. One uses decimals (0.5), another percentages (50). Dates are DD/MM, MM-DD, and "Q1 2024."
- Extract: Pull files from email, Drive, or API. Get raw bytes into the system.
- Transform: Map "recycled %" from whichever column it's in. Normalize 0.5 and 50 to same format. Parse all date variants to ISO. Join with SKU reference table.
- Load: Write clean rows to a table (or datastore) that analytics and dashboards read from.
- Next week: New files arrive. Pipeline runs again. Same transforms. Same output shape. No manual cleanup.
- The pipeline is the contract: "Give us messy input; we give you clean output."
Pipelines turn "someone sent a spreadsheet" into "we have queryable, validated data."
Why transforms matter
View details →
When do you need a pipeline?
View details →
Common mistakes
View details →
Need a data pipeline for your analytics?