Data Pipeline

Data Pipeline is a skill for designing and implementing robust data pipelines. It covers the full spectrum from simple Python ETL scripts to complex orchestrated workflows with Airflow, dbt, and Prefect. The skill emphasizes reliability patterns: idempotency, incremental loads, data quality checks, and proper error handling.

How It Works

1. Source analysis — Understands your data sources (APIs, databases, files, streams) and their characteristics
2. Architecture design — Selects between ETL, ELT, or streaming patterns based on volume and latency needs
3. Pipeline generation — Produces code for extraction, transformation, and loading with proper dependency ordering
4. Orchestration setup — Creates Airflow DAGs, dbt project structure, or Prefect flows with scheduling and retries
5. Quality checks — Adds data validation, row count assertions, and freshness monitoring

Best For

• Building data warehouses with proper dimensional modeling
• Migrating legacy ETL jobs to modern orchestration tools
• Creating reliable data feeds for analytics and ML pipelines
• Implementing CDC (Change Data Capture) for real-time sync

Design Principles

All generated pipelines follow these principles: idempotent operations (safe to re-run), incremental by default (process only new/changed data), fail loudly with clear error messages, and produce lineage metadata for debugging.