data-ingestion
Design and implement data ingestion pipelines for ML workflows. Covers batch ingestion from CSV, Parquet, JSON, Avro, databases (PostgreSQL, MySQL, BigQuery), and cloud storage (S3, GCS, Azure Blob), streaming ingestion with Kafka, Kinesis, Pulsar, and Flink, ETL/ELT pipeline design, data lake ingestion (Delta Lake, Iceberg, Hudi), schema evolution, data versioning with DVC and LakeFS, data catalog integration, incremental loading, partitioning, compression, error handling, retry logic, dead letter queues, idempotency, and exactly-once semantics. Use when building or debugging data ingestion pipelines, connecting ML systems to data sources, moving data into a lake or feature pipeline, setting up streaming ingestion, or versioning training data.
- Version
- 1.0
- License
- Apache-2.0
Pinned to revision 45cf0fa3c5e7, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/data-ingestion/SKILL.md
- skills/data-ingestion/references/REFERENCE.md
- skills/data-ingestion/scripts/ingest_batch.py
- skills/data-ingestion/scripts/ingest_streaming.py
Every link opens the file at its source, pinned to the revision this page describes.