moheetsubudhi-isb/data-engineering-toolkit
Data-engineering skills for choosing where data should live, laying out distribution keys and partitions in MPP warehouses, and designing pipelines with data quality, reconciliation and governance built in.
Decide where data should live: a relational OLTP database, a columnar warehouse, a document store, a key-value cache, a graph database, a data lake or lakehouse, or a combination. Use whenever someone asks which database or storage to use for an application, feature or data product; SQL or NoSQL; Postgres vs MongoDB vs Cassandra vs Redis vs Neo4j vs ClickHouse vs BigQuery; data lake vs warehouse vs lakehouse or Delta Lake; OLTP vs OLAP; whether a system needs ACID transactions or can live with eventual consistency; how the CAP theorem applies; or how to design polyglot persistence for a platform with very different kinds of data. Also use to review a storage choice that is struggling with scale, latency or cost. Not for picking distribution or partition keys inside a chosen warehouse, and not for pipeline design or data quality rules.
Choose how a large table is spread across nodes and split into partitions in an MPP warehouse or distributed store, and fix the skew and data movement that make queries slow. Use whenever someone asks which distribution key, DISTKEY, DISTRIBUTED BY, hash distribution, sort key, partition key, shard key or clustering key to use; whether to hash, round-robin or replicate a table; why one node or partition is much bigger or slower than the rest; why joins shuffle or broadcast huge amounts of data; or how to partition a big fact table by date, region or tenant, in Redshift, Synapse, Greenplum, BigQuery, Snowflake, Databricks, Spark, Cassandra or similar. Also use when reviewing warehouse table DDL before it goes live. Not for choosing which kind of database to use, and not for pipeline or data quality design.
Design how data moves from source systems to its consumers, and how its quality is proven along the way. Use whenever someone asks whether to use ETL, ELT or ETLT; batch, micro-batch or streaming; copying data or using federation or virtualisation; how to lay out bronze, silver and gold (medallion) layers; what data quality checks, reconciliation, SLAs or freshness alerts a pipeline needs; how to write a data contract between producer and consumer teams; how to capture lineage or set up a data catalog; how to classify data as public, internal, confidential or restricted and handle personal data under GDPR or similar rules; or why a dashboard number disagrees with the source system. Also use for data mesh and data product ownership. Not for choosing a database, not for distribution or partition keys, and not for judging whether a dataset is fit to train a machine learning model.