data-engineering-wiki
Retrieve data engineering wiki concepts. Use when the question is about data engineering: Pipeline design; Loading dimensional and analytical structures; Data quality engineering; Orchestration; Analytical platform architecture; Data lifecycle engineering. Scan concepts.json in this skill folder. Read only matching notes. Follow basename links hop by hop. Do not load every note. Do not answer from fleeting/ notes.
Pinned to revision 846c7b7891a5, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/data-engineering-wiki/SKILL.md
- skills/data-engineering-wiki/accumulating-snapshot-load-mechanics.md
- skills/data-engineering-wiki/aggregate-table-load-consistency.md
- skills/data-engineering-wiki/aggregated-fact-as-dimension-attribute.md
- skills/data-engineering-wiki/allowed-lateness-horizon.md
- skills/data-engineering-wiki/audit-dimension.md
- skills/data-engineering-wiki/audit-statistics-tie-back.md
- skills/data-engineering-wiki/backflow-of-durable-keys.md
- skills/data-engineering-wiki/batch-streaming-unification-architectures.md
- skills/data-engineering-wiki/batch-vs-streaming-ingestion.md
- skills/data-engineering-wiki/behavior-study-group-table.md
- skills/data-engineering-wiki/behavioral-dimension-load-mechanics.md
- skills/data-engineering-wiki/bounded-vs-unbounded-data.md
- skills/data-engineering-wiki/bridge-table-incremental-maintenance.md
- skills/data-engineering-wiki/centralized-cleansing-logic.md
- skills/data-engineering-wiki/centralized-vs-decentralized-orchestration.md
- skills/data-engineering-wiki/change-data-capture.md
- skills/data-engineering-wiki/change-stream-consumption-pattern.md
- skills/data-engineering-wiki/cleansing-scope-consistency-test.md
- skills/data-engineering-wiki/common-data-model-for-multi-source-integration.md
- skills/data-engineering-wiki/compliance-driven-bitemporal-fact-versioning.md
- skills/data-engineering-wiki/compute-storage-separation.md
- skills/data-engineering-wiki/concepts.json
- skills/data-engineering-wiki/conformed-dimension-publish-subscribe.md
- skills/data-engineering-wiki/copy-on-write-load-cost.md
- skills/data-engineering-wiki/custom-streaming-windowing.md
- skills/data-engineering-wiki/data-access-policy-enforcement.md
- skills/data-engineering-wiki/data-catalog.md
- skills/data-engineering-wiki/data-contract.md
- skills/data-engineering-wiki/data-engineering-lifecycle.md
- skills/data-engineering-wiki/data-fabric-architecture.md
- skills/data-engineering-wiki/data-federation-and-virtualization.md
- skills/data-engineering-wiki/data-governance-program.md
- skills/data-engineering-wiki/data-hub-integration-pattern.md
- skills/data-engineering-wiki/data-lake-architecture.md
- skills/data-engineering-wiki/data-lake-zone-layering.md
- skills/data-engineering-wiki/data-lakehouse.md
- skills/data-engineering-wiki/data-lineage.md
- skills/data-engineering-wiki/data-mesh-domain-data-types.md
- skills/data-engineering-wiki/data-mesh-topologies.md
- skills/data-engineering-wiki/data-mesh.md
- skills/data-engineering-wiki/data-minimization-at-ingestion.md
- skills/data-engineering-wiki/data-profiling-before-ingestion.md
- skills/data-engineering-wiki/data-quality-dimensions.md
- skills/data-engineering-wiki/data-quality-validation-tests.md
- skills/data-engineering-wiki/data-retention-and-lifecycle-management.md
- skills/data-engineering-wiki/data-temperature-tiering.md
- skills/data-engineering-wiki/data-vault-load-mechanics.md
- skills/data-engineering-wiki/data-warehouse-architecture.md
- skills/data-engineering-wiki/dataflow-model-four-questions.md
- skills/data-engineering-wiki/dataops-pillars.md
- skills/data-engineering-wiki/dataset-versioning-and-zero-copy-cloning.md
- skills/data-engineering-wiki/dead-letter-queue.md
- skills/data-engineering-wiki/declared-vs-enforced-constraints.md
- skills/data-engineering-wiki/deduplication-and-survivorship.md
- skills/data-engineering-wiki/default-value-masks-missing-data.md
- skills/data-engineering-wiki/delivery-guarantees-exactly-once-vs-at-least-once.md
- skills/data-engineering-wiki/derived-data-vs-distributed-transactions.md
- skills/data-engineering-wiki/distributed-scheduler-reliability.md
- skills/data-engineering-wiki/durable-key.md
- skills/data-engineering-wiki/error-event-schema.md
- skills/data-engineering-wiki/etl-vs-elt.md
- skills/data-engineering-wiki/event-ingestion-process-time.md
- skills/data-engineering-wiki/extract-archival-for-reprocessing.md
- skills/data-engineering-wiki/extraction-transfer-method.md
- skills/data-engineering-wiki/fact-to-fact-join-cardinality-trap.md
- skills/data-engineering-wiki/file-format-selection-for-pipelines.md
- skills/data-engineering-wiki/generalized-streaming-state-api.md
- skills/data-engineering-wiki/hash-based-change-detection.md
- skills/data-engineering-wiki/header-to-line-fact-allocation.md
- skills/data-engineering-wiki/heuristic-source-watermarks-pubsub.md
- skills/data-engineering-wiki/hierarchy-bridge-type2-ripple-effect.md
- skills/data-engineering-wiki/historic-vs-incremental-load-separation.md
- skills/data-engineering-wiki/hybrid-scd-current-value-backfill.md
- skills/data-engineering-wiki/idempotent-and-replayable-jobs.md
- skills/data-engineering-wiki/incremental-combining-vs-raw-grouping.md
- skills/data-engineering-wiki/incremental-vs-full-extraction.md
- skills/data-engineering-wiki/inferred-member-pattern.md
- skills/data-engineering-wiki/ingestion-time-sampling-tradeoff.md
- skills/data-engineering-wiki/insert-only-history-pattern.md
- skills/data-engineering-wiki/kimball-bus-vs-inmon-cif-architecture.md
- skills/data-engineering-wiki/lake-bypass-for-clean-relational-sources.md
- skills/data-engineering-wiki/lakehouse-vs-warehouse-copy-decision.md
- skills/data-engineering-wiki/late-arriving-data.md
- skills/data-engineering-wiki/latency-tier-triage.md
- skills/data-engineering-wiki/log-based-message-broker.md
- skills/data-engineering-wiki/log-compaction.md
- skills/data-engineering-wiki/log-retention-deadline-for-cdc-consumers.md
- skills/data-engineering-wiki/master-data-management.md
- skills/data-engineering-wiki/message-checksum-corruption-detection.md
- skills/data-engineering-wiki/micro-batch-vs-true-streaming.md
- skills/data-engineering-wiki/mini-dimension-load-mechanics.md
- skills/data-engineering-wiki/modern-data-warehouse-architecture.md
- skills/data-engineering-wiki/multi-cloud-data-platform-tradeoffs.md
- skills/data-engineering-wiki/multi-currency-fact-storage.md
- skills/data-engineering-wiki/multi-table-conditional-insert.md
- skills/data-engineering-wiki/multi-unit-fact-storage.md
- skills/data-engineering-wiki/multiple-data-lakes-rationale.md
- skills/data-engineering-wiki/name-address-parsing-and-standardization.md
- skills/data-engineering-wiki/name-objects-by-content-not-implementation-type.md
- skills/data-engineering-wiki/non-destructive-derived-field-transformation.md
- skills/data-engineering-wiki/null-handling-facts-vs-dimensions.md
- skills/data-engineering-wiki/oltp-vs-olap.md
- skills/data-engineering-wiki/operational-data-store.md
- skills/data-engineering-wiki/orchestration-vs-scheduling.md
- skills/data-engineering-wiki/orchestrator-selection-tradeoffs.md
- skills/data-engineering-wiki/oss-vs-managed-platform-choice.md
- skills/data-engineering-wiki/partitioning-and-clustering-for-query-pruning.md
- skills/data-engineering-wiki/pipeline-granularity-and-blast-radius.md
- skills/data-engineering-wiki/pipeline-metadata-categories.md
- skills/data-engineering-wiki/pipelines-as-code.md
- skills/data-engineering-wiki/polyglot-persistence.md
- skills/data-engineering-wiki/processing-time-windowing-methods.md
- skills/data-engineering-wiki/pull-based-consumer-model.md
- skills/data-engineering-wiki/quality-screens.md
- skills/data-engineering-wiki/ratio-decomposition-and-derived-fact-materialization.md
- skills/data-engineering-wiki/read-path-vs-write-path-precomputation.md
- skills/data-engineering-wiki/read-replicas-for-analytics-offload.md
- skills/data-engineering-wiki/real-time-partition-pattern.md
- skills/data-engineering-wiki/reconciliation-with-set-operators.md
- skills/data-engineering-wiki/recovering-deleted-records-via-forward-insertion.md
- skills/data-engineering-wiki/reference-data-anchored-to-prior-output.md
- skills/data-engineering-wiki/relational-serving-layer.md
- skills/data-engineering-wiki/relational-warehouse-anti-patterns.md
- skills/data-engineering-wiki/reverse-balance-fact-loading.md
- skills/data-engineering-wiki/reverse-etl.md
- skills/data-engineering-wiki/scd-overwrite-load-mechanics.md
- skills/data-engineering-wiki/scd-type2-load-via-change-stream.md
- skills/data-engineering-wiki/schema-change-governance-tradeoffs.md
- skills/data-engineering-wiki/schema-compatibility-modes.md
- skills/data-engineering-wiki/schema-evolution-in-source-systems.md
- skills/data-engineering-wiki/schema-on-write-vs-schema-on-read.md
- skills/data-engineering-wiki/schema-registry.md
- skills/data-engineering-wiki/self-referencing-snapshot-drill-down.md
- skills/data-engineering-wiki/semantic-metrics-layer.md
- skills/data-engineering-wiki/semi-additive-fact-aggregation.md
- skills/data-engineering-wiki/source-system-evaluation.md
- skills/data-engineering-wiki/stepping-stone-architectures-to-mdw.md
- skills/data-engineering-wiki/stovepipe-mart-etl-redundancy.md
- skills/data-engineering-wiki/stream-enrichment-and-joins.md
- skills/data-engineering-wiki/stream-partition-keys-and-hotspotting.md
- skills/data-engineering-wiki/stream-processing-fault-tolerance.md
- skills/data-engineering-wiki/stream-replay-for-reprocessing.md
- skills/data-engineering-wiki/streaming-accumulation-modes.md
- skills/data-engineering-wiki/streaming-accuracy-vs-completeness.md
- skills/data-engineering-wiki/streaming-analytics-replica-cluster.md
- skills/data-engineering-wiki/streaming-persistent-state.md
- skills/data-engineering-wiki/streaming-platform-lineage-mapreduce-to-spark.md
- skills/data-engineering-wiki/streaming-platform-lineage-millwheel-to-beam.md
- skills/data-engineering-wiki/streaming-sql-extensions.md
- skills/data-engineering-wiki/streaming-triggers-and-panes.md
- skills/data-engineering-wiki/streaming-window-types.md
- skills/data-engineering-wiki/streams-and-tables-duality.md
- skills/data-engineering-wiki/strong-vs-weak-productions.md
- skills/data-engineering-wiki/surrogate-key-pipeline.md
- skills/data-engineering-wiki/surrogate-key-range-recovery.md
- skills/data-engineering-wiki/surrogate-vs-business-keys.md
- skills/data-engineering-wiki/synchronous-vs-asynchronous-ingestion.md
- skills/data-engineering-wiki/table-load-patterns.md
- skills/data-engineering-wiki/temporal-validity-windows.md
- skills/data-engineering-wiki/time-varying-relations.md
- skills/data-engineering-wiki/transactional-outbox-pattern.md
- skills/data-engineering-wiki/transformation-vs-query.md
- skills/data-engineering-wiki/two-phase-mutation-pattern.md
- skills/data-engineering-wiki/type1-change-aggregate-corruption-trap.md
- skills/data-engineering-wiki/unbounded-data-processing-approaches.md
- skills/data-engineering-wiki/warehouse-layering-source-staging-presentation.md
- skills/data-engineering-wiki/watermark-aggregation-centralized-vs-in-band.md
- skills/data-engineering-wiki/watermark-and-wall-time-timers.md
- skills/data-engineering-wiki/watermark-computation-and-propagation.md
Every link opens the file at its source, pinned to the revision this page describes.