Data Engineering Knowledge Capture (DEKC)

Git-native second brain for data platforms — schemas, tables, views, queries, SQL/DAX, lineage, medallion layers (bronze → silver → gold), semantic models, dashboards, reports, streams, jobs, business objects, and glossary terms.

DEKC extends Project Knowledge Capture (PKC) and depends on OKF. Multi-agent walk loops are designed with OKF Agent Graph (AGER) so orchestrators, scouts, judges, and synthesizers reverse-engineer real platforms (Azure Fabric, AWS, GCP) into the same portable graph.

Docs

Multi-host

Write isolation (worktree + PR) is in docs/ISOLATION.md. Public examples use fictional lumenfield-detector and northstar-console only. Point SECOND_BRAIN_ROOT at a path the human already has. Never hard-code a private remote.

Why DEKC

Data teams lose institutional memory in tribal knowledge: “what does this gold table mean?”, “which Event Hub lands into bronze?”, “which Glue job promotes silver → gold?”, “which DAX measure powers the exec dashboard?”

DEKC turns lakehouse / warehouse / stream reality into a reviewable OKF knowledge graph that agents can walk, pack, and search — while promoting technical assets into business objects with glossary definitions.

Nouns (this plugin)

DEKC owns the data plane. Catalog / ContextPack live in okf-plugin. AgentNode / Workflow live in AGER. Diagram / Wireframe live in SAC.

BusinessObject, Column, DQRule, Dashboard, DataCatalog, DataContract, DataDomain, DataLake, DataMart, DataProduct, Dataset, DaxArtifact, DesignPattern, GlossaryTerm, IngestionJob, Layer, LineagePath, Metric, Query, Report, Schema, SemanticModel, SourceSystem, SqlArtifact, StorageLocation, Stream, Table, Transformation, View.

Dashboard, DataLake, and GlossaryTerm also exist in SAC with architecture/runtime meaning. Orchestration graphs (Workflow) and walk-loop agents (AgentNode) belong to AGER — DEKC records jobs as IngestionJob / Transformation.

Standard concept schemas: schemas/okf-concepts/. Registry: schemas/README.md.

Second brain (core goal)

DEKC is a schema-typed OKF second brain for data engineering work. Capture platform truth once; reuse it when you:

Standard concept schemas: schemas/okf-concepts/ (Table, SourceSystem, Metric, Dashboard, BusinessObject, …). Registry: schemas/README.md.

python3 scripts/dekc_schemas.py list
python3 scripts/dekc_schemas.py validate --bundle sample-knowledge
python3 scripts/dekc_index.py status --bundle sample-knowledge
python3 scripts/dekc_index.py refresh --bundle sample-knowledge   # self-heals; search/pack also refresh

Diagrams, wireframes & platform concepts

Capture Mermaid and PlantUML listings inside OKF Markdown for:

• Report/dashboard wireframes (PlantUML salt)
• Architecture, component, activity, state, class, ERD, sequence diagrams (jobs, lakes, models)

python3 scripts/dekc_diagram.py wireframe --name "Exec UI" --subject /dashboards/executive-revenue.md
python3 scripts/dekc_diagram.py job-pack --workflow /workflows/daily-medallion-orders.md
python3 scripts/dekc_diagram.py capture --name "Orders ERD" --kind erd --language mermaid --subject /tables/gold-order-daily.md

First-class platform types: DataLake, DataMart, DataCatalog, DataDomain, DataProduct, Stream, StorageLocation, DQRule, IngestionJob.

python3 scripts/dekc_platform.py lake --name "Retail Lake" --platform fabric-onelake
python3 scripts/dekc_platform.py catalog --name "Workspace Catalog" --engine unity
python3 scripts/dekc_platform.py dq-rule --name freshness --target gold-order-daily --rule-type freshness
python3 scripts/dekc_platform.py ingestion --name orders-stream-landing --mode streaming \
  --streams orders-events --lands-as bronze-orders-raw
python3 scripts/dekc_diagram.py ingestion-pack --job /ingestion/orders-stream-landing.md

See docs/diagrams.md.

Ecosystem

okf-plugin (okf-graph-eng)          portable OKF graph ops (validate, pack, impact)
        ▲
        │ depends on
okf-agent-graph (AGER)              multi-agent loop schema (orchestrator/worker/judge,
        ▲                           LoopPolicy, ScratchPad, tools, triggers)
        │ agent loops modeled with
project-knowledge-capture (PKC)     meetings, decisions, features
        ▲
        │ extends
data-engineering-knowledge-capture  data assets, lineage, streams/jobs, glossary (this repo)

What gets captured

Install

Claude Code

# Graph + agent-loop substrate
claude plugin marketplace add SpillwaveSolutions/okf-plugin
claude plugin install okf-graph-eng@okf-plugin-marketplace
claude plugin marketplace add SpillwaveSolutions/okf-agent-graph
claude plugin install okf-agent-graph@okf-agent-graph-marketplace

# Project reasoning (optional but recommended)
claude plugin marketplace add SpillwaveSolutions/project-knowledge-capture
claude plugin install project-knowledge-capture@pkc-plugin-marketplace

# DEKC
claude plugin marketplace add SpillwaveSolutions/data-engineering-knowledge-capture
claude plugin install data-engineering-knowledge-capture@dekc-plugin-marketplace

Grok Build

Grok Build loads Claude-compatible plugins with zero extra config. Optional identity pin: .grok-plugin/marketplace.json.

Codex

codex plugin marketplace add SpillwaveSolutions/data-engineering-knowledge-capture
# /plugins → install data-engineering-knowledge-capture

OpenCode

Skills + AGENTS.md policy; registration in .opencode/plugin/dekc.json.

See PORTS.md.

Quick start

# Scaffold knowledge/
python3 scripts/dekc_common.py init-bundle --repo . --bundle knowledge

# Breadth-first RE plan (pause before specialist fan-out)
python3 scripts/dekc_orchestrate.py --repo . --system "Retail Lake" \
  --scan-root path/to/lake --plan-only

# Walk a lake / SQL / job root (filesystem reverse engineer)
python3 scripts/dekc_walk.py path/to/lake --repo . --bundle knowledge
# or: --from-plan knowledge/.dekc/re-plan.json --area lake

# Materialize lineage + promote gold → business objects
python3 scripts/dekc_lineage.py --repo . --bundle knowledge materialize
python3 scripts/dekc_business.py --repo . --bundle knowledge promote-layer --layer gold

# Index second brain + health (search/pack refresh the index themselves)
python3 scripts/dekc_index.py status --repo . --bundle knowledge
python3 scripts/dekc_doctor.py --repo . --bundle knowledge
python3 scripts/dekc_search.py "revenue" --repo . --bundle knowledge

Slash / skill entry points: /dekc-init · /dekc-plan · /dekc-walk · /dekc-lineage · /dekc-business-object · /dekc-glossary · /dekc-semantic · /dekc-retrieve · /dekc-context · /dekc-search · /dekc-index · /dekc-doctor

Agent loop (AGER-shaped): producers + adversarial judges

Orchestrators: data-lake-walker (default) · reverse-engineering-orchestrator (Fabric/AWS/GCP).

Query-time: data-retriever (/dekc-retrieve) — search + pack stay in the sub-agent; the parent gets a summary card only. Not an RE worker.

Producer workers fan out, then adversarial skeptics grade reverse engineering with rubrics before index/pack. Lead judge: re-adversary-judge (threshold 0.75). Fail → re-plan or retract unproven claims — never invent edges to pass.

Trigger → Orchestrator (LoopPolicy: goal · max_turns · no_progress)
              │ Plan (dekc_plan.py) → ranked task list
              │ FanOut producers (only areas the plan listed)
              ├─ schema-scout · lineage-tracer · stream-job-scout
              ├─ report-cataloger · semantic-mapper
              ├─ airflow/glue/fabric/dbt/duckdb/notebook/dq scouts (signal-gated)
              │ FanOut adversaries (rubrics) — unchanged
              ├─ lineage-skeptic · business-skeptic
              ├─ stream-job-skeptic · coverage-skeptic · layer-auditor
              ▼
         re-adversary-judge  → reverse-engineering-rubric.md
              │ pass? ──no──► revise / retract
              ▼ yes
         index + pack (Synthesizer) + judgment receipt

Automated baseline: python3 scripts/dekc_grade.py --bundle knowledge (skill /dekc-grade). Full cloud RE playbooks: design doc.

Sample knowledge

sample-knowledge/ — retail commerce medallion:

bronze.orders_raw → silver.orders → gold.order_daily + LTV view, DAX measure, semantic model, executive dashboard, business objects, glossary (GMV).

python3 tests/test_dekc.py
python3 scripts/dekc_validate.py --bundle sample-knowledge
python3 scripts/dekc_doctor.py --bundle sample-knowledge

Second-brain index

Git + Markdown is the source of truth. Search and pack use a disposable ladder (SQLite FTS5 index → ripgrep → full scan). Deleting the accelerators is always valid recovery. See retrieval ladder.

Do not commit the index. Add this to your .gitignore (a copy ships at templates/gitignore-fragment):

**/.index/
**/.dekc/index.sqlite
**/.dekc/index.sqlite-journal
**/.dekc/index.sqlite-wal
**/.dekc/index.sqlite-shm

Do not ignore the whole .dekc/ directory — repo-level config lives there. **/.index/ still ignores the old JSON inverted index from 0.4.x so it cannot be re-committed.

dekc_index.py writes:

<bundle>/.dekc/index.sqlite   # incremental, mtime+size self-heal

python3 scripts/dekc_index.py status --bundle knowledge
python3 scripts/dekc_index.py refresh --bundle knowledge
python3 scripts/dekc_index.py drop --bundle knowledge

Search, pack, and doctor refresh the index themselves. build is an alias for refresh --force. Ranking stays in Python so scores match a full scan unless you pass --engine fts. DEKC_NO_INDEX=1 / --no-index fall through to rg then scan. Missing rg is not an error.

Config

See .dekc/config.example.yml.

License

MIT — see LICENSE.