Skip to content

spillwavesolutions/data-engineering-knowledge-capture

v0.5.3MIT

Data Engineering Knowledge Capture: schemas, lineage, medallion, semantic models, and glossary into a durable OKF knowledge graph. Multi-host bindings and write isolation.

GitHubOpen Website

Changelog

Unreleased

0.5.3 — 2026-09-12

Breadth-first reverse-engineering plan, matching SAC 0.5.6’s plan → task list → signal-gated specialist fan-out, without replacing DEKC’s adversarial judges.

Added

  • dekc_plan.py: cheap map of scan roots + optional export paths. Writes .dekc/re-plan.md / .json / progress (ranked areas, checklists, suggested sub-agents). mark / show for done/blocked. Signal-gated: no Airflow markers → no airflow-scout.
  • dekc_orchestrate.py --plan-only and --from-plan --area. Same flags on dekc_walk.py. Unattended orchestrate plans first, then captures only domains the plan detected.
  • Skill dekc-plan / command /dekc-plan. Orchestrators teach plan → ranked task list → producer specialists → existing skeptics → re-adversary-judge. Query-time data-retriever stays separate.
  • Signal-gated specialists: airflow-scout, glue-job-scout, fabric-pipeline-scout, thin adf-scout / stepfunctions-scout / composer-scout / cron-loader-scout, dbt-elt-scout, duckdb-scout, notebook-scout, dq-scout.
  • Cheap walk capture for DuckDB, .ipynb SQL cells, Delta _delta_log, orchestration job stubs, and DQ markers (GE / Soda / dbt tests — no runtime).

Notes

  • DEKC owns data orchestration + ELT/ETL. SAC owns CI/CD. Actions that only trigger Glue/dbt are a cross-link note, not a Pipeline noun.
  • No live Glue/S3/Fabric control-plane calls. Exports/mirrors only.

0.5.2 — 2026-09-12

Query-time retrieval sub-agent, matching the PKC/SAC parity pattern: search + score + pack + deepen stay off the parent; the parent gets a summary card only.

Added

  • data-retriever agent — retrieval-only. No capture, walk, reverse engineer, or knowledge writes. Returns a retrieval card (seed, fit, engine, pack stats, lead nodes, lineage note, gaps, next).
  • /dekc-retrieve skill + slash command. Parent must spawn data-retriever and must not run dekc_search.py / dekc_pack.py in-process for Q&A. Orthogonal fan-out: PKC knowledge-retriever, SAC architecture-retriever.
  • dekc_pack.py --summary — compact card-friendly output, bodies off (root included), fail-closed token budget. Retriever prefers --tiny --summary.

Notes

  • Reverse-engineering agents (data-lake-walker, lineage-tracer, …) are unchanged. Query-time is a separate section on the RE orchestrator.

0.5.1 — 2026-08-31

Correctness patch for catalog rendering and the rg-backed reverse index. Both defects were found by porting the system-architecture-capture v0.5.4 fixes to the shared DEKC code.

Fixed

  • refresh_catalog_index() no longer crashes on a YAML scalar title. title: 421 parses as an int, and _escape_link_label() assumed a string, so one legacy title aborted the refresh after capture had already written concepts. A falsy-but-real title (0, false) also no longer falls back to the file stem. (#51)
  • The rg-backed lineage pack no longer drops matches when the bundle is addressed through a symlink. rg_list_files() resolves its hits, so path.relative_to(bundle) raised and the handler silently discarded a real lineage neighbor while the pack still reported reverse_index: rg. (#52)

0.5.0 — 2026-08-30

Retrieval ladder: Git + Markdown stays source of truth. Accelerators are disposable. Replaces the JSON knowledge/.index/ inverted index (which was committed by accident).

Added

  • ripgrep accelerator for search and pack. dekc_search.py uses rg -l as a candidate prefilter when rg is on PATH (or DEKC_RG_PATH / PKC_RG_PATH / OKF_RG_PATH). Ranking stays in Python, so scores match a full scan. --no-rg forces the linear walk. Missing rg is not an error. Never installed from a hook.
  • rg-backed reverse index in dekc_pack.py. Inbound discovery is rg -lF of the concept path; the current file is always parsed for outbound typed-flow + SQL lineage (the file almost never contains its own path).
  • dekc_doctor.py reports a toolchain section (rg found/missing, FTS5, index).
  • SQLite/FTS5 incremental index. scripts/dekc_index.py writes knowledge/.dekc/index.sqlite (gitignored). Search and pack self-heal via mtime+size on every call. LIKE on a stored haystack is the default candidate prefilter so Python scores stay identical to a scan. --engine fts is the FTS5 MATCH opt-in. --no-index / DEKC_NO_INDEX=1 fall through to rg then scan. /dekc-index skill for status/refresh/drop. build remains an alias for refresh --force (CI / existing skills).
  • SQL lineage edges store origin (the file that authored them) so incremental delete does not leak table-to-table edges after the source file vanishes.
  • Retrieval ladder locked in docs/designs/retrieval-ladder.md.

Changed

  • Host manifests (.opencode/plugin/dekc.json, public/data/catalog.json) lockstep with root plugin.json 0.5.0. They were still labeled 0.4.2.

Notes

  • Pack identity: lineage is undirected for every visited node; extra non-lineage links[] still attach to the focus only. Graphs match a scan unless --engine fts.
  • dekc_pack.py --mermaid still calls build_graph (full scan). Accepted leftover.
  • Do not ignore the whole .dekc/ directory — .dekc/config.example.yml is committed. Gitignore only **/.dekc/index.sqlite*.

0.4.3 — 2026-08-30

  • dekc_doctor.py reports toolchain (ripgrep / SQLite FTS5). Search already goes through knowledge/.index; rg is optional, not a second index.

0.4.2 — 2026-08-25

Fabric reverse-engineering walk fixes (#44).

Fixed

  • #26 YAML description: > / | is a block scalar, not a nested map. _patch no longer eats timestamp / rel / status.
  • #27 dekc_grade.py coerces non-string descriptions instead of TypeError.
  • #28 Schema contains merges; last table no longer wins.
  • #29 slugify keeps a pipe token; --slug / --fabric-id; colliding titles suffix -2.
  • #30 Generated captures default verified: false unless --verified or SQL/columns.
  • #31 --source without SQL/evidence is related_to, not sourced_from.
  • #32 LineagePath accepts absolute paths; BI hops use queries; no automatic transforms_to.
  • #33 IngestionJob layer writes_to only with --lands-as; --refreshes for semantic refresh.
  • #34 DataLake mermaid lists only the layers passed.
  • #35 workflow subcommand errors and points at ingestion / transformation.
  • #36 Vacuous SemanticModel body sentence removed.
  • #37 WriteEvents opt-in (DEKC_WRITE_EVENTS=1 / --write-event); gitignore fragment.
  • #40 build_graph reads PKC rel: maps as well as links[].
  • #42 vw* / x_vw* / CREATE VIEW capture as View.
  • #43 One log.md line per CLI invocation.

Added

  • #38 dekc_walk.py --fabric-items / --pbi-bindings / --inventory (filesystem walk remains).
  • #39 dekc_grade.py / dekc_doctor.py default to the 29 DEKC nouns; --prefix / --tag / --since / --all.
  • #41 --fabric-id / --workspace / --dataset-id / --fabric-type on capture CLIs.

0.4.1 — 2026-08-24

0.4.0 — 2026-08-24

Changed

  • Noun ownership. DEKC keeps the data plane. Removed leaked types: AgentNode, Workflow (AGER), ContextPack / BaseConcept (okf-plugin), Diagram / Wireframe (SAC).
  • Registry 0.3.0 lists the 29 DEKC nouns. README has the full table.
  • Dual-owned names documented: Dashboard, DataLake, GlossaryTerm (also SAC).

0.3.5

  • WikiTicket SDD (worklog) is the tracking system for this plugin.

0.3.4

  • Three-host hooks: Codex + Cursor-native when Claude hooks exist.

0.3.3 — 2026-08-17

  • Cursor host. .cursor-plugin/plugin.json (Cursor Plugins) plus .cursor/rules/second-brain.mdc. Docs: docs/CURSOR.md. docs/GROK_BOT.md now covers Grok Bot spawning Cursor cloud agents.

Notable changes to data-engineering-knowledge-capture. Newest first.

0.3.2 — 2026-08-16

Added

  • ContextPack token budget matches second-brain-core 0.3.3 / PKC 0.7.2: default 1/4 of SECOND_BRAIN_WINDOW_TOKENS (128000 → 32000). Override with --max-tokens or SECOND_BRAIN_PACK_MAX_TOKENS.
  • Pack is fail-closed when the rendered subgraph exceeds the budget. --write is skipped.
  • Bodies off unless that node is the pack root. Neighbors keep title, type, path, and frontmatter description only.
  • Node clip (--max-nodes / --tiny) is not a token budget.
  • Implements part of okf-plugin#55.

0.3.1 — 2026-08-16

Added

  • Required identity on every knowledge write: --author or SECOND_BRAIN_IDENTITY.
  • write_knowledge() stamps author and emits a WriteEvent. write_concept stays pure.
  • Wired through capture, walk, platform, diagram, business, brain --write, pack --write, grade --write, and link.
  • Fail-closed tests. Scan / pack / grade / brain print-only paths do not require identity.

0.3.0 — 2026-08-15

Added

  • Multi-host bindings + write isolation. Root Agent Plugins 1.0 plugin.json, Grok Bot / Deep Agents / isolation / onboarding docs, host wrappers, vendored scripts/brain_session.py, and dekc-session skill/command.
  • Concurrent writers read main and write brain/<actor>/<session-id>. Close via PR against the checkout's existing remote.
  • Isolation tests use fictional lumenfield-detector / northstar-console actors only.

0.2.1 — 2026-08-13

Changed

  • Aligned BaseConcept with the shared okf-plugin envelope. Required frontmatter is now type + title only (description and timestamp stay recommended). dekc_validate.py and every type schema were updated so a mixed second brain with PKC/SAC nodes that omit those fields still validates.
  • truth_state union. Accepts PKC/SAC values (snapshot, superseded, archived) in addition to DEKC's current | historical | proposed.

0.2.0 — 2026-08-10

Six fixes, all found by running this plugin alongside project-knowledge-capture and system-architecture-capture against a single shared bundle.

Fixed

  • Frontmatter round-trip doubled backslash escaping. The dumper escaped backslashes and quotes; the reader stripped only the surrounding quotes. Every write-modify-write cycle re-escaped already-escaped text, doubling the backslash count each pass. Worse here than in the sibling plugins, because both write_concept and refresh_catalog_index do a read-modify-write and the latter is driven by the curate hook — so ordinary editing compounded it. It was also self-concealing: reading back with the same parser returned a value that looked correct. (#3)

  • A bracketed concept title dropped the catalog edge. Fact [Sales] rendered as [Fact [Sales]](/tables/a.md), which the graph reader's link regex cannot match. That produces a missing edge rather than a broken one, and validate reports only broken edges — so the concept lost its catalog backlink silently. Note this half needs the matching reader change to take effect: backslash escaping does not rescue a reader whose label class is [^\]]+. (#2)

  • The · <layer> annotation broke shared bundles. It fired for any concept with a layer key in any of the 32 catalogs, including the 8 that system-architecture-capture also declares — whose renderer emits a bare - [label](path). So a shared catalog flipped on every alternation between plugins. Now scoped to catalogs only this plugin owns; all three plugins render a shared catalog byte-identically. (#4)

  • refresh_catalog_index accepted any catalog name. It now refuses catalogs this plugin does not declare, so an outside caller cannot drive this renderer over a sibling's catalog. On its own this does not stabilise a shared bundle — for a catalog two plugins both declare it passes in both — which is why the annotation scoping above is the load-bearing change. (#4)

  • resolve_knowledge_root fell through to sample-knowledge/ in silence. When the configured root is not an initialized bundle, resolution probes other candidates. That is reasonable; not saying so was not. There are 16 call sites and only dekc_doctor announced the bundle it used — and this repo ships a sample-knowledge/, so a capture run inside a clone wrote there. (#5)

  • The curate hook refreshed all 32 catalogs on every edit. Each refresh is a whole-file read-modify-write, so rapid edits raced by construction. Now scoped to the catalog containing the edited file. (#6)

  • dekc_link --help advertised eight relations, none of which produced a lineage edge. The intersection of DEFAULT_RELATIONS[:8] with the set build_graph honours was empty, and --rel is unvalidated. Four relations the plugin itself emits and documents as flow — lands_as, lands_into, visualizes, consumes_stream — were also ignored, so packs built from them were incomplete. The honoured set is now named constants, those four are honoured, and the help names what actually works. implements/documents still produce no edge: graph.json is lineage adjacency by design. (#7)

Added

  • A .gitignore fragment at templates/gitignore-fragment and a README note, since nothing previously told users the derived .index/ should not be committed. Uses **/.index/, which holds at any depth and for every bundle rather than one hardcoded name. (#8)

  • dekc_index now explains a zero-edge lineage graph instead of writing a bare 0 that is indistinguishable from a failed build.

0.1.0

Initial release.