Code-Crawler
npm version npm downloads npm total downloads
Static + dynamic call tracing MCP server backed by a persistent property graph of your codebase.
Code-Crawler indexes a project into a SQLite-backed property graph — files, packages, modules, classes, interfaces, methods, fields, imports, annotations/decorators, call edges, virtual dispatch, aspect advice, and the wiring frameworks add (entry points, routes, injected dependencies, reflective lookups) — then exposes that graph to LLM clients as MCP tools: find callers, trace call paths, explore neighborhoods, follow runtime dispatch, and search symbols across the whole codebase.
Supports Java, Python, TypeScript, and JavaScript projects, and can index all of them into the same graph.
📖 Documentation site: https://nithin554.github.io/code-crawler/
📘 Task-oriented guide: USAGE.md
🗺️ Roadmap: ROADMAP.md
┌─────────────────────── MCP (stdio) ───────────────────────┐
│ crawl_project search_symbols get_callers get_callees │
│ trace_path trace_execution find_aspects get_decorators│
│ find_implementations find_hotspots find_dead_code ... │
└───────────────┬──────────────────────────────┬───────────────┘
│ │
┌──────▼───────┐ ┌────────▼────────┐
│ crawler │ │ graph queries │
│ parse+resolve│ │ recursive CTEs │
└──────┬───────┘ └────────┬────────┘
│ │
┌──────▼──────────────────────────────▼──────┐
│ SQLite property graph (WAL) │
│ nodes (kind, fqn, props) + edges (kind) │
└─────────────────────────────────────────────┘
Why a graph?
Plain-text grep can't answer questions like:
- "Who transitively calls
OrderRepository#create?" - "Is there a static path from
Main#maintoTracking#dispatch?" - "What is the impact radius of changing
OrderService#place?"
These need graph traversal. Code-Crawler builds the graph once (idempotently, file-by-file) and answers them with bounded recursive SQL queries.
Features
- Four languages, one graph: Java (
tree-sitter-java), Python (tree-sitter-python), TypeScript (tree-sitter-typescript,tree-sitter-tsx), and JavaScript (tree-sitter-javascript) — all WASM, no native build step. Language is detected per file, so a polyglot repo lands in a single graph. - Dynamic tracing on a static graph —
DYNAMIC_DISPATCHedges connect interface/base calls to every override in the project, andADVISESedges weave AspectJ/Spring AOP advice into execution traces (before,after,around,after-returning,after-throwing, named@Pointcutreferences,&&/||/!andexecution,within,@annotation,@within,target,this,argsprimitives). - Run-time indirection is named, not lost — Java reflection/
ServiceLoader, Pythongetattr/importlib, and JSrequire/dynamicimport()produceREFLECTSedges to the resolved symbol when the target is statically determinable, or to a__reflect__:marker when it is not. - Framework wiring — dependency injection (
@Autowired, NestJS constructor injection, FastAPIDepends) becomesINJECTSedges, and HTTP routes (@GetMapping,@RequestMapping, NestJS@Controller/@Get, FastAPI/Flask/Quart decorators, Expressapp.get) becomeROUTESedges to the handler, which is also marked as an entry point. - Control flow on call edges — every
CALLSedge records whether the call is conditional and the compact source text of its enclosing guards/loops, so an agent can tell a happy path from an error path without reading the file. - External dependencies are explicit — third-party imports and runtime globals
become package-level
external_symbolnodes withREFERENCESedges and afind_external_dependenciessummary, distinct from project code. - Annotations as first-class nodes —
@Transactional,@Cacheable,@Async,@Retryable, Python decorators, and TS decorators becomeannotationnodes withANNOTATED_WITHedges, so you can query decorators on types, methods, and fields. - Call resolution heuristics that go beyond naive string matching:
receiver types inferred from locals, parameters, fields,
new T()expressions,self/thisscoping, dotted paths (service.repo.find), loop-variable element types, bare/this/supercalls, static/qualified calls, and superclass chains. Unresolvable calls becomeunknown_symbolphantom nodes withresolved:falseedges, so the graph stays traversable. - Incremental, file-scoped indexing — each crawl records a per-file manifest
(
size, mtime, SHA-256) and replays the cached parse result for files whose content is unchanged, so a re-crawl re-parses only what actually changed. Changes are reported asfilesAdded/filesUpdated/filesUnchanged/filesRemoved. Derived dynamic edges and orphan pruning are rebuilt only when something changed. A full re-crawl still removes files that disappeared, without duplicating the graph, and scoped crawls (fileFilteror a forcedlanguage) never remove files outside their scope. When a checkout,rsync -t, an mtime-preserving unzip, or clock skew restores an edit with the old size and mtime, the shortcut misses it — passforce(--forceon the CLI) to re-parse everything. - FTS5-backed symbol search — symbol lookup runs against an external-content FTS5 index
with BM25 ranking and exact/prefix boosting, falling back to a
LIKEscan only when FTS finds nothing. Search does not degrade as the graph grows. - Versioned schema with automatic migration — the database carries a schema version and upgrades itself in place on open. A database written by a newer build is rejected with a clear message instead of being read as if the shape were known.
- A database per project — the index lives in
<project>/.code-crawler/index.db, so two repositories can never collide. Pointing a populated database at a different project fails with aclear_indexhint instead of silently wiping the first index. - Analytics — hotspots (fan-in/fan-out/advice counts) and dead-code candidates.
- Self-contained: single SQLite file, no external services.
- LLM-friendly: every tool returns structured JSON, unknown symbols come back with fuzzy search suggestions.
Install & build
Requires Node.js ≥ 22.12 (better-sqlite3@13 and the Vitest toolchain both need it).
npm install
npm run build # tsc → dist/
npm test # vitest (full suite; set CODECRAWLER_BENCH=1 for the scale test)
npm run lint # biome check (lint + format, fails CI)
npm run format # biome format --write
npm run coverage # vitest run --coverage (thresholds in vitest.config.ts)
npm run bench # crawl/query benchmark on a generated ~137k-line project
CLI
codecrawler crawl <path> # index a project
codecrawler crawl <path> --force # re-parse every file, ignoring cached metadata
codecrawler status # index summary
codecrawler query <symbol> # fuzzy-lookup a symbol
codecrawler serve # start the MCP server over stdio (default)
Environment variables:
| Variable | Default | Purpose |
|---|---|---|
CODECRAWLER_DB | <project>/.code-crawler/index.db | Database path. Resolved from the cwd upward, so each project gets its own file; falls back to ~/.code-crawler/index.db when no project encloses the cwd. The home directory is only that fallback, never an enclosing project |
CODECRAWLER_EXCLUDE_DIRS | .git,node_modules,dist,build,target,... | Dirs skipped during crawl |
CODECRAWLER_DEFAULT_LIMIT | 50 | Default result cap |
CODECRAWLER_MAX_DEPTH | 10 | Traversal depth cap (1–100) |
CODECRAWLER_MAX_FILE_BYTES | 1000000 | Files larger than this are skipped, not parsed |
CODECRAWLER_RESPECT_GITIGNORE | true | Honor the project's .gitignore during discovery |
CODECRAWLER_HELPFUL_ERRORS | true | Fuzzy suggestions on symbol misses |
CODECRAWLER_PLUGIN | unset | Plugin mode: resolve one database per project at call time instead of opening one file at startup |
CODECRAWLER_PLUGIN_ROOT | unset | Plugin install directory, never indexed as a project |
CODECRAWLER_PLUGIN_DATA | temp dir | Writable directory for the no-project fallback database |
Invalid values never crash the server: booleans fall back to their default and integers are clamped to their documented range, each with a one-line warning on stderr.
What gets indexed
crawl walks the project and skips .git, node_modules, build output, and
other CODECRAWLER_EXCLUDE_DIRS entries. On top of that:
-
the project's
.gitignoreis honored (disable withCODECRAWLER_RESPECT_GITIGNORE=false), -
an optional
.crawlerrcin the project root can narrow or widen discovery with gitignore-style globs, and relocate where module ids start:{ "include": ["src/", "packages/core/"], "exclude": ["**/generated/"], "moduleRoots": ["src"] }When
includeis non-empty, only matching files are indexed. A malformed.crawlerrcis ignored with a warning rather than failing the crawl.moduleRootsis opt-in: with["src"],src/orders/service.tsbecomes the moduleorders.serviceinstead ofsrc.orders.service, which is what abaseUrl-or-PYTHONPATHproject actually imports. It is never inferred, because inferring it would silently renumber an existing index. TypeScript and JavaScript projects also get path aliases for free:compilerOptions.pathsandbaseUrlare read fromtsconfig.json(orjsconfig.json), followingextendswithin the project, soimport { x } from "@app/orders/service"resolves to the real module instead of being recorded as external. Changing either setting re-parses cached files on the next crawl. -
files above
CODECRAWLER_MAX_FILE_BYTESand files that look binary are skipped;crawl_projectreports them inskippedFileswith a reason. -
per-file parse failures never abort the crawl; they are reported in
parseErrors.
Only the root .gitignore is read — nested .gitignore files are not yet
supported.
Incremental crawls
A crawl records a manifest row per file (size, mtime, SHA-256, language). On the
next crawl a file whose size and truncated mtime both match is served from a cached
parse result instead of being re-read and re-parsed; a file whose mtime changed but
whose hash did not is also replayed from cache. Resolution, phantom-node creation and
derived edges still run over the whole in-memory file set, so cross-file edges stay
correct — only I/O, parsing and database writes are skipped.
crawl_project reports the split:
| Field | Meaning |
|---|---|
filesAdded | First seen by this crawl |
filesUpdated | Content changed, re-parsed |
filesUnchanged | Replayed from the parse cache |
filesRemoved | Disappeared from disk and were pruned |
Known limitation: because an unchanged file's edges are not rewritten, a call that
becomes newly resolvable only because some other file changed (for example a method
gained an overload, or a class was introduced that a previously-ambiguous call could
now bind to) will not be upgraded in the untouched file. Touch that file, or run
clear_index, to force full resolution.
Benchmarks
npm run bench generates a synthetic TypeScript project (575 modules, ~137k lines,
~9.8k methods), crawls it, and reports crawl phases and query latency percentiles.
tests/perf.test.ts asserts the same budgets and is opt-in, because a realistic
corpus takes time to crawl:
npm run bench # human-readable report
CODECRAWLER_BENCH=1 npm test -- tests/perf.test.ts # enforce the budgets
CODECRAWLER_BENCH_LINES=20000 npm run bench # smaller corpus
Measured on a Windows dev laptop with a warm disk cache; absolute numbers vary, the ratios and the index usage are what matter:
| Phase | Duration | Added | Updated | Unchanged |
|---|---|---|---|---|
| Cold crawl | 22.5 s | 575 | 0 | 0 |
| Re-crawl, nothing changed | 0.91 s | 0 | 0 | 575 |
| Re-crawl after a one-file edit | 0.75 s | 0 | 1 | 574 |
| Query | p50 | p95 | Budget (p95) |
|---|---|---|---|
search_symbols (exact) | 0.69 ms | 1.96 ms | 50 ms |
search_symbols (prefix) | 1.60 ms | 4.09 ms | 50 ms |
get_callers (depth 2) | 0.16 ms | 0.21 ms | 50 ms |
get_callees (depth 2) | 0.11 ms | 0.15 ms | 50 ms |
get_subgraph (radius 2) | 0.16 ms | 0.27 ms | 50 ms |
find_entry_points | 0.12 ms | 0.29 ms | 50 ms |
find_hotspots | 145.5 ms | 181.4 ms | 1000 ms |
find_dead_code | 99.8 ms | 290.3 ms | 1000 ms |
The index is 66 MB for those 14.4k nodes / 38.8k edges. find_hotspots and
find_dead_code are whole-project aggregations over every edge and node, which is why
their band is an order of magnitude looser than the symbol lookups; both were profiled
with EXPLAIN QUERY PLAN and are served by idx_edges_target_kind.
CI & publishing
GitHub Actions runs on every pull request (pull-request.yml): npm ci →
typecheck → tests → build → smoke-test of the built CLI.
Merges to main trigger publish.yml, which re-runs the same checks and then,
if green, auto-releases:
- Reads the most recent
v*tag (orpackage.jsonversion on the first run), computes the next patch version, and pushes avX.Y.Ztag. - Creates a matching GitHub release.
- Publishes the package to the npm registry with provenance (OIDC trusted publishing) — no token or repository secret is needed. The step is idempotent and retries: it skips versions npm already has, and a transient transparency-log or registry failure no longer aborts the release.
- Publishes
server.jsonto the official MCP Registry asio.github.nithin554/code-crawlerfrom a separate, re-runnable job usingExaDev/mcp-registry-publish. The job runs inside themcp-registry-publishenvironment, whose deployment branch policy is limited tomain. It pinsserver.jsonto the release version, authenticates with GitHub OIDC, skips versions the registry already lists, and retries transient npm propagation or registry errors. Ownership is verified through themcpNamefield inpackage.json.
Using as an MCP server
Install as an agent plugin
The repository is also an agent plugin, so Claude Code and Copilot CLI can install the server and the skill together:
# Claude Code
/plugin marketplace add nithin554/code-crawler
/plugin install code-crawler@code-crawler
# Copilot CLI
copilot plugin marketplace add nithin554/code-crawler
copilot plugin install code-crawler@code-crawler
The plugin runs npx -y @nithin554/code-crawler serve. Agent Plugins clients start a
plugin's stdio servers inside the plugin directory, so the server does not treat that
directory as your checkout: it resolves a project from crawl_project, the client's
workspace roots, or the project this session already selected, and keeps that graph in
<project>/.code-crawler/index.db — one database per repository, the same layout the CLI
uses. CODECRAWLER_DB still pins a single file when you want one.
The same skill is published to ClawHub as code-crawler, and the
plugin is published there too:
- Skill —
clawhub install code-crawler. ClawHub assigns this version by bumping the patch digit of whatever it already published, so it advances one release at a time instead of copying the npm version; a release whose skill publish fails leaves the two numbers one apart until the next publish. - Plugin — published as the
code-crawlerbundle plugin, versioned from the release tag: releasev0.1.67publishes the bundle at0.1.67. The bundle's ownplugins/code-crawler/package.jsonis packaging metadata — ClawHub needs a package manifest to inspect the bundle and build its ClawPack artifact, and warns when one is missing — so the release workflow stamps the tag's version into it, packs the bundle, and publishes that tarball; the committedversionis only a placeholder.
The plugin publish runs the official clawhub CLI itself, under Node, rather than calling
upstream's reusable package-publish.yml. That workflow cannot publish this bundle: it
inspects the packed artifact with bun ... package validate, and the CLI refuses to run under
Bun ("Plugin validation requires Node.js 22.13+ (22.x) or 24+"). A ClawPack always contains
package/package.json and no input skips the step, so the call only ever succeeded while the
bundle had no manifest — the very thing ClawHub warns about.
Publishing is restricted to main. The publishing jobs deploy through the clawhub-publish
environment, whose deployment branch policy rejects any other ref, so a manual dispatch from
a feature branch stops before it can upload. CLAWHUB_TOKEN is a repository secret, not an
environment secret: the skill call hands it to upstream, which reads it as clawhub_token, and
a job that calls a reusable workflow cannot declare an environment. Forwarding the value
through a job output does not help either — GitHub refuses to emit a secret-valued output and
logs Skip output 'clawhub_token' since it may contain secret.
One command
add-mcp registers the server in whichever
client you name — Claude Code, Cursor, VS Code, Claude Desktop, Codex, OpenCode, Zed, and
more (npx add-mcp list-agents). Run it from inside the repository you want indexed:
npx add-mcp @nithin554/code-crawler --args serve -a cursor
That writes project-scoped config next to the code (.mcp.json, .cursor/mcp.json,
.vscode/mcp.json, …), so each repository carries its own registration and its own
.code-crawler/index.db. The server name code-crawler is derived from the package.
--env CODECRAWLER_DB is optional — add it only to put the database somewhere else:
--env CODECRAWLER_DB=/absolute/path/to/index.db. -g writes user-level config instead. Prefer that only for clients that launch the server
with your checkout as its working directory: with -g and an unrelated working directory
(Claude Desktop starts in its own install folder) the server falls back to
~/.code-crawler/index.db, and every repository then competes for that one file.
One server per project
Outside plugin mode a server process opens one database at startup and a database holds exactly one project, so the model is one registration per repository — which is what the per-project config above gives you. The plugin install instead resolves a database per project at call time, so one plugin serves every repository you open. Re-crawling a repository reuses its existing file and only re-parses what changed; two repositories never share a file, so they cannot clash. Only a few project databases stay open at once (idle ones are closed), and every handle is released when the client disconnects.
MCP Registry / npx
The server is listed in the MCP Registry as io.github.nithin554/code-crawler.
Any npx-capable client can run it without a local checkout:
{
"mcpServers": {
"code-crawler": {
"command": "npx",
"args": ["-y", "@nithin554/code-crawler", "serve"]
}
}
}
Claude Desktop
claude_desktop_config.json:
{
"mcpServers": {
"code-crawler": {
"command": "node",
"args": ["/absolute/path/to/code-crawler/dist/index.js", "serve"],
"env": { "CODECRAWLER_DB": "/absolute/path/to/your/index.db" }
}
}
}
Cursor / other stdio MCP clients
Point the client at the same command. The typical flow for an LLM agent:
crawl_projectthe repo once.find_entry_pointsto see where execution actually starts (routes, listeners, mains).search_symbolsto locate the symbol of interest.get_callers/get_callees/trace_pathto reason about the call graph.trace_executionto follow runtime dispatch and aspect advice;find_aspects,get_decorators, andfind_annotatedto inspect behaviour-changing annotations.find_routes,find_injections,find_reflections, andfind_external_dependenciesto map framework wiring and third-party surface.
See USAGE.md for the full task-oriented walkthrough.
MCP surface
Tools
| Tool | Purpose |
|---|---|
crawl_project | Index a directory (idempotent; one project per database) |
index_status | Node/edge/file counts + last crawl |
search_symbols | Fuzzy lookup by name or FQN, optional kind filter |
get_node | Metadata + source location for one symbol |
get_callers | Direct or transitive callers (depth-bounded) |
get_callees | Direct or transitive callees (depth-bounded) |
trace_path | Static call paths between two symbols (BFS, shortest first) |
trace_execution | Dynamic trace: static calls + virtual dispatch + woven advice |
find_implementations | Implementers/subclasses of an interface or superclass |
find_aspects | Aspects, advice kinds, pointcuts, and intercepted methods |
get_decorators | Annotations/decorators on a symbol and optionally its members |
find_annotated | Every symbol annotated with a given annotation |
find_hotspots | Rank methods by fan-in, fan-out, and advice interceptions |
find_dead_code | Methods with no incoming call/dispatch/advice edges |
find_entry_points | Framework-invoked roots: HTTP routes, listeners, schedulers, CLI mains, tests |
find_routes | HTTP route → handler mappings (@GetMapping, NestJS, FastAPI, Express) |
find_injections | DI-supplied dependencies (@Autowired, NestJS, FastAPI Depends) |
find_reflections | Run-time lookups: Class.forName, getattr, importlib, dynamic import() |
find_external_dependencies | Third-party packages with reference and file counts |
find_unresolved | Call sites the resolver could not bind, grouped by failure reason |
impact_analysis | Callers + entry points + routes + overrides + risk band in one call |
explain_symbol | One-shot digest: members, decorators, callers, callees, wiring |
find_cycles | Call cycles (SCCs) with a sample closed path, plus direct recursion |
export_graph | Neighborhood rendered as Mermaid or Graphviz DOT |
get_subgraph | Neighborhood extraction for visualization |
get_schema | Describe the graph model |
clear_index | Reset the database |
Resources: codecrawler://schema, codecrawler://status
Prompt: trace-call-path(from, to)
Token budgeting. The 18 list-returning query tools — every tool whose payload is a
single result list — accept max_tokens (100–200000) and cursor. Passing either adds a
page block with totalCount, returnedCount, truncated, estimatedTokens, and
nextCursor; omit both and the response shape is unchanged, so existing clients keep
working. Paging never returns an empty page while items remain, so a continuation loop
always terminates. Token counts are estimates from JSON length (~4 characters per token) —
no tokenizer dependency, deterministic across models.
get_subgraph is the one exception: its nodes and edges are two views of the same slice,
so cutting one list would leave edges pointing at omitted nodes. It is bounded by radius and
limit instead. Aggregate tools (impact_analysis, explain_symbol, export_graph) and
single-object tools (get_node, index_status, get_schema) return one object and take no
budget.
Agent skill
The server tells an agent what it can call; the skill tells it when. skills/code-crawler/
ships in the npm package and is plain Markdown — SKILL.md for discovery and the
crawl-first workflow, reference/tools.md mapping all 27 tools to the questions they
answer, and reference/edge-semantics.md explaining confidence, external, and phantom
nodes.
npx skills add nithin554/code-crawler
The skills CLI finds it by convention and installs to whichever agents
you pick; npx skills update refreshes it. To install by hand instead, copy the directory
into your client's skills folder (~/.claude/skills/code-crawler/,
.claude/skills/code-crawler/, or a plugin's skills/ root). No build step. See
USAGE.md.
tests/skill.test.ts asserts the skill's tool table against the live registry, so it
cannot drift as tools are added.
Graph model
Nodes — kind is one of file | package | module | class | interface | enum | method | constructor | field | import | external_symbol | aspect | annotation_type | annotation | unknown_symbol. Every node has a globally unique qualifiedName and a JSON properties
blob (signature, modifiers, lines, annotations, ...).
Edges — kind is one of CALLS | DECLARED_IN | CONTAINS | IMPLEMENTS | EXTENDS | IMPORTS | REFERENCES | DYNAMIC_DISPATCH | ADVISES | ANNOTATED_WITH | REFLECTS | INJECTS | ROUTES. CALLS edges carry line, resolved, confidence, and — when the call sits
behind control flow — conditional, controlFlow, and guards; ANNOTATED_WITH edges
carry line and arguments; ADVISES edges carry adviceKind and the resolved
pointcut; INJECTS carries injectedName and framework; REFLECTS carries
mechanism; REFERENCES carries external.
Id scheme:
| Symbol | Example |
|---|---|
| type | com.acme.OrderService |
| method | com.acme.OrderService#place |
| constructor | com.acme.OrderService#<init> |
| field | com.acme.OrderService$repo |
| annotation | @Transactional |
| import | __import__:java.util.List |
| unresolved call | __unknown__:System.out.println |
| Python/TS module | module:app.service |
| module-level function | module:app.api#read_order |
| module-level field | module:app.service$AUDIT_LOG |
| inline arrow handler | module:src.routes#<anonymous@7> |
| external package | ext:fastapi |
| reflective marker | __reflect__:./plugins/${name} |
| injection marker | __inject__:Repo |
Dynamic behaviour
Static call edges answer "what does the source literally call?". Runtime behaviour differs in two ways, both modelled as derived edges rebuilt on every crawl:
Virtual dispatch
A call to an interface or base-class method may execute any override. After all files
are indexed, every resolved CALLS edge whose target is an overridable method gets
DYNAMIC_DISPATCH edges to each override found in the project:
Checkout#checkout --CALLS--> Payment#pay
Payment#pay --DYNAMIC_DISPATCH--> CardPayment#pay
Payment#pay --DYNAMIC_DISPATCH--> UpiPayment#pay
Constructors and private/static/final methods are excluded because they cannot be
overridden. Use find_implementations to list candidates and trace_execution to follow
them.
Aspects, advice, and decorators
@Aspect types are indexed as aspect nodes. Advice methods are recognised by
@Before, @After, @Around, @AfterReturning, @AfterThrowing, and @Pointcut.
Named pointcuts (@Around("paymentCall()")) are resolved to their @Pointcut
expression, then matched against every project method to create ADVISES edges:
AuditAspect#auditPayment --ADVISES--> Payment#pay
AuditAspect#auditPayment --ADVISES--> CardPayment#pay
AuditAspect#auditPayment --ADVISES--> UpiPayment#pay
AuditAspect#beforeCheckout --ADVISES--> Checkout#checkout
trace_execution weaves these in as INTERCEPT hops, so the trace reads:
Checkout#checkout
INTERCEPT AuditAspect#beforeCheckout
CALLS Payment#pay
INTERCEPT AuditAspect#auditPayment
DYNAMIC_DISPATCH CardPayment#pay -> authorize
DYNAMIC_DISPATCH UpiPayment#pay -> settle
Every other annotation is indexed as an annotation node with an ANNOTATED_WITH edge.
Framework decorators (@Transactional, @Cacheable, @Async, @Retryable,
@PreAuthorize, @Scheduled, ...) are additionally flagged decorator: true so
get_decorators and find_annotated can surface runtime-semantic annotations.
Python decorators and TypeScript decorators are modelled the same way: @router.get(...),
@injectable, @Controller(...), @Get(':id') and friends land in annotations on the
declaring type/method, and framework ones additionally produce routes, injections, or
entry-point marks.
Reflection and run-time indirection
Lookups the compiler cannot see through are recorded as REFLECTS edges. When the target
is statically determinable — a string literal, a literal prefix plus a variable
(importlib.import_module("app.plugins." + name)), getattr(obj, "run"),
require("./x") — the edge points at the resolved symbol. Otherwise it points at a
__reflect__: marker carrying the mechanism:
module:app.dynamic#load_plugin --REFLECTS--> __reflect__:app.plugins. (importlib.import_module)
module:src.dynamic#loadPlugin --REFLECTS--> __reflect__:./plugins/${name} (import())
Dynamic module prefixes that belong to the project's own namespace stay in-project; only genuinely foreign packages become external dependencies.
Dependency injection
Dependencies the framework supplies are INJECTS edges instead of missing calls
(find_injections):
| Framework | Detected from |
|---|---|
| Spring | @Autowired fields/methods/constructors |
| NestJS | constructor parameters with decorators; @Injectable() classes |
| FastAPI | Depends(...) parameters |
| dependency-injector | constructor parameters of @injectable classes |
| plain TS/JS | constructor parameters typed with a project class |
Primitive and structural parameter types (string, { id: string }, list[str]) are
counted but produce no edge, so @Body() body: { id: string } does not pollute the graph.
Framework entry points and routes
Methods a framework invokes have no source-level caller, so marking them keeps dead-code
analysis honest and gives trace_execution a root (find_entry_points, find_routes):
| Language | Recognised |
|---|---|
| Java | @GetMapping, @PostMapping, @RequestMapping, @Scheduled, @EventListener, listener annotations, tests, main |
| Python | FastAPI (@router.get, @app.post, @router.websocket), Flask (@bp.route, @app.get), Quart (@bp.route), if __name__ == "__main__", pytest tests |
| TypeScript | NestJS @Controller + @Get/@Post/... , Express app.get('/x', handler), Jest tests |
Express handlers written inline get a synthetic method node
(module:src.routes#<anonymous@7>) so their body still gets call edges, containment, and
a route edge.
Control flow on call edges
Each CALLS edge records whether it is guarded:
{
"kind": "CALLS",
"source": "app.repository.OrderRepository#create",
"target": "ext:builtins",
"properties": { "line": 14, "conditional": true, "controlFlow": ["branch", "loop"],
"guards": ["row.get(\"id\") == order_id", "self.rows"] }
}
controlFlow uses branch | loop | exception | predicate. Guards are source text, not
symbolic constraints: they tell an agent that a call is conditional and what the
condition looked like, without pretending to model its truth value.
Resolution heuristics
Java:
- Constructor calls (
new Foo(...)) → resolveFoo→#<init>(default ctor if implicit). - Bare /
this./super.calls → enclosing type + superclass chain. - Type-qualified calls (
com.acme.Foo.bar) → resolve the type directly. - Instance calls (
x.method()) → infer receiver type from locals → params → fields, then search that type and its ancestors. - Otherwise → phantom
unknown_symbolnode,resolved:false, with the inference failure reason attached to the edge.
Python and TypeScript add module-aware resolution on top of the same pipeline:
- Import-aware scope —
from app.repository import OrderRepositorymaps the local name toapp.repository.OrderRepository; relative and project modules are resolved by suffix, bare specifiers fall back to package roots. self.x/this.xresolve against the enclosing class's fields; a receiver that starts withself/this/cls/supernever becomes an external package.- Dotted receivers (
service.repo.find) are walked segment by segment: the first segment resolves to a type, the next to a field on it, and container members such aslist.appendare recognised as builtin rather than unknown. - Locals are typed from parameters, annotated assignments,
new T(), and inferred loop/variable element types (for row in self.rows:→row: dict). - Module functions (
module:app.api#read_order) participate in dispatch, routes, and entry-point marking exactly like class methods.
Each resolved edge carries a confidence score and a human-readable reason trace
(e.g. via local orders: OrderService), so LLM clients can weigh how much to trust it.
Implemented languages
| Language | Extensions | Extracted | Dynamic features |
|---|---|---|---|
| Java | .java | packages, classes, interfaces, enums, annotations, constructors, methods, fields | virtual dispatch, AspectJ/Spring AOP advice, @Autowired, @RequestMapping family, Class.forName, ServiceLoader |
| Python | .py, .pyi | modules, classes, functions, methods, fields, decorators | @injectable/Depends injection, FastAPI/Flask/Quart routes, getattr, importlib, __main__, tests |
| TypeScript | .ts, .tsx, .mts, .cts | modules, classes, interfaces, type aliases, enums, methods, fields, arrow functions | NestJS/Angular decorators, DI constructor injection, NestJS/Express routes, import(), require() |
| JavaScript | .js, .jsx, .mjs, .cjs | modules, classes, functions, fields | decorators, Express routes, require() |
Crawling a directory mixes all of them automatically — language is chosen per file from
the extension, and a single crawl can return languages: ["java", "python", "typescript"].
Adding a language requires:
src/parser/languages/<lang>.ts— aLanguageDescriptorimplementingparseFile(module/package extraction, type/member extraction, raw call sites, local type tables) plus three declarative fields:resolver(which call resolver handles it),frameworks(what its parser recognises), andfixture(its conformance directory).- One row in
src/parser/languages/index.ts. - One row in the
CONTRACTStable intests/language-conformance.test.ts— the suite then holds the new language to the same graph invariants as every other one. - Optionally a resolver in
src/resolver/exposed throughsrc/resolver/registry.ts; most languages work with the sharedmoduleresolver and only override language-specific builtins. - Any framework it should recognise that is not registered yet — a module in
src/parser/frameworks/declaring its entry points, injections and markers. That is a data change: no parser is edited.
Frameworks are resolved per file by import evidence, then by the descriptor's
frameworks order, then by every registered module. A file that imports
dependency_injector attributes its @inject there, while a Java file with no such
import still falls back to the registry rather than losing the annotation. Parsers keep
what is genuinely syntactic: receiver names for call-style routes, path joining, and the
text recorded as an entry point's detail.
No existing language module needs editing: dispatch in src/crawler.ts reads the
descriptor's resolver, so a language never appears in a conditional.
The graph schema, store, traversal queries, and MCP surface stay the same for every
language. Candidate next languages, and the modelling each one needs, are ranked in
ROADMAP.md Phase 3.
What is not modelled
Language-specific gaps, in addition to the shared ones below:
| Language | Not modelled |
|---|---|
java | A record is indexed as a class with record: true; components become private final fields plus a synthesized accessor each, but the implicit canonical constructor gets no signature until a call resolves to it. Lambda bodies are indexed, but a method reference such as this::handle has no static target and produces no call edge. static {} and instance initialiser blocks are not extracted. Generic type parameters are erased, so a call through a List<T> field binds to the declared bound rather than the concrete argument. |
python | No metaclass, descriptor, __getattr__, or __getattribute__ modelling. Members a decorator generates rather than writes — @dataclass's __init__ and __eq__, anything a metaclass adds — have no declaration to point at, so calls to them stay unresolved. Monkey-patching at import time is not tracked. A lambda stored on an attribute and invoked later has no target. |
typescript | A namespace X {} block produces no module node, so members declared inside one are not reachable by name. JSX parses cleanly but the element tree is not modelled. Type-level constructs — conditional, mapped, template-literal, intersection — are not evaluated, so a receiver typed that way falls back to unresolved. static {} blocks and accessor fields are not extracted. |
javascript | Handled by the TypeScript parser, so every gap above applies. With no annotations available, receivers come only from new, assigned initialisers, and the language-agnostic heuristics — dynamically typed code resolves less often than annotated code does. |
Not yet modelled, by design:
- Value flow and branch feasibility — edges say that a call is conditional and record the guard text, not which branch is taken.
- Cross-repository resolution — libraries outside the crawled tree are summarised as
ext:packages rather than crawled. - Fully dynamic targets (
getattr(obj, name),require(userInput)) point at a__reflect__:marker instead of a symbol; there is nothing static to point at. - Runtime DI graphs assembled from configuration files rather than code.
- Receiver inference from an expression rather than a name —
getService().run(),items[0].run(), a ternary, or a cast — so those calls report as unresolved. - Overload selection uses argument count first, then the type of each argument where
the call site states one (literals,
newexpressions, casts). Two overloads that a call cannot distinguish —f(count)againstf(int)andf(String)— still bind to the first declaration in source order. Overloaded declarations keep distinct ids (Type#f,Type#f(int),Type#<init>(String)) after the first one; seeUSAGE.mdfor the id scheme.
The full list, and what is modelled in exchange for each gap, is in
USAGE.md.
Architecture
src/
├── index.ts CLI (crawl | status | query | serve)
├── server.ts MCP server: tools, resources, prompt (thin handlers)
├── crawler.ts file discovery → parse → index → resolve → persist
├── config.ts env-overridable configuration
├── graph/
│ ├── schema.ts node/edge kinds, id scheme, property shapes
│ ├── store.ts better-sqlite3 schema + idempotent file-scoped upserts
│ └── queries.ts recursive-CTE traversal (callers/callees/path/subgraph)
├── parser/
│ ├── language.ts LanguageConfig / LanguageDescriptor contracts
│ ├── treeSitter.ts WASM runtime + grammar loading + CST walk helpers
│ ├── controlFlow.ts enclosing branch/loop/try/catch + guard text
│ ├── moduleId.ts import-aware module id derivation
│ ├── methodId.ts method id scheme (overload signatures, constructor ids)
│ ├── frameworks/
│ │ ├── types.ts FrameworkModule / FrameworkScope contracts
│ │ ├── index.ts registry + scope/entry-point/injection lookups
│ │ └── <framework>.ts one module per framework (spring, nestjs, fastapi, …)
│ └── languages/
│ ├── index.ts descriptor table + per-file language detection
│ ├── java.ts Java structural extraction
│ ├── python.ts Python structural extraction
│ └── typescript.ts TypeScript, TSX and JavaScript extraction
└── resolver/
├── types.ts TypeIndex (global name/method index) + Resolver contract
├── registry.ts resolver lookup, capability query, default fallback
├── common.ts builtin types, external packages, receiver inference
├── dynamic.ts virtual dispatch + AspectJ pointcut matching/advice edges
├── genericResolver.ts the `module` resolver (imports, barrels, overloads)
└── javaResolver.ts the `java` resolver (declared types, hierarchy, overloads)
Roadmap
Language-agnostic architecture, more languages, and better resolution are tracked in
ROADMAP.md. The near-term focus is making a new language a one-module
change, then using that to add Go, C#, and Kotlin.
Documentation site
The static documentation site in docs/ is deployed to GitHub Pages by
.github/workflows/pages.yml. It covers quickstart, the MCP tool reference, and worked
examples. Open docs/index.html locally or visit
https://nithin554.github.io/code-crawler/.
License
MIT