Skip to content

vasiache/agent-run-metrics

v0.0.0

agent-run-metrics

Counters for AI coding agents: how many tokens a session burned, which tools it ran, how long it took. The data lands in your own monitoring (Prometheus PushGateway, VictoriaMetrics, Grafana) and stays on your infrastructure.

Supported agents: OpenCode (V1 plugin API — 1.18.x and 1.19.x distributions) and Claude Code. Others are planned; see docs/adapter-template.md for the adapter contract.

Repository layout

├── integrations/            one folder per agent
│   ├── common/              host-neutral core: config, token, PushGateway,
│   │                        session aggregation, delivery queue, report helper
│   ├── opencode/            plugin.js (V1 adapter, live counters) + run_timeline.py
│   └── claude-code/         install.mjs + push-metrics.mjs + session_timeline.py
├── contract-tests/          adapter contract suites (event / snapshot styles)
├── dist/                    built, self-contained artifacts (npm run build)
├── tests/                   unit + artifact tests
├── dashboards/              Grafana dashboard
├── alerts/vmalert.yml       anomaly rules (token burst, slow feedback)
└── deploy/                  local monitoring stack in docker compose

Quick start

  1. Start the monitoring stack:
cd deploy && docker compose up -d

Grafana opens at http://localhost:3000. The "Agent run cost" dashboard is already loaded against the provisioned VictoriaMetrics datasource. Metrics are received on port 9091.

  1. Connect your agent (next two sections).

OpenCode

Copy the self-contained plugin into the project, or into ~/.config/opencode/plugin/ to cover all projects:

npm run build   # or use the checked-in dist/opencode/v1/plugin.js
mkdir -p <project>/.opencode/plugin
cp dist/opencode/v1/plugin.js <project>/.opencode/plugin/metrics.js
export AGENT_METRICS_PUSHGATEWAY=http://localhost:9091

OpenCode V1 also accepts per-plugin options (the "plugin": [[name, options]] tuple in opencode.json); explicit options win over environment variables:

{ "plugin": [["./metrics.js", { "endpoint": "http://localhost:9091", "labels": { "user": "alice" } }]] }

Install via npm

The repository is an npm-installable OpenCode plugin: the package entry point is the built artifact, so an npm delivery mechanism can install and load it without any local file copying.

npm install agent-run-metrics
{ "plugin": [["agent-run-metrics", { "endpoint": "http://localhost:9091" }]] }

Config file

npm-installed plugins receive no options, so the OpenCode integration also reads a JSON settings file from the host's configuration root — the first existing file wins:

  1. $OPENCODE_CONFIG_DIR/agent-metrics.json
  2. ~/.config/opencode/agent-metrics.json
  3. ~/.opencode/agent-metrics.json
{
  "endpoint": "http://localhost:9091",
  "tokenFile": "~/gateway.token",
  "labels": { "user": "alice" }
}

The file understands the same fields as the options table above. Precedence per field: plugin options, then AGENT_METRICS_* environment variables, then this file, then defaults — a local override beats the deployment-provided file. A malformed file disables delivery with a single configuration error.

Report for a finished session:

python3 integrations/opencode/run_timeline.py --list
python3 integrations/opencode/run_timeline.py --session <id> \
    --push-gateway http://localhost:9091 --label user=alice

The report reads the local OpenCode database read-only and writes run.md plus run.timeline.csv. It can also answer "how much time passed between the code edit and the test result" (--gate-regex, --watch).

Claude Code

One command: it copies the built hook artifact, adds the hooks and the gateway env to ~/.claude/settings.json (idempotent, existing hooks are preserved), checks the pushgateway and runs a self-test:

npm run build   # refresh dist/claude-code/push-metrics.mjs first
node integrations/claude-code/install.mjs
# or: node integrations/claude-code/install.mjs --push-gateway http://host:9091
# undo: node integrations/claude-code/install.mjs --uninstall

Manual install (fallback): copy dist/claude-code/push-metrics.mjs to ~/.claude/hooks/agent-metrics/ and merge settings.example.json into ~/.claude/settings.json. Restart Claude Code (or open /hooks) afterwards.

Report for a finished session:

python3 integrations/claude-code/session_timeline.py --list
python3 integrations/claude-code/session_timeline.py --session <id> \
    --push-gateway http://localhost:9091 --label user=alice

Configuration (live counters)

Explicit plugin options win over environment variables.

Option / VariableDefaultMeaning
endpoint / AGENT_METRICS_PUSHGATEWAYrequiredPushGateway URL, used verbatim (tenant or path routing is part of the URL the caller passes), e.g. https://gateway.example/push. Without it the integration stays idle and performs no I/O
job / AGENT_METRICS_JOBopencode / claude-codejob label
runtime / AGENT_METRICS_RUNTIMEruntime defaultruntime label on every series
instance / AGENT_METRICS_INSTANCEhostnameinstance label
labels / AGENT_METRICS_LABELSnoneextra labels, k=v,k2=v2; the names runtime, session, project, instance, job, kind, tool, skill, plugin, agent are reserved
token / AGENT_METRICS_TOKENnonebearer token
tokenFile / AGENT_METRICS_TOKEN_FILEnoneraw token file, trimmed, reread per push (rotation applies on the next delivery); a missing, empty, or unreadable explicitly configured file fails that push
debug / AGENT_METRICS_DEBUGoffconcise failure logging to stderr; never logs credentials

The user label defaults to the OS username. Previous implicit deployment-specific identity and grant-discovery fallbacks (vendor telemetry and token files) were removed from the generic integrations; configure them via labels or a token file instead.

Metrics reference

Both runtimes emit the same families; runtime (opencode / claude-code) tells them apart. Live: agent_tokens_total{runtime,kind}, agent_tool_calls_total{runtime,tool}, agent_skill_calls_total{runtime,skill}, agent_plugin_calls_total{runtime,plugin}, agent_context_max_tokens{runtime}, agent_run_duration_seconds{runtime}. Skills and plugins come from Skill tool calls in the Claude Code transcript (plugin:skill names feed the plugin family); every plugin is counted, including the metrics integration itself.

Edit and permission families appear when the host provides the data (graceful degradation): agent_file_edits_total{runtime,kind} counts approximate added and deleted lines (OpenCode: unified diff from edit tool metadata; Claude Code: LCS delta of old_string/new_string from the transcript); agent_permission_decisions_total{runtime,decision} counts permission outcomes (allow / deny) — a permission decision is not "accepted code" (OpenCode only: Claude Code hooks expose no deny signal, so it emits no permission family); agent_tool_duration_seconds_total{runtime,tool} sums wall-clock tool execution time (OpenCode pairs tool.execute.before/after by call id; Claude Code pairs transcript tool_use/tool_result timestamps).

agent_context_max_tokens is the maximum observed prompt usage per session (input + cache_read + cache_write), not the model's context-window capacity. OpenCode derives it from usage deltas; Claude Code from transcript snapshots.

Subagent runs (Claude Code) are counted separately, per agent type (Explore, Plan, ...): agent_subagent_spawns_total{agent}, agent_subagent_tokens_total{agent,kind}, agent_subagent_tool_calls_total{agent,tool}, agent_subagent_context_max_tokens{agent}, agent_subagent_duration_seconds{agent}. The hook reads them from <session>/subagents/ transcript files next to the main transcript; runs of the same agent type merge into one series. The main-session series do not include subagent work, so a session's full spend is main + subagent series.

Parallel sessions are separated, not merged: every live series carries session (the full session id) and project (basename of the working directory), so two concurrent runs on one host get their own grouping keys instead of overwriting each other. Delivery is serialized per session: one active request, one trailing request with the newest totals, a 15-second throttle, and an idle flush that never stacks a second concurrent request. Subagent series carry the parent session's session label. Session groups expire from the dashboard's "Active sessions" panel 5 minutes after their last push. Alerts aggregate over sessions (sum by (instance, user)).

Per-session summary (both report scripts): agent_session_tokens_total, agent_session_turns, agent_session_edits_total, agent_session_tool_calls_total, agent_session_duration_seconds, all with a runtime label. agent_session_cost_total comes from OpenCode only (its DB tracks cost); Claude Code transcripts carry no pricing, so cost needs a pricing table first. The OpenCode analyzer emits agent_edit_to_gate_seconds{gate} when you pass --gate-regex and --watch.

Compatibility

  • OpenCode V1 plugin API (MetricsPlugin(input, options)) — supported: 1.18.x and 1.19.x distributions. The next-generation OpenCode plugin API is pending feasibility: its current dev documentation does not expose the usage snapshots and completed-tool events this project needs.
  • DeepSeekCode, KiloCode, QwenCode — not started; the adapter contract and contract suites are in docs/adapter-template.md.
  • Artifacts are built with esbuild (npm run build); npm run check:artifacts verifies regeneration produces no diff.

License

Apache-2.0