Skip to content

talkops-ai/talkops-prometheus

v0.2.0Apache-2.0

Operational skills and MCP execution server for Prometheus 3.0+, Thanos, Mimir, Cortex, and VictoriaMetrics: PromQL authoring, RED/SLO triage, trace exemplar pivoting, rule simulation & testing, TSDB cardinality FinOps, Kubernetes Operator scrape CRDs, and exporter lifecycle management.


Architecture: Strict Skill & Execution Separation

Following the TalkOps Skill & Execution Separation Architecture (docs/skill-and-execution-separation.md), this repository separates how to think (Agent Skills) from how to act (MCP Execution Layer):

src/prometheus-mcp-server/
├── plugin.json                                     # Agent Plugins v1.0.0 manifest
├── mcp.json                                        # Portable MCP launcher config
├── skills/                                         # 6 Progressive-Disclosure Agent Skills
│   ├── triaging-prometheus-incidents-and-slos/     # RED/USE triage, multi-window SLOs, exemplars
│   ├── authoring-promql-and-analyzing-metrics/     # PromQL 3.0, Native Histograms, UTF-8 syntax
│   ├── authoring-and-testing-prometheus-rules/     # Alerting/recording rules & promtool testing
│   ├── optimizing-tsdb-cardinality-and-finops/     # Cardinality FinOps, relabeling, remote-write
│   ├── configuring-prometheus-scraping-and-discovery/ # ServiceMonitor/PodMonitor/Probe/ScrapeConfig/file_sd
│   └── onboarding-exporters-and-instrumentation/   # OpenMetrics snippets & 19-exporter K8s catalog
└── prometheus_mcp_server/                          # FastMCP Execution Server (19 Tools, 11 Resources)
    ├── tools/                                      # 5 Toolsets (diagnostics, query, rules, scrape, exporters)
    ├── resources/                                  # 11 Dynamic prom:// Resources
    ├── services/                                   # Prometheus HTTP API v1 + Kubernetes CRD merge services
    └── utils/                                      # Redaction, JSONL audit trail, PromQL 3.0 token-boundary AST

Clear Bounded Context (Zero Overlap with Sibling MCP Servers)

  • prometheus-mcp-server strictly owns Prometheus signal generation, PromQL querying (including Native Histograms and UTF-8 names), Exemplar extraction (/api/v1/query_exemplars), TSDB cardinality & health (/api/v1/status/tsdb, /api/v1/status/walreplay), scrape targets (ServiceMonitor, PodMonitor, Probe, ScrapeConfig, file_sd), promtool validation/unit testing, and Prometheus's own /api/v1/rules & /api/v1/alerts rule evaluation state.
  • alertmanager-mcp-server (src/alertmanager-mcp-server) exclusively owns Alertmanager v2 APIs (/api/v2/alerts, /api/v2/silences, routing trees, receivers, and inhibition rules).
  • tempo-mcp-server (src/tempo-mcp-server) owns distributed trace retrieval using trace_id values extracted via prom_query_exemplars.

6 Bundled Agent Skills (skills/)

Skill DirectoryPurposeLocal Deterministic Script (scripts/)
triaging-prometheus-incidents-and-slosIncident triage using RED/USE, Google SRE multi-window multi-burn-rate SLOs, and Exemplar-to-Trace handoffsscripts/generate_slo_rules.py
authoring-promql-and-analyzing-metricsSafe PromQL 3.0 query construction, Native Histograms, and anomaly period comparisonsscripts/lint_promql.py
authoring-and-testing-prometheus-rulesAlerting & recording rule lifecycle, promtool unit tests, and safe multi-group CRD mergesscripts/build_promtool_test_suite.py
optimizing-tsdb-cardinality-and-finopsTSDB cardinality investigation, metric_relabel_configs, and remote-write FinOpsscripts/generate_finops_config.py
configuring-prometheus-scraping-and-discoveryKubernetes Operator CRDs (ServiceMonitor, PodMonitor, Probe, ScrapeConfig) and VM file_sdscripts/validate_scrape_manifest.py
onboarding-exporters-and-instrumentationLanguage SDK instrumentation (Go/Python/Java/Node) and 19-exporter K8s lifecyclescripts/render_exporter_bundle.py

19 MCP Tools Across 5 Toolsets

Every tool has $\le 8$ parameters and explicit MCP ToolAnnotations (readOnlyHint, destructiveHint, idempotentHint, openWorldHint).

1. diagnostics Toolset (2 Read-Only Tools)

ToolDescription
prom_get_diagnosticsConsolidated control-plane & TSDB diagnostics (overview, runtime_config, tsdb, wal_replay, scrape_pools).
prom_analyze_cardinalityLive TSDB cardinality analysis combining /api/v1/status/tsdb with per-metric label fanout via /api/v1/series.

2. query Toolset (7 Read-Only Tools)

ToolDescription
prom_explore_metricsUnified discovery of metric names, metadata (TYPE/HELP/UNIT), series matches, or label values.
prom_query_instantExecute point-in-time PromQL query (or validate_only=True syntax/counter preflight check).
prom_query_rangeExecute time-range PromQL query with automatic downsampling and summary/full/chart response modes.
prom_query_exemplarsQuery /api/v1/query_exemplars to extract OpenTelemetry trace_id/span_id exemplars for tempo-mcp-server.
prom_compare_periodsCompare current range query against a baseline offset (e.g. 1d, 7d) with delta% and z-score shift.
prom_analyze_service_redCompound RED (Rate, Error ratio, Latency p50/p95/p99) analysis supporting classic and Native Histograms.
prom_evaluate_slo_burnCompound Google SRE multi-window multi-burn-rate SLO evaluation (1h/5m, 6h/30m, 24h/2h, 72h/6h).

3. rules Toolset (3 Read-Only, 1 Write, 1 Destructive)

ToolDescription
prom_get_firing_alertsInspect active (pending/firing) alerts from Prometheus /api/v1/alerts & /api/v1/rules.
prom_validate_rulesValidate rule group YAML via promtool check rules and optional promtool test rules unit tests.
prom_simulate_rule_firingBacktest an alert expression against historical range data (mode="historical") or synthetic series (mode="synthetic").
prom_upsert_rule_groupCreate/update a rule group via yaml_only, local_file, k8s_crd (non-destructive group merge), or http_ruler.
prom_delete_rule_groupDelete a rule group (preserves sibling groups in PrometheusRule CRDs; deletes CRD only when last group removed).

4. scrape Toolset (1 Read-Only, 1 Write, 1 Destructive)

ToolDescription
prom_verify_metrics_endpointScrape and validate a /metrics endpoint + optional Prometheus up{} series verification.
prom_apply_scrape_configApply servicemonitor, podmonitor, probe, scrapeconfig, or file_sd target configuration.
prom_delete_scrape_configDelete a Kubernetes scrape CRD or remove a file_sd job entry.

5. exporters Toolset (1 Write, 1 Destructive)

ToolDescription
prom_install_exporterDeploy one of 19 supported exporters to Kubernetes (dry_run=True by default).
prom_uninstall_exporterUninstall a previously deployed exporter from Kubernetes (confirm=True required).

11 Dynamic prom:// Resources

Resource URIDescription
prom://system/backendsAll configured Prometheus backends with health status
prom://system/backends/{backend_id}Detailed capabilities, version, and build info for a single backend
prom://config/runtimeSecret-redacted runtime configuration, flags, and TSDB summary
prom://topology/targetsConsolidated service topology and failed/down scrape targets
prom://metadata/catalogLive metric catalog (TYPE, HELP, UNIT)
prom://tsdb/cardinalityLive TSDB cardinality overview from /api/v1/status/tsdb
prom://rules/groupsSummary of all alerting and recording rule groups
prom://rules/{backend_id}/groups/{group_name}Full YAML/JSON definition and evaluation health of a specific rule group
prom://kubernetes/scrape-crdsInventory of ServiceMonitor, PodMonitor, Probe, ScrapeConfig, and PrometheusRule CRDs
prom://kubernetes/manifests/{kind}/{namespace}/{name}Secret-redacted Kubernetes CRD manifest
prom://audit/recentRecent mutating action audit log entries (ring buffer + optional JSONL persistence)

CLI Flags & Environment Configuration

# Default (Read-Only safe mode: mutating tools are hidden unless --allow-write is set)
uv run prometheus-mcp-server --prometheus-url http://localhost:9090

# Enable write tools and restrict to specific toolsets
uv run prometheus-mcp-server \
  --prometheus-url http://localhost:9090 \
  --allow-write \
  --toolsets diagnostics,query,rules \
  --tenant-id team-payments \
  --audit-log-path ./audit.jsonl
Environment VariableCLI FlagDefaultDescription
PROMETHEUS_BASE_URL--prometheus-urlhttp://localhost:9090Prometheus HTTP API base URL
PROMETHEUS_ALLOW_WRITE--allow-write / --read-onlyfalseRegister mutating/destructive tools (false = read-only)
PROMETHEUS_TOOLSETS--toolsetsallComma-separated toolsets (diagnostics,query,rules,scrape,exporters)
PROMETHEUS_TENANT_ID--tenant-id(none)Default X-Scope-OrgID header for Mimir/Cortex/Thanos
PROMETHEUS_RULER_URL—(none)Optional Mimir/Thanos Ruler base URL
PROMETHEUS_AUDIT_LOG_PATH--audit-log-path(none)Optional JSONL file path for persistent mutation audit trail

Running Tests

The repository includes a 3-tier verification suite (115 tests across unit, integration, and skill/plugin compliance):

cd src/prometheus-mcp-server
uv run pytest -v

License

Apache 2.0 — see LICENSE.