Skip to content

talkops-ai/talkops-loki

v0.2.0Apache-2.0

Operational Agent Skills and MCP execution server for Grafana Loki 3.x and Loki Operator (loki.grafana.com/v1): discovery-first LogQL authoring, structured metadata & Bloom filter optimization, service error triage, release regression comparison, cross-pillar trace correlation, FinOps volume attribution, Loki Ruler alerting/recording rules, Compactor log deletion, and Kubernetes LokiStack management.

Following the TalkOps Skill & Execution Layer Separation architecture:

  • Procedural Intelligence (skills/): 6 modular Agent Skills containing progressive-disclosure SOPs (SKILL.md $\le 250$ lines), deep domain references (references/*.md), and deterministic local Python CLI helpers (scripts/*.py).
  • Execution Plane (loki_mcp_server/): 18 bounded, type-annotated MCP tools across 7 selectable toolsets + 9 live loki:// state resources, with read-only defaults (MCP_ALLOW_WRITE=false), mandatory dry_run=True previews, and interactive ctx.elicit() / confirm=True mutation gates.

Architecture Overview

src/loki-mcp-server/
├── plugin.json                              # Agent Plugins v1.0.0 manifest (talkops-loki v0.2.0)
├── mcp.json                                 # Default stdio MCP server wiring (read-only by default)
├── skills/                                  # 6 Modular Agent Skills (Procedural Layer)
│   ├── authoring-logql-and-analyzing-logs/
│   ├── triaging-loki-incidents-and-errors/
│   ├── discovering-loki-schema-and-log-structure/
│   ├── optimizing-loki-cardinality-and-finops/
│   ├── authoring-loki-alerting-and-recording-rules/
│   └── operating-loki-deployments-and-ingestion/
└── loki_mcp_server/                         # FastMCP Execution Plane
    ├── config.py                            # Frozen config + CLI flags (--read-only, --allow-write, --toolsets)
    ├── services/
    │   ├── loki_service.py                  # Async HTTP client (Query, Volume, Ruler, Compactor, Diagnostics)
    │   └── kubernetes_service.py            # Async K8s client (LokiStack, RulerConfig, AlertingRule, RecordingRule)
    ├── resources/
    │   └── loki_resources.py                # 9 live dynamic loki:// resources (zero static markdown bloat)
    └── tools/                               # 18 tools across 7 selectable toolsets (all <= 8 params)

6 Bundled Agent Skills (skills/)

Skill NamePurposeLocal Deterministic Script
authoring-logql-and-analyzing-logs4-stage LogQL pipeline construction, cost preflight checks, metric unwrap queries, and A2UI visualizationscripts/lint_logql.py
triaging-loki-incidents-and-errors$N \to 1$ service log summarization, period-over-period regression diffs, and cross-service trace_id correlationscripts/extract_error_signatures.py
discovering-loki-schema-and-log-structureProgressive schema discovery (labels $\to$ values $\to$ series $\to$ detected_fields $\to$ patterns) and OTLP mappingscripts/infer_log_parser.py
optimizing-loki-cardinality-and-finopsLog volume attribution, high-cardinality label detection, Alloy/Promtail structured_metadata migration, and Bloom filtersscripts/audit_label_cardinality.py
authoring-loki-alerting-and-recording-rulesNoise-free Loki Ruler alerting/recording rules, backtesting, and Kubernetes AlertingRule/RecordingRule CRDsscripts/validate_loki_rules.py
operating-loki-deployments-and-ingestionloki_discarded_samples_total diagnostics, Compactor PII log deletions, and Loki 3.x / LokiStack sizingscripts/render_loki_architecture_config.py

18 MCP Tools Across 7 Selectable Toolsets

All 18 tools follow the loki_* namespace convention across 7 selectable toolsets ($\le 8$ parameters per tool):

ToolsetTool NameModeDescription
discoveryloki_get_labelsRead-OnlyList indexed label names (service_name, namespace, cluster)
discoveryloki_get_label_valuesRead-OnlyList values for a specific label, optionally scoped by query
discoveryloki_get_seriesRead-OnlyInspect active label combinations and stream count
discoveryloki_get_log_patternsRead-OnlyDiscover recurring <_> structural templates and line volumes
discoveryloki_get_detected_fieldsRead-OnlyDiscover JSON/logfmt keys and Loki 3.x structured metadata fields
queryloki_get_query_statsRead-OnlyPreflight query cost check (streams, chunks, entries, bytes)
queryloki_query_instantRead-OnlyExecute point-in-time LogQL metric queries
queryloki_query_rangeRead-OnlyExecute range LogQL log/metric queries with size and cost guardrails
queryloki_query_a2uiRead-OnlyExecute LogQL and render an interactive A2UI dashboard component
analysisloki_summarize_service_logsRead-Only$N \to 1$ service digest: volume, error ratio, top signatures, sample errors, and trace_ids
analysisloki_compare_periodsRead-OnlyCompare current vs. baseline window to isolate newly introduced error signatures
analysisloki_correlate_trace_logsRead-OnlyReconstruct chronological cross-service log timelines by trace_id / request_id
analysisloki_analyze_volumeRead-OnlyAttribute ingested log bytes by label dimension (/loki/api/v1/index/volume + fallback)
diagnosticsloki_get_diagnosticsRead-OnlyInspect /ready, /loki/api/v1/status/buildinfo, and /metrics loki_discarded_samples_total
rulesloki_upsert_rule_groupWrite-GatedValidate, diff (dry_run=True), and upsert a Loki Ruler rule group
rulesloki_delete_rule_groupWrite/DestructivePreview (dry_run=True) and delete a Loki Ruler rule group
governanceloki_manage_delete_requestsWrite/DestructivePreview blast radius (dry_run=True), create, or cancel Compactor /loki/api/v1/delete requests
operatorloki_apply_operator_crWrite-GatedServer-side dry-run diff and apply for LokiStack, RulerConfig, AlertingRule, and RecordingRule CRs

9 Dynamic loki:// Resources

Resource URIDescription
loki://server/capabilitiesServer version, active toolsets, allow_write gate, guardrails, and Loki 3.x feature flags
loki://system/healthLive Loki reachability and readiness status
loki://schema/labelsLive indexed label inventory
loki://config/guardrailsQuery byte, time window, line limit, and cardinality thresholds
loki://config/backendsRedacted Loki backend URL, timeout, tenant (X-Scope-OrgID), and auth mode
loki://rules/groupsActive Loki Ruler alerting and recording rule groups across namespaces
loki://compactor/delete-requestsActive and completed Compactor log deletion requests
loki://operator/crs/{namespace}Inventory of Kubernetes Loki Operator custom resources in {namespace}
loki://operator/manifests/{kind}/{namespace}/{name}Sanitized, secret-redacted Kubernetes manifest for a Loki Operator CR

CLI Flags & Environment Configuration

# Default: Read-only mode (14 read-only tools registered, write tools hidden)
loki-mcp-server --read-only

# Enable write/governance/operator tools (18 tools registered; still guarded by dry_run=True & confirm=True)
loki-mcp-server --allow-write

# Register only specific toolsets
loki-mcp-server --toolsets discovery,query,analysis,diagnostics
Environment VariableDefaultDescription
LOKI_URLhttp://localhost:3100Base URL of the Grafana Loki instance
LOKI_ORG_ID(optional)Multi-tenant X-Scope-OrgID header
LOKI_AUTH_TOKEN(optional)Bearer token for Loki authentication
MCP_ALLOW_WRITEfalseSet true (or pass --allow-write) to register write/destructive tools
MCP_TOOLSETSallComma-separated subset of discovery,query,analysis,diagnostics,rules,governance,operator
KUBERNETES_ENABLEDfalseEnable live Kubernetes API queries for Loki Operator CRDs

Running Tests

pytest