alchemyincode/plumloom-autoeval

v0.1.0Apache-2.0

Run Plumloom Autoeval evaluations and release gates from a coding agent. Checks setup and validates evaluation files without spending a run, runs an eval or a suite gate, and reports the verdict Autoeval returns.

GitHub Open Website

MCP servers

Declared configuration, as published in mcp.json. The directory shows indexed content; it never connects to or executes these servers.

plumloom-autoevalstdio

{
  "type": "stdio",
  "command": "autoeval-mcp",
  "env": {
    "AUTOEVAL_API_BASE_URL": "https://api.plumloom.ai"
  }
}

What this package declares

The files a client reads when it loads this plugin, exactly as this revision carries them.

{
  "$schema": "https://agent-plugins.org/schemas/1.0.0/plugin.schema.json",
  "name": "plumloom-autoeval",
  "version": "0.1.0",
  "description": "Run Plumloom Autoeval evaluations and release gates from a coding agent. Checks setup and validates evaluation files without spending a run, runs an eval or a suite gate, and reports the verdict Autoeval returns.",
  "author": {
    "name": "Plumloom"
  },
  "homepage": "https://github.com/AlchemyInCode/plumloom-autoeval-oss#readme",
  "repository": "https://github.com/AlchemyInCode/plumloom-autoeval-oss",
  "license": "Apache-2.0",
  "keywords": [
    "autoeval",
    "plumloom",
    "evals",
    "evaluation",
    "llm evaluation",
    "agent evaluation",
    "release gate",
    "ci",
    "mcp",
    "validate evaluation",
    "eval file"
  ],
  "extensions": {
    "com.openai": {
      "interface": {
        "displayName": "Plumloom Autoeval",
        "shortDescription": "Evals and release gates",
        "longDescription": "Run Plumloom Autoeval evaluations and release gates for AI agents and LLM features. The skill checks setup, validates evaluation files without spending a run, runs an evaluation or a suite gate, and reports the PASS, FAIL, INCONCLUSIVE or ERROR verdict exactly as Autoeval returns it. It never edits evaluations or thresholds to change a verdict. Requires the autoeval CLI and a Plumloom CLI key.",
        "developerName": "Plumloom",
        "category": "Developer Tools",
        "websiteURL": "https://github.com/AlchemyInCode/plumloom-autoeval-oss#readme",
        "capabilities": [],
        "defaultPrompt": [
          "Check my Autoeval setup without starting a run.",
          "Run the eval suite for this repo and report the verdict.",
          "Explain why the last release gate was inconclusive."
        ],
        "composerIcon": "./assets/logo.svg",
        "logo": "./assets/logo.svg"
      }
    }
  }
}

What else this package ships

These files come with the package and this site does not publish them. They are listed so you know what is there before you install it.

2other files

View on GitHub

Client extensions

Data this package carries for particular clients. The directory lists the clients named and never reads what is addressed to them.

com.openai