Changelog

All notable changes to this project will be documented in this file.

The format is based on Keep a Changelog, and this project adheres to Semantic Versioning.

[3.9.0] - 2026-10-08

Added

• orq-shared/resources/trace-queries.md §5.4: orq telemetry query | list-capabilities | list-facet-values, served by API 4.16 and exposed by the CLI 11.5.0 line (verified on 11.5.0-rc.6; 11.4.0 has no telemetry command). Covers the required --source and window, the --compute metric/op pairs, <metric>:<op> result keys, the --limit semantics, list-capabilities as the discovery probe, and worked examples, with a fallback to orq reporting query on older binaries. Live-probed against production API 4.16 on 2026-10-08.

Changed

• orq-shared/resources/trace-queries.md §5: the orq reporting query contract now carries the full flag surface (--metric, --from/--to, --grain, --mode, --group-by, --filters, --sort, --time-zone, --include-totals, --limit, -o json), the timeseries-vs-scalar split, and worked examples for CI and cron. It also records that --limit bounds rows in both modes (default 1000, maximum 5000, larger values return HTTP 400), that reporting filters[].field is a closed 35-value enum with op limited to eq/neq/in/not_in and does not share the trace field vocabulary, that jq masks a rejected request unless pipefail is set, and that help-input is a root command (orq reporting query help-input is parsed as a body fragment). Read from the generated command surface at CLI 10.3.0 and re-probed on CLI 11.5.0-rc.6 against API 4.16.
• orq-shared/resources/trace-queries.md §2: --from / --to are no longer described as RFC3339-only. The flag form accepts relative values (7d, now-24h, now); a body read from --from-file or stdin is sent as written and must carry RFC3339 from/to, so the window belongs on the flags.
• orq-cli skill: its orq-shared companion entry now routes metrics questions ("what did we spend last week", "p95 latency", "evaluator pass rate") straight to §5, including §5.4 for orq telemetry. resources/command-map.md no longer says there is no telemetry group.
• orq-shared skill: orq-cli added to the consumer list for resources/trace-queries.md.
• tests/skills.md: smoke scenario 6c accepts either orq reporting query or a probed orq telemetry query, and scenario 8 states the pipefail rationale correctly (the CLI exits 1 on rejection; jq exits 0 on empty input).

Fixed

• skills-factual workflow: the offline step collected all of tests/scripts, including test_run_evals.py from 3.8.0, which imports evaluatorq and failed with ModuleNotFoundError. It now runs test_factual_scripts.py only; skills-ci.yml already runs test_run_evals.py with its pinned evaluatorq.

[3.8.0] - 2026-10-06

Added

• Skill invocation and first-step behavioural evals through orq launch, with reviewed YAML cases, a maintainer skill-tests workflow, per-run costs, and merged factual/eval reports.

Fixed

• Eval runs require container isolation unless host-file access is explicitly accepted; container copies preserve symlinks instead of copying host targets. Missing agent results and empty merged reports cannot report a clean batch, concurrent runs reserve their estimated cost, documentation URL errors stay advisory, and server permission errors cannot mask out-of-allowlist tool calls.

[3.7.2] - 2026-10-06

Changed

• orq-cli, orq-shared, orq-analyze-traces, orq-improve-agent, orq-recommend-evaluators, tests: orq traces thread is orq traces conversation again, short alias conv, to match orq CLI 11.4.0 (orq-cli#122). That release keeps thread as a hidden, deprecated spelling that warns on stderr, and its xml render frames the turns in <conversation …> under either name; flags and the -o json schema are unchanged. Every call site moved: the "Reading a conversation" section in orq-cli/SKILL.md, the command tree, global -o and ORQ_OUTPUT_FORMAT rows and per-trace read list in orq-cli/resources/command-map.md, the full-content bullet and layer 3b in orq-shared/resources/trace-queries.md, the trace-reading steps in orq-recommend-evaluators, scenario 9a in tests/skills.md, the factual CSV rows, and the allowed-tools entries, which carry both Bash(orq traces conversation:*) and Bash(orq traces conv:*). orq-cli/SKILL.md now says which release has which name: conversation from 11.4.0 (thread deprecated there), thread only on 10.0.0 through 11.3.x.

Fixed

• orq-cli skill: the conversation command's flags now match CLI 11.0.0+. --tool-max-chars is documented; the [omitted: N characters] stub is listed with the other markers, with its -o json shape; -i and -x are stated as not combinable; and --max-chars's 4000 default is scoped to the xml and markdown renders, since -o json|yaml|toon are uncut unless a cap is passed. orq-recommend-evaluators' reading-traces.md said the same 4000 default applied to its -o json read.
• evaluatorq contract test: test_datapoint_parallelism_still_resolves_to_ten read default=10 from the source of evaluatorq(). evaluatorq 1.57.0 moved that value into DEFAULT_DATAPOINT_PARALLELISM, so the test failed while the default was still 10. It now checks the value the call actually resolves to.

[3.7.1] - 2026-10-01

Added

• A factual drift tester for documented MCP tools and arguments, SDK imports and methods, CLI commands and flags, documentation URLs, repositories, and packages across the skills. Reviewed CSV fixtures, a nightly and pull request workflow, and a warning for skills without a fixture make drift visible without making the check required.

Fixed

• orq-manage-skills skill: the delete-safety reference scan in resources/known-caveats.md called list_skills(paginated=True), a parameter the MCP tool does not have (it takes only limit, starting_after, ending_before). It is now the explicit cursor loop the skill's Pagination section already documents.
• orq-manage-skills skill: the same scan called search_entities() with no arguments, which the live MCP schema rejects now that type is required. It now queries prompt, deployment and agent separately and pages each to completion; an unpaged call stopped at 50 results per type, so a large workspace got a partial scan that could wrongly clear a Skill for hard delete. SKILL.md Phase 5 and resources/governance-guide.md described the same unpaged scan; both now point at the one procedure in known-caveats.md. Both paging loops (this scan and the SKILL.md Pagination loop) now stop on an empty page instead of reading a cursor from it, and the _id cursor field names where it was read from, since the factual tester cannot check response fields. The scan now reads each prompt's body with orq prompts retrieve <id> instead of from the search_entities result, so it needs the orq CLI; a cli_subcommand row checks the command.
• orq-invoke-deployment skill: four dead doc links. Invoke and Stream now point at /reference/deployments/invoke and /reference/deployments/stream; the Python and Node.js SDK pages are gone, so the SDK line links the single /reference/client-libraries page plus the orq-python and orq-node repos.
• orq-cli skill: orq traces thread documented --reasoning=false, which CLI 11.0 removed. It now documents -x/--exclude (-x reasoning drops the thinking), the flag that replaced it.

[3.7.0] - 2026-10-01

Added

• orq-evaluator-alignment jury mode (RES-1638): the audited judge and any extra panel_models score the same rows with independent repetitions. A three-model panel defaults to three repetitions per model; one-model runs retain their existing repeat count. The cost estimate includes every model call.
• metrics.json records per-model wobble, cross-model disagreement, ties, typed abstentions, failed or unmeasurable votes, and per-model correctness where dataset labels are valid. The review queue ranks abstentions, ties, disagreement with wobble, disagreement, then wobble, with stable controls held separately.
• retest_metrics.json reports each panel model's agreement with human labels on its scored rows, including per-example numeric tolerance bands, allowing rare-label recall and balanced accuracy to expose a useful model hidden by majority voting.

Changed

• The alignment workflow asks for panel models before the cost gate and diagnoses model capability from the panel before another prompt rewrite. Failed judge calls are excluded from annotation priority.

Fixed

• orq-evaluator-alignment converts evaluatorq's per-repetition objects to primitive verdicts before writing stability artifacts and counting off-contract answers; fresh evaluatorq releases no longer break live jury runs.

[3.6.0] - 2026-09-30

Learnings from the PyData 2026 evaluation-flywheel run (RES-1565) folded into orq-evaluator-alignment.

Added

• orq-evaluator-alignment: lib/agreement.py reports balanced_accuracy and cohen_kappa for boolean and categorical judges. metrics.py's correctness block and retest_metrics.json's agreement block carry both. On skewed labels (27 passes, 3 fails, judged pass every time) accuracy reads 90% while kappa is 0, so the metrics report and a new retest caveat flag high accuracy with kappa under 0.2.
• orq-evaluator-alignment step 8: a "What to do with the result" gate. When the new judge got steadier but is still steadily wrong, re-run the retest with a stronger model on a copy of the run before offering another rewrite, and stop iterating if no model reproduces the user's answer. When instability rises after a rule is encoded but agreement did not regress, treat it as a sharper next question rather than a failed rewrite.
• orq-evaluator-alignment rewrite meta-prompt: checks the verdict depends on are written as obligations, not permissions. In the PyData run a "you may compare claims with the tool results" rule was skipped by two of three judges.

Changed

• evaluatorq and orq-compare-agents: add TraceInput replay guidance for evaluatorq 1.47.1 and cover it in the inference contract test.
• orq-evaluator-alignment: SKILL.md is a 142-line router (was 777); each step's instructions live in resources/ (judge-and-examples.md, measure.md, grey-zone.md, rewrite-and-create.md, retest.md) and are read when that step starts. Configuration and the run-directory contract moved to resources/configuration.md and resources/run-directory.md. The step text itself is unchanged apart from the additions above.
• orq-evaluator-alignment: step 4, step 8 and the final summary lead with how many of the rare label the judge caught before quoting overall accuracy.
• orq-evaluator-alignment annotation UI: the judge's vote spread moved into the collapsed "votes and reasoning" panel, so neither anchors a first-pass label.

[3.5.0] - 2026-09-24

Added

• orq-recommend-evaluators (RES-1560): recommends the evaluators an agent or deployment is missing and creates the ones the user approves. With an error-analysis-*.md or 20+ traces in the last 14 days it grounds candidates in trace evidence; below that it runs config-only, reading each hard rule in the instructions and each config signal (knowledge bases, response_format, tools) as a candidate criterion. It skips evaluators already in settings.evaluators / settings.guardrails, offers project evaluators as reuse, sends specification gaps to orq-improve-agent, and writes eval-recommendations-<key>-<timestamp>.md. Create and attach are separate approval gates; each created evaluator is smoke-invoked once, and LLM judges are labelled unvalidated with a pointer to orq-evaluator-alignment. Contract verified against orq CLI on 2026-09-14: settings.evaluators[] entries are {id, execute_on, sample_rate} and the keys are absent (not empty) when nothing is attached; orq evals all returns llm_eval, python_eval, function_eval and ragas types. Re-probed on CLI 10.3.1 on 2026-09-24: evals all --project-id answers 404 for every project (so project scoping is client side), and the conversation subcommand is traces thread; the evals quirks now live in orq-cli's command map.
• orq-recommend-evaluators data model (RES-1589): the skill also writes eval-recommendations-<key>-<timestamp>.json, defined by resources/evaluations.schema.json. It is Bauke's {"evaluations": [{"name"}]} shape with the fields a caller needs to create or attach each one: description, type, existing_evaluator_id (null for a new evaluator), execute_on, priority, reason and caveats, plus schema_version, target and grounding (what the list rests on: config-only, traces or error-analysis, with a reason). At most 5 items is the skill's own cap, so a promoted recommendation cannot invalidate the file; the schema takes any length, in rank order, with no other keys. The skill validates the file it writes with ajv before presenting it, and CI checks six valid and nineteen invalid fixtures against the schema, each file on its own so an empty fixture directory fails the run.
• orq-recommend-evaluators layout: SKILL.md keeps the workflow (about 2,800 words, down from 7,700) and loads step detail from resources/ when a step starts: reading-traces.md, candidate-signals.md, ranking.md, matching.md, output-files.md, create-and-attach.md. The description now lists trigger conditions only. The skill stops on a 404 or 401 from orq agents retrieve instead of recommending without a config, and scopes a multi-agent target to the coordinator's own rules, pointing at each sub-agent separately.

Fixed

• evaluatorq skill: inference now defaults to None upstream (1.47.0) and resolves from data, False for a replay source and True otherwise, so the contract suite asserted a literal default that no longer exists and turned skill-tests red on every PR. The test asserts the resolved value instead, and SKILL.md / resources/inputs-and-data.md no longer say ExperimentInput requires inference=False.

[3.4.0] - 2026-09-21

Added

• orq-compare-agents skill: a gotcha for the TypeScript install — @orq-ai/evaluatorq 1.3.2 declares an optional peer on @orq-ai/node ^3.9.26, so installing both resolves the node SDK to 3.x, where evals.invoke() does not exist. Found by the new type-check fixture on its first run.
• orq-compare-agents skill: tests/ts/contract.ts — a type-only fixture pinning the TypeScript claims in resources/evaluatorq-api.md (the evaluatorq() signature and return type, the { datasetId } input, the invokeEvaluatorRequest key, the flat response). A new ts-contract-tests CI job type-checks it with tsc --noEmit against unpinned packages; the compiler is the right oracle because the risk is a rename.
• tests/scripts/validate-skills.mjs check 12: no skill markdown may name a hardcoded reasoning-effort value. The ladder is the model's and changes per release, and an unsupported value is dropped silently rather than raised. none / off pass — they are the spelling for sending no reasoning parameter at all. Negative tests in validate-skills.test.sh cover all three cases.
• evaluatorq skill: tests/test_documented_api.py — a contract suite asserting every symbol, signature, default and response field the skill teaches, against unpinned evaluatorq / orq-ai-sdk. The existing skill-tests CI job discovers it via tests/requirements.txt, so an upstream rename fails a PR instead of shipping a stale signature into generated code.
• create-skill skill: Phase 4b — when to pin a skill's API claims in an executable suite, when not to, and the rules that keep one useful (leave the documented packages unpinned, assert only what the markdown claims, make no network call).

Changed

• evaluatorq skill: reasoning-effort guidance lives in resources/tuning.md only; SKILL.md carries a pointer instead of a second copy of the knob table.
• evaluatorq skill: no effort value is named anywhere in the skill — the accepted ladder is per model and changes per release, so the examples read it from the catalogue (get_model_info) and pre-validate with validate_reasoning_effort(). validate-skills.mjs check 12 enforces this across every skill.
• orq-invoke-deployment: resources/api-reference.md no longer lists the reasoning_effort rungs; it points at GET /v2/models instead.
• The three new evaluatorq resource files carry a Probed against version stamp, matching the convention in orq-cli and evaluatorq-api.md.
• evaluatorq skill: the judge-template namespace note now warns about names shared with the orq platform that resolve with different contents, not only the names unique to evaluatorq — the shared ones are the silent failure.

[3.3.0] - 2026-09-21

Added

• evaluatorq skill: resources/inputs-and-data.md — the four shapes data accepts (inline rows, awaitables, DatasetIdInput, ExperimentInput), inference=False experiment replay, turning production traces into datapoints via the simulation helpers, the full result object, the raw-dict job error contract, and check_pass_failures CI gating.
• evaluatorq skill: resources/judges-and-juries.md — llm_jury() verdict modes and panel configuration, jury presets, cyclic assignment, the judge prompt template namespace, reading raw_output["jury"] back as JuryResult, llm_jury_pairwise() swap-and-reconcile plus build_report() / BT-sigma metrics, and the structured-result trade-offs.
• evaluatorq skill: resources/tuning.md — the four distinct reasoning-effort knobs, token budgets for reasoning models, datapoint_parallelism vs llm_parallelism (with llm_slot()), target and pipeline timeouts/retries, extra_kwargs vs extra_body, model-catalogue registration, and the environment variable reference.

Changed

• evaluatorq skill: SKILL.md gains a scorer-selection table, judge/jury and reasoning-model sections, and an input section; Phase 2 and Phase 4 renamed accordingly.
• Cross-links between the evaluator skills: orq-run-experiment, orq-build-evaluator and orq-evaluator-alignment now route code-based evaluation, code-defined judge panels and platform-evaluator invocation to evaluatorq, and evaluatorq routes judge design and human-label validation back to them.
• orq-compare-agents/resources/evaluatorq-api.md re-probed against Python evaluatorq 1.39.0, @orq-ai/evaluatorq 1.3.2 and orq-ai-sdk / @orq-ai/node 4.15.6: full evaluatorq() signature (datapoint_parallelism, llm_parallelism, path, inference, ExperimentInput), a Python-vs-TypeScript parity table, and llm_jury() in the built-in evaluator list.

Fixed

• evaluatorq skill: parallelism is documented as the deprecated alias of datapoint_parallelism in the Python examples, and the dataset input uses DatasetIdInput instead of the bare dict. The TypeScript example keeps parallelism / { datasetId }, with a note that @orq-ai/evaluatorq 1.3.2 has no jury, pairwise or experiment-replay support.
• orq.evals.invoke() returns a flat EvaluationResult on orq-ai-sdk 4.15.6 — result.value / result.explanation / result.passed, not result.value.value. Corrected in evaluatorq, orq-compare-agents/resources/evaluatorq-api.md and orq-compare-agents/resources/gotchas.md, whose "Response structure is nested" section documented the old shape.
• orq-compare-agents/resources/evaluatorq-api.md: the TypeScript framework wrappers take the agent first and the name in an options object (wrapLangGraphAgent(agent, { name })), not ("Name", agent); the TS evals.invoke() request key is invokeEvaluatorRequest, not requestBody; the TS parallelism default is 1 (sequential), not 10.

[3.2.3] - 2026-09-21

Changed

• orq-cli, orq-shared, orq-analyze-traces, orq-improve-agent, tests: reverted the orq traces thread → orq traces conversation rename from 3.2.1. The 8.6.9 re-probe confirmed the shipped command is still traces thread, with no conv alias (orq traces conv prints the traces group help at exit 0), so every call site goes back to thread: the "Reading a conversation" section and its examples in orq-cli/SKILL.md, the command tree and per-trace read list in orq-cli/resources/command-map.md, the full-content bullet and layer 3b in orq-shared/resources/trace-queries.md, scenario 9a in tests/skills.md, and the allowed-tools entries, which are one Bash(orq traces thread:*) rule again. The version note about which release carries which name is gone with it.

[3.2.2] - 2026-09-21

Changed

• orq-cli re-probed against orq CLI 8.6.9 (API 4.14.20), RES-1577. A first pass ran without a session and a second on a live OAuth session covered auth whoami, workspace list / use, doctor, default-format, update --check, auth profile list and traces thread on a real conversation. Every section of SKILL.md and resources/command-map.md now opens with its own Probed against line, and the top-of-file banner points at those lines instead of naming one version, so it cannot drift again. Some claims could not be re-run (for example --profile precedence, which needs a configured profile); they keep their 5.1.0 or 8.5.2 stamp and are marked "not re-probed" or "not re-run", so grep -n 'not re-probed\|not re-run' lists the full set.
• --verbose no longer leaks the API key: it prints **HIDDEN**. It still writes the whole profile config to stderr, so the rule against using it in shared output stays.
• auth whoami exits 1 when there is no usable session. doctor still exits 0 whatever it finds, and its auth.status now reads as "configured, not proven". The doctor output shape in command-map.md was rewritten from real 8.6.9 output.
• Default output is table per --help, but piped it renders TOON. The global -o flag is validated client side (json|yaml|toon|table), and ORQ_OUTPUT_FORMAT=json works.
• agents list is unpaginated again. The 30-day retention 400 on trace queries no longer reproduces, and a .env file is no longer autoloaded. Trace queries: 168 fields with aliases, an unknown field is a 400, from/to are optional, and only end_time sorts.
• New surface documented: auth login --api-key, auth profile, auth sessions, status, switch, projects use, workspace list use, models list-preview, insights-service-*, and orq request returning ok,status,headers,body_text,body.
• traces conversation is still traces thread on 8.6.9, and there is no conv alias there.

[3.2.1] - 2026-09-20

Changed

• orq-cli, orq-shared, orq-analyze-traces, orq-improve-agent: orq traces thread is now orq traces conversation, short alias conv (orq-cli#102). The old name was dropped outright rather than kept as a deprecated alias, so every call site here moved: the "Reading a conversation" section and its examples in orq-cli/SKILL.md, the per-trace read list and global -o row in orq-cli/resources/command-map.md, the full-content bullet and layer 3b of the layered read in orq-shared/resources/trace-queries.md, scenario 9a in tests/skills.md, and Bash(orq traces thread:*) in two allowed-tools lists — replaced with both spellings, since a prefix rule for conversation does not match conv. The version note says which name an installed CLI has: thread through 8.6.x, conversation after. The xml render's framing tag moved with the command, from <thread …> to <conversation …>.

[3.2.0] - 2026-09-14

Added

• orq-cli: documented orq traces thread, the normalized conversation view added in orq CLI 7.4.0 (RES-1507), probed live against 8.5.2 — the span it selects and --spans to see why, the xml and markdown renders and why the default is xml (recorded content cannot forge a turn), --slice / --match / --include / --max-chars / --reasoning and the order they apply in, the state markers, the canonical structure behind -o json|yaml|toon, and the exit codes (an empty selection is 0; a bad slice, and a trace with no conversational span, are 1).
• orq-shared: resources/trace-queries.md now lists traces thread as the CLI path to full conversation content and as layer 3b of the layered read; MCP get_span mode=full is still the only source of per-message finish_reason.
• orq-analyze-traces, orq-improve-agent: added Bash(orq traces thread:*) to allowed-tools.

Changed

• All skills: every command example now asks for machine output with -o json. The global JSON alias flag and its environment-variable twin were removed in orq CLI 8.x — the flag exits 1 with unknown flag, and the variable is ignored silently, which leaves a caller parsing TOON. 146 call sites across 18 files were swept, and the places that documented the alias itself were rewritten rather than substituted: the global-flag table in resources/command-map.md, the --version paragraph and troubleshooting rows in orq-cli/SKILL.md, and the three sentences that contrasted the alias against traces thread's own -o (in orq-cli/SKILL.md, orq-cli/resources/command-map.md and orq-shared/resources/trace-queries.md) — a substitution there reads "takes -o json, never -o json". Spot-checked live on 8.5.2: auth whoami, agents list, traces list-fields, traces search, doctor, version.
• orq-cli: orq --version takes no format at all — orq --version -o json fails with unknown command "json" for "orq" because the value lands in subcommand position. Only orq version -o json serializes. Re-verified on 8.5.2.

[3.1.1] - 2026-09-01

Fixed

• orq-evaluator-alignment: on Windows, TLS verification is no longer disabled. Two independent copies of this policy existed — lib/orq_client.tls_verify() (used by OrqClient and judge.make_judge_client) and lib/model_backend._tls_verify() (used by the default orq_router backend and its router-price lookup) — both returned False on win32 to dodge an OPENSSL_Uplink(...): no OPENSSL_Applink abort some Windows Python builds hit. orq_client.tls_verify() now returns a truststore.SSLContext, which validates against the OS trust store (SChannel) instead of loading a CA bundle from disk — the disk-load path is what triggers the abort, not verification itself; the missing-dependency case now raises a clear error rather than a bare ModuleNotFoundError. model_backend._tls_verify() is no longer a second implementation — it's a re-export of the same function, so the policy can't drift out of sync again. ORQ_API_KEY no longer rides unverified connections on any platform. truststore is declared sys_platform == 'win32'-only in the eight scripts that reach either path (create_eval.py, dataset_inputs.py, fetch_evaluator.py, fetch_traces.py, recommend.py, rewrite_eval.py, seed_inputs.py, stability.py), and with the same marker in tests/requirements.txt. The win32 branch is exercised for real by the new skill-tests-windows CI job on windows-latest, not by spoofing sys.platform on a Linux runner. Confirmed on Windows (RES-1387): verify=True reproduces the crash, this fix does not, and SChannel correctly rejects self-signed/expired/wrong-host certs.

[3.1.0] - 2026-08-31

Changed

• orq-cli: document that the --version flag silently ignores --json (and that orq --version -o json fails with unknown command "json"); only orq version --json emits JSON.
• orq-cli skill re-probed against orq CLI 5.1.0 (built against orq API 4.14.3); it was previously verified against 4.13.0. Several load-bearing claims were wrong and are corrected:
  • --workspace / ORQ_WORKSPACE are now real CLI surfaces. The skill previously stated the CLI ignored them and that they were an evaluatorq-only convention. orq --workspace <key> <cmd> scopes one invocation without changing the session's active workspace.
  • agents list now paginates (default 10, has_more: true). The old text claimed it returned everything unpaginated, and built the workspace-canary guidance on that. Omitting --limit blocked for 4m16s and then returned HTTP 503, so the flag is documented as required. The 503 itself is not attributed to pagination: orq agents retrieve <valid-key> — one entity, no pagination — hung 4m14s with the same error, while a nonexistent key returned HTTP 404 in 0.19s. The skill says so explicitly, since "add --limit" is useless advice for a hanging single-entity read.
  • orq doctor reports auth.status: "authenticated"; the documented ok / missing / invalid vocabulary is gone. The checks array gained coding-agent, MCP, gateway-key-expiry and credential-permission rows, and an info status that is not a failure.
  • --example works on traces search, where it previously errored.
  • --server / ORQ_SERVER is now the only way to set the host. --api-base-url and ORQ_API_BASE_URL are deprecated spellings of the same value and warn on use; the default host moved to https://my.orq.ai. The old "two different hosts, not aliases" rule no longer holds — until 5.0.0 the six built-ins took --api-base-url and rejected --server while generated commands did the reverse, so a single run could reach two hosts. The skill now leads with the rule and marks the old flag as replace-on-sight.
  • orq server list replaced its per-entry override field with a top-level overridden; orq server current gained profile_server and server_default.
  • CLI versioning decoupled from the orq API version at 5.0.0 — orq version --json is now the way to read both.
• resources/command-map.md re-derived against 5.1.0: every top-level group is now listed. The previous pass omitted twelve groups as "not covered" and named two (activities, people) that do not exist.

Added

• orq-cli: install instructions — npm (Node ≥ 14, and the only supported route on Windows) and the curl | sh installer, with https://cli.orq.ai/install.sh as the canonical URL in place of the raw GitHub link the skill used (verified: it redirects there and is byte-identical, same SHA-256). Calls out that the installer offers to edit the shell profile and then runs orq setup — an interactive OAuth login that rewires the machine's coding agents — so an unattended install needs --no-modify-path --no-setup. Full flag/env set, rc channel, version pinning, checksum behaviour, pre-built binary targets, building from source, and the # >>> orq cli >>> profile markers moved to resources/install.md; flags confirmed by running the live installer's --help.
• orq-cli: delete commands require --force off a terminal as of CLI 5.0.0 — 40 generated commands plus orq request DELETE. Every agent and CI invocation hits this. Promoted to a Constraint, with the verified failure recorded: exit 1, empty stdout, Error: refusing to run "..." without --force in a non-interactive shell on stderr, and no request sent — confirmed because the same id with --force reached the API and returned 404. Unlike most traps in this skill this one is loud, so it costs a retry rather than data. Only DELETE is gated; reads and updates need no flag.
• orq-cli: an unknown subcommand prints help to stdout and exits 0 (verified: orq traces bogus → exit 0, 3160 bytes on stdout). A typo is invisible to a wrapping script. Promoted to a Constraint.
• orq-cli: OQL section for orq traces query-oql and orq logs query — the fixed fetch <source> prefix, the fact that equality is in (…) and that = / eq / : all fail with the same opaque invalid oql: invalid filter, the results nesting under search rather than at the top level, and the -j crash (formatting failed Invalid type for: <nil>) that a missing key produces there instead of a null.
• orq-cli: coding-agent surface (connect, disconnect, setup, launch, update) documented, with resources/coding-agents.md covering the capability matrix, the --local scoping rule, the orq skills / skills-capability name collision, and the over-permissioned keys minted by v4.13.10.
• orq-cli: credential-file permissions. Versions before 5.0.0 could leave ~/.orq/credentials.json world-readable permanently; orq doctor now flags it and orq doctor --fix repairs it.
• orq-cli: an explicit --profile outranks an exported ORQ_API_KEY and warns when it does — the escape hatch for the stray-key problem. Also documents that a stray key masks an unknown-profile error behind a misleading HTTP 401.
• orq-cli: the npm upgrade trap — npm update -g never crosses 4.x → 5.x, so that hop needs npm install -g @orq-ai/cli@latest.

Fixed

• orq-cli: the --verbose warning was understated. It prints to stderr, not stdout, and still dumps every stored API key in full plaintext even though auth list-profiles has masked them since CLI 5.0.0 — re-verified on 5.1.0.

[3.0.0] - 2026-08-31

Two shipped skills were removed and replaced under new names. Per the versioning table in CLAUDE.md, a removed or renamed skill is a MAJOR bump: an installer following semver would otherwise treat this as safe to auto-apply and silently lose two skills. There is no alias or redirect stub for the old names — update any script, macro or saved prompt that invokes them.

Added

• orq-analyze-traces skill: trace-grounded failure analysis using open/axial coding, replaces orq-analyze-trace-failures (deleted)
• orq-improve-agent skill: prompt rewrite and config knob fixes grounded in error-analysis artifacts, replaces orq-optimize-prompt (deleted)
• orq-shared skill: reference bundle the other skills read — holds no procedure, is not invoked on its own
• skills/orq-shared/resources/trace-queries.md: verified CLI trace query contract, now also the single home for the CLI-vs-MCP rules, the lever/knob/unobservable vocabulary, and the settings.tools[] read-vs-write schema translation that orq-analyze-traces and orq-improve-agent previously each restated
• HTTP 000 / TLS interception handling in run-key-preflight.md

Changed

• Shared references moved from docs/ into the orq-shared skill. npx skills add copies a skill folder and nothing else, so a ../../docs/... link resolved only in a repo checkout — for every skills-only install the references were simply missing. Sibling links (../orq-shared/resources/<file>.md) resolve after any install, because skills land next to each other. This repoints all 11 links across 7 skills, including the 9 that already pointed at docs/run-key-preflight.md before this release.
• The doc-resolution ladder is now stated once, in skills/orq-shared/resources/doc-resolution.md. Six skills carried the same four-step order and the same closing sentence, differing only in which live-query tools they named; each now links the shared order and keeps only its own first step. ## orq.ai Documentation is deliberately untouched — those deep links and Key Concepts are per-skill and should differ.
• orq-simulate-agent: dropped resources/redteam-mode.md. Three of its four sections were a thinner copy of the orq-red-team skill, including an SDK example that had already drifted from orq-red-team/resources/python-sdk.md; the routing decision it existed for is now two lines in the skill pointing at orq-red-team.
• commands/quickstart.md: the eval cycle now reads analyze → improve → run-experiment, matching what the skills themselves do. The previous order put validation before the fix and made the evaluator step unconditional.

Removed

• orq-analyze-trace-failures skill (replaced by orq-analyze-traces)
• orq-optimize-prompt skill (replaced by orq-improve-agent)
• docs/trace-queries.md and docs/run-key-preflight.md (moved into skills/orq-shared/resources/)

[2.8.0] - 2026-08-31

Added

• orq-build-evaluator: categorical output type — a verified single-request POST /v2/evaluators body with categorical_labels, plus per-label criteria in Phase 2, per-label few-shot examples in Phase 3, per-label sample-size targets in Phase 4, and a confusion-matrix / per-label precision-recall gate in Phase 5. MCP create_llm_eval cannot create categorical evaluators: categorical_labels is required at creation and the tool has no such field.
• orq-build-evaluator: Python evaluator log dict full 7-key reference with three runnable examples
• orq-build-evaluator: POST /v3/evaluators/{id}/invoke for rapid Phase 5 iteration, with the response shape and a payload-field-to-template-variable table. Evaluator CRUD is v2-only and invoke is v3-only — /v3/evaluators 404s, POST /v2/…/invoke returns 403 insufficient_scope for a workspace key.
• orq-build-evaluator: orq evals create invocation for categorical evaluators alongside the raw HTTP body, for scripts and CI
• orq-build-evaluator: orq evals invoke flag reference — the flat flags fold into context as the JSON shorthand does, --context takes the inner object while --from-file takes the full body, and system_instructions/tools_called have no flag of their own
• orq-build-evaluator: Phase 6 step 16 recommends running orq-evaluator-alignment after creating an LLM evaluator

Changed

• orq-build-evaluator: HTTP Fallback section now enumerates what MCP cannot do at all (categorical creation, categorical/string Python evals, listing, invoke) rather than a create-then-PATCH pattern
• orq-build-evaluator: evaluator variables updated from legacy {{log.*}} to v4.14+ {{input.*}}/{{output.*}}; resources/judge-prompt-template.md §6 is now the single canonical list
• orq-run-experiment: {{log.*}} occurrences updated to the v4.14+ names with a pointer to the canonical list
• orq-build-evaluator: softened binary-only stance to recommend binary Pass/Fail while allowing numeric scales with detailed rubrics and categorical for 3+ labels; frontmatter, "Done When" and the quality checklist follow suit
• orq-build-evaluator: TPR/TNR and the prevalence-correction formula are now explicitly scoped to binary evaluators

Fixed

• orq-build-evaluator: duplicate step 16 — Phase 7's maintenance step is now 17
• orq-build-evaluator: the Python response-length example scored Pass on an empty output — a failed generation reads as a short response. It now guards for empty and checks one criterion instead of two.
• orq-build-evaluator: the create examples used openai/gpt-4o-mini, contradicting the skill's own "start with the most capable judge model" constraint; both now use openai/gpt-4.1
• orq-run-experiment, orq-build-agent: evaluator invoke endpoint corrected from /v2/evaluators/<ID>/invoke (403) to /v3/…
• orq-run-experiment: evaluator list pagination corrected from after= (silently ignored, re-serves page 1) to starting_after=
• orq-build-evaluator: dropped the unverified string output type row and the unverified JSON-schema/HTTP evaluator types; function_eval and ragas are documented with their real parameter shapes
• orq-build-evaluator: the Python evaluator's library list was invented (a stdlib enumeration plus nltk). Replaced with what the docs actually name as preloaded — numpy 1.26.4, and since v4.10 requests and pydantic, the replacement for the retired HTTP and JSON evaluator types — and the byte-exact 1 MB limit with its Code exceeds maximum size error
• orq-build-evaluator: log["messages"] is documented as excluding the graded turn, matching the same caveat already carried for legacy {{log.messages}}
• orq-build-evaluator: the Phase 5 thresholds are targets, not pass/fail gates. The categorical pair (macro recall > 90%, no label under 80%) is mine and the numeric one (Spearman > 0.7, MAE inside the downstream tolerance) is a convention; neither is measured, and a skill that presents a number as a standard gets that number obeyed instead of examined. The table now says to pick your own before starting and to judge a miss on the disagreements rather than the number
• orq-build-evaluator: the HTTP/CLI reference — the fallback table, the verified categorical POST /v2/evaluators body, and the invoke payload with its body-path-to-variable mapping — moved to resources/api-reference.md, leaving a pointer. SKILL.md drops from 550 to 470 lines, under the spec's 500-line recommendation. It is lookup material, not procedure, so it survives the split; the procedure steps that need it link to it
• orq-build-evaluator: invoke payload documented as the context form the docs prefer, with the flat shorthand as the legacy alias. context is the request envelope, not a prompt namespace — there is no {{context.…}} variable. Of its four sub-objects only input and output share names with the variables they feed: context.messages renders as {{input.all_messages}} and context.variables.<name> as bare {{<name>}}. input and expected_output are not flat aliases and are dropped silently; system_instructions and tools_called are reachable only through context.
• orq-build-evaluator: resources/judge-prompt-template.md §6 now carries all seven template variables, the indexing syntax, the message and tool-call shapes, and the full legacy {{log.*}} mapping
• orq-build-evaluator, orq-run-experiment: {{log.messages}} is not equivalent to {{input.all_messages}} — it omits the graded turn. The previous entry mapped them as the same variable.

[2.7.0] - 2026-08-31

Added

• create-skill: meta-skill that probes a capability surface (API, CLI, or MCP server), builds a verified inventory, and writes a tested SKILL.md contract. Seven-phase workflow: gather requirements, build inventory, check existing skills, test/verify, write contract (including an adversarial self-review), register the skill on every surface the repo's contribution contract requires, clean up. Includes resources/writing-guide.md (failure-mode checklist, invocation decision, allowed-tools security boundary) and resources/template.md (output skeleton). Verified against the orq CLI and the orq MCP server on 2026-08-31.

[2.6.0] - 2026-08-31

Added

• orq-evaluator-alignment: {{input.retrievals}} and {{input.system_instructions}} are now fillable, so a RAG or instruction-following evaluator can be aligned instead of skipping every row. They are real judge input — verified by invoking a live evaluator built on each (2026-08-31): orq substitutes retrievals as newline-joined chunks and system instructions verbatim, and both arrive from the caller's payload, not from a knowledge base. lib/content gained the two leaves plus stringify_retrievals matching that rendering, and the row shape carries both end to end (fetch_traces → seed → judge). orq emits no span attribute for either, so a trace supplies them only through the template stencil — the same reversal that already recovers reference — and a dataset column or --map is the reliable source; SKILL.md says so at the dataset step. judged_input_key appends them only when present, so a row carrying neither keys exactly as before and no in-flight run's traces_fingerprint is invalidated.
• orq-evaluator-alignment: {{output.tools_called}} (legacy {{log.tool_calls}}) is fillable too, so an agent evaluator that grades tool choice can be aligned. orq does not hand the judge the list the caller passed — it formats it — and this skill re-renders judge prompts locally, so stringify_tools_called reproduces the format measured against a live evaluator (2026-08-31): 1. name(args) / Status: ✅ Success / Response: …, blank line between calls, ({}) for absent arguments, Response: N/A for a call with no result, No tool calls were made. for an empty list, and a nameless call dropped before numbering. A string passes through untouched — that is a block orq already rendered, recovered from a judge prompt by the trace stencil. One ceiling, named in the code: ToolCalled.status is '' | in_progress | completed | incomplete | failed, but the evaluator-invoke API drops it, so ✅ Success is the only marker any probe could confirm; a non-empty, non-completed status prints its own value rather than a guessed ❌.
• orq-build-evaluator: the variable list, the create-step mapping and resources/judge-prompt-template.md now name the v4.14+ {{input.*}}/{{output.*}} variables, including {{input.system_instructions}} and {{output.tools_called}}, with the legacy {{log.*}} names marked as still-resolving rather than current. This skill authors judge prompts, so leaving it on the legacy list meant every new evaluator came out in the format the rest of this release migrated away from — and orq-run-experiment/resources/rag-evaluation.md, updated in the same work, hands the reader straight to it.

Added

• validate-skills.mjs: check 11 fails any markdown under skills/ that names a legacy {{log.*}} template variable without saying "legacy" on the same line. orq kept the old names resolving, so a skill left on the legacy list breaks nothing loudly and simply keeps teaching the pre-v4.14 spelling — which is exactly how orq-build-evaluator sat on it while every skill around it migrated, found by hand rather than by CI. Markdown only: Python docstrings and test fixtures name {{log.*}} on purpose, to describe past bugs and to pin backward compatibility.

Fixed

• orq-evaluator-alignment: _FIELD_BY_LEAF now resolves the v4.14+ leaves (user_query, all_messages) so evaluators written with {{input.user_query}} or {{input.all_messages}} no longer silently skip every row as unmappable. The anti-drift test asserted only the log. spelling, which is why the rename passed it; it now runs every namespace the platform has used (log., input., output., bare), and a companion test pins the renderer — a leaf that maps but renders blank re-judges against an empty variable, which is the same silent loss one layer down.
• orq-evaluator-alignment: meta prompt example lists the v4.14+ input variables alongside {{output.response}} and the legacy {{log.output}}; the --map example in SKILL.md and dataset_inputs.py no longer teaches the legacy spelling.
• orq-run-experiment: template variable references updated from legacy {{log.*}} to v4.14+ {{input.*}}/{{output.*}} format across SKILL.md and resource files.

[2.5.1] - 2026-08-27

Fixed

• evaluatorq, orq-compare-agents: the skills never said jobs takes more than one job, so every generated script defined one job and any comparison became a loop of evaluatorq() calls. Both library examples passed jobs: [agent_job], the only multi-job example lived in orq-compare-agents/resources/evaluatorq-api.md (reachable via a link at the bottom of the evaluatorq skill), and parallelism: 5 appeared in both examples with no explanation of what it gates. In the source (src/evaluatorq/processings.py) every job runs against every data point and the jobs for a data point are dispatched together with asyncio.gather under a Semaphore(parallelism), while evaluatorq.py gates concurrent data points with a second semaphore of the same size; the results table pivots evaluator x job into one comparison. A loop throws all of that away: N unrelated experiments, N tables to diff by eye, no shared sampling, and serialized passes over the dataset. evaluatorq/SKILL.md now opens Phase 3 with the rule and a correct/wrong pair before either example, both examples ship two jobs, and the section hands separate-agent comparisons to orq-compare-agents, whose Phase 4 now says "a single evaluatorq() call" outright.
• orq-compare-agents: the documented Python signature had parallelism: int = 1; upstream (src/evaluatorq/evaluatorq.py) defaults it to 10. Anyone reading the reference and omitting the argument expected sequential execution and got ten-way concurrency against their provider.

[2.5.0] - 2026-08-26

Added

• orq-evaluator-alignment: one list now gates both the step-1 fetch and the step-7 create, and free-form string judges are out of scope. The two gates had drifted apart: fetch_evaluator accepted boolean | categorical | number | numeric | string while build_create_body accepted boolean | categorical | number, so a string — or a numeric, the alias the fetch gate explicitly tolerates and create_eval even special-cases for scale — passed step 1, paid for the full stability run, the grey-zone loop and the rewrite, and then died at create_eval --approve, after approval.json had been written. orq_client.SUPPORTED_OUTPUT_TYPES and normalise_output_type are now the single source of truth both paths resolve through, numeric is de-aliased to number on the way in so only one spelling reaches the request body, and a test asserts the round trip — every accepted type can actually be created, and every refused type is refused in both places — so the two cannot drift apart again silently. string is refused rather than fixed: orq's API accepts the type, but instability over prose is exact-match entropy, which scores nearly every row as maximally unstable and therefore ranks nothing, and gate (b) has no mechanical way to tell a correct answer from one merely worded differently — the alternative being a human reading every pair, which is the manual labelling this skill exists to remove. Removing it takes the whole stop-and-resume-for-a-reader path out of retest.py (string_pairs.json, string_pairs_key.json, string_verdicts.json, the blind-shuffle key, the fingerprint check and the sub-run reuse that existed only to keep that resume from livelocking), which simplifies the retest rather than merely shrinking it.
• orq-evaluator-alignment: a dataset whose exchange lives under messages no longer maps zero rows (flow-friction spec §2). seed.map_datapoint filled output only from inputs keys, so a dataset whose inputs carried metadata — category, difficulty, ground_truth — while the actual exchange sat in messages reported ✓ Added 0 rows; 42 skipped (unmappable) with missing ['log.output'] on every one. The output was right there: log.output is the assistant's last turn. Production traces never hit this because orq's gen_ai.input supplies output outright; the dataset path is the one that has to derive, and had no rule to derive with. lib/content.derive_io_from_messages is now that rule — last assistant turn → output, last user turn → query, read through message_text so Responses-API parts[] and multimodal turns resolve exactly as the trace scanner resolves them. It lives in content.py rather than seed.py because that module is the shared vocabulary and the scanner will need the same rule the day it meets a span with messages and no explicit output. inputs still wins; derivation is fallback only; system/tool turns are never the output; and a message with no role is still skipped rather than guessed at positionally, so genuinely unmappable data fails exactly as before. Diagnostics are now per dataset, not per row — one inventory naming what the judge needs, what the dataset holds, and what each field could map to, in place of N identical skip lines — plus a --map "log.output=messages.assistant.last" escape hatch over a deliberately tiny grammar (an unknown source is an error, never a silent skip) recorded in input_mapping.json. A pull that maps nothing now stops instead of appending zero rows and reporting success.
• orq-evaluator-alignment: ground truth is used instead of collected and dropped (flow-friction spec §3). map_datapoint correctly routed a dataset's expected_output into reference — and nothing downstream ever read it, because an evaluator that declares no {{reference}} variable never renders it. So the skill kept repeating that it "structurally cannot find cases the judge gets wrong with total confidence" while sitting on the labels that answer exactly that. reference now rides through stability.json, and metrics.json carries a correctness block: accuracy, a confusion tally, and — the point — by_band. Accuracy among the rows the judge was stable on is the consistently-wrong blind spot, measured rather than caveated: a judge at 0.0 mean instability and 60% correct on stable rows is the exact failure everything else here is blind to, and it now leads the report. build_queue gained a third confuser class, wrong_vs_reference — stable and disagreeing with the label, the highest-information rows in a run and previously invisible — appended as its own reason rather than interleaved, so the instability ranking stays a clean single-metric order. The grey-zone payload carries reference and judge_correct so the conductor can group by how the judge is wrong rather than only by what it is unsure about, and the retest may fill unanswered rows from dataset labels, tagged dataset_reference in label_provenance and never merged into the human-answer count. Guardrails throughout: a label that will not parse into the verdict space is excluded and warned about rather than counted as the judge being wrong; SKILL.md now branches — state the blind spot when there are no labels, state what was actually verified when there are — and requires the user to confirm a dataset label before it drives a rewrite.
• orq-evaluator-alignment: the regression check now covers every re-judged row nobody labelled, and the report says how far it looked. _regression_report was called with _low_flip_indices(out_dir) & scope — the ~5 stable spot-check rows the queue holds back — so a rewrite could move behaviour across the rest of the dataset and the retest would still print "0 of 5 previously-stable rows changed". True, and read by everyone as "nothing regressed". --all_rows already re-judged everything, but the regression view stayed pinned to those five, so even the expensive option did not widen the answer. retest_metrics.json now carries regression_on_unlabelled_rows (every re-judged row without a human label — the labelled ones are supposed to change, so counting them would report the rewrite working as if it were misfiring) alongside the existing regression_on_stable_rows, plus regression_scope recording how many of the original datapoints were actually re-judged. A caveat fires whenever that is narrower than the original run, and SKILL.md step 8 now asks — "want me to re-run the new judge over all 200 original datapoints and show you what moved? about the same cost as the first run" — rather than leaving --all_rows as a flag nobody offers. Worth asking out loud because the answer depends on something the skill cannot see: whether the evaluator is already scoring live traffic.
• orq-evaluator-alignment: the input source is now a question, not a default. fetch_evaluator.py chained straight into a 200-trace scan, which answered "where should the examples come from?" silently and framed the other three sources — an orq dataset, examples the user brings, generated cases — as recovery options, reachable only once the scan had come back starved. That reading is backwards for a judge that went live yesterday, fires rarely, or is older than the scan window: those users met an empty scan that looks like a dead end rather than a menu they were always entitled to. fetch_evaluator now fetches the judge and stops (--with_traces re-enables the chained scan once the user has picked that route), and SKILL.md step 1a asks the question outright with all four routes on it. The trace scan is still led with and still named as the only source that shows what the judge meets in production — a real difference, and the reason to prefer it — but it is presented as the option to beat rather than the one already taken. Sources compose, so "scan production, then top up with generated borderline cases" is now an ordinary answer; because the scan overwrites traces.jsonl while the other three append, it must run first, and it now refuses (rather than silently deleting) when it finds rows from another source, with --replace as the deliberate override.
• orq-evaluator-alignment: the scan reports its depth, and offers to go deeper before the run is priced. The trace scan reads the 200 most recent traces, and it reported only the datapoint count — so "18 examples" read as everything this judge has, when it is 18 out of a 200-trace slice of a longer history. The deeper scan existed but was filed under step 1a, reachable only once the scan had come back starved (fewer than 10 usable examples); a judge with 18 went straight on. fetch_traces now prints the count next to the window it came from and writes scan.json, and the conductor stops there to offer --trace_limit 2000. The offer is placed before step 2 deliberately: pulling more traces costs no judge calls, and it rewrites traces.jsonl, which is safe now and unsafe once anything is labelled — so this is the last moment the choice is free. It is also conditional, on the new scan_truncated flag: the deeper scan is offered only when the scan stopped at the cap, which is the tell that history remains behind it. A scan that came back under the cap already saw every trace in the window, so a bigger limit would re-scan the same traces and find the same rows; there the note points at trace_start_date/trace_end_date instead. Offering a remedy that cannot help is how a gate teaches people to click past it.
• orq-evaluator-alignment: orq_router is the new default backend for the recommendation and rewrite stages — any workspace model over the OpenAI-compatible /v3/router, using the ORQ_API_KEY the skill already requires, defaulting to deepseek/deepseek-v4-pro. The rewrite is a text transform, not an agentic task, and the previous default (claude_subagent) silently assumed the claude CLI was installed and logged in. backend_model is now optional per backend, so switching backend alone is a valid edit instead of quietly handing a Claude model id to the router. Calls are priced from the registry's own per-1K rates rather than reported as zero. Step 7 states the model, notes it must be enabled on the workspace, and offers the two alternatives (another model by refId, or the Claude subagent); both scripts preflight with one tiny call and stop with an actionable reason — 403 vs 404 vs 429 are distinguished — rather than failing partway through a paid run.
• orq-evaluator-alignment: the backend's endpoint, model and credential are now an explicit, parameterised surface. recommend.py and rewrite_eval.py take --backend, --backend_model and --backend_base_url, each resolving flag → config.toml → environment variable → built-in default, so a self-hosted or proxied orq (or an Anthropic-compatible gateway) is a flag rather than a code edit. --help prints all three with their fallbacks, BACKEND_ENV in lib/model_backend.py is the one table the docs and error messages read, and describe_backend names the resolved endpoint at the step-7 gate so the user can see which host is about to receive their rubric. ORQ_API_BASE_URL now also moves every control-plane call (lib/orq_client.py had the API host hardcoded with no override at all, so a custom host could be set for the router and the deployment backend but not for reading evaluators, traces, projects or datasets). API keys stay environment-only by design — a --backend_api_key flag would put a live credential in shell history and in every ps listing — and a missing key names the variable it wanted.
• orq-evaluator-alignment: a narrow escape hatch to the "never read the raw artifacts" rule. Reading a single datapoint by source_index is now sanctioned when the payload lacks a field the decision needs; the blanket prohibition was pushing the conductor toward guessing instead.
• orq-evaluator-alignment: a step 0 that asks whether the user actually has a judge in orq before anything else runs. Three routes: an evaluator id continues into the flow; a bare prompt is set up as an evaluator first (via orq-build-evaluator) and then continues; neither hands off to orq-build-evaluator and stops. Previously the flow opened by demanding an id, which is the one thing a user who needs this skill most may not have.
• orq-evaluator-alignment: grey_zone.py assemble --include_low_flip to opt the stable spot-check rows back into the payload, and n_low_flip_excluded / n_confusers in the budget block so the conductor can state what actually entered context.
• orq-evaluator-alignment: the retest reports a before for signal (b), and the caveats that make both signals softer than they look. Agreement previously had no baseline, so "accuracy 0.78, PASS" read as a win even when the original judge had scored 0.85 on the same labels — and that number costs nothing, because the original run already judged those very rows. retest_metrics.json now carries agreement.before alongside a caveats list the conductor is told to read out: the retested rows were selected for maximum observed instability (so re-measuring them regresses toward the mean on its own), the same examples produced the rewrite guidance and the labels scoring it (no holdout in v1), and any label the user did not confirm is the conductor's application of their rule rather than their verdict.
• orq-evaluator-alignment: retest.py --baseline_rerun re-runs the original judge over the same rows in the same pass. It doubles the cost and is the only version of gate (a) that isolates the rewrite from the selection effect; off by default, offered rather than assumed.
• orq-evaluator-alignment: retest.py --with_low_flip re-judges the queue's stable spot-check rows as an unlabelled regression check, reporting how many changed verdict. The step-5 promise to spot-check the rows the judge was completely sure about was never kept on the default path — they were excluded from the payload and, being unlabelled, from the retest too — so a rewrite that settled the grey zone by unsettling everything else was invisible. SKILL.md step 6 now also has the conductor read those rows with the user before the rewrite.
• orq-evaluator-alignment: grey_zone_policy.json labels carry a label_source (derived | human_confirmed) and SKILL.md gates on reading the derived labels back to the user. The human answers a rule question and the conductor derives each point's value from it, so by default the ground truth the retest scores against is model-derived; apply warns with the counts and retest_metrics.json records them, rather than presenting one model's reading of the rule as a human verdict.
• orq-evaluator-alignment: validate_policy checks every label against the evaluator's own verdict space. SKILL.md says never invent a label or move the scale and nothing enforced it, so an invented categorical label or an out-of-scale score passed validation, entered the rewrite guidance, and then scored 0 accuracy at step 8 — which reads as a judge failure rather than the policy error it is. Numeric tolerances must also be non-negative numbers, and a label may not reference a grey zone the policy never declares.
• orq-evaluator-alignment: stability.json and queue.json record a traces_fingerprint of the traces.jsonl their source_index values point into (lib/content.traces_fingerprint, sharing the scanner's own row-identity key). The grey-zone assemble warns on a mismatch and retest.py refuses outright.

Removed

• orq-evaluator-alignment: scripts/run_experiment.py, superseded by retest.py since the Part 2 feedback loop and left behind rather than kept. It was not merely unused — it was wrong on the skill as it now stands. It selected labelled rows with isinstance(value, bool), so on a categorical or numeric judge it silently scored zero rows; it hard-required annotations.json, which is the UI fallback artifact rather than the default grey_zone_policy.json; and it checked agreement alone, when the whole premise since Part 2 is that agreement without an instability drop is gameable. retest.py covers every case it did, for all types, on both signals. The one thing genuinely lost is --recommend_only, its variance-aware suggestion for the retest repeat count — that logic was boolean-only (it keyed off flip_rate) and SKILL.md already replaced it with an explicit instruction to use at least the stability run's N. Its two orphaned config keys (retest_repeats, retest_repeats_cap) go with it; the retest reads n_repeats.

Changed

• orq-evaluator-alignment: the skill's README.md and the repo-level surfaces (root README.md, agents/AGENTS.md, tests/skills.md) describe what the skill now is. All three still advertised a binary Pass/Fail judge and a per-row annotation queue — the pre-RES-978 shape — so the skill's own docs were the last thing claiming it couldn't do categorical or numeric, and the smoke tests were verifying the flow it no longer runs. The skill README's pipeline table had also drifted past four new scripts and named a default backend that is no longer the default. The SKILL.md description now names the question-led grey-zone flow, which is the trigger a user most plausibly types ("my judge keeps changing its mind").
• orq-evaluator-alignment: lib/seed.py reads the shared lib/content.field_for_variable table instead of keeping a hand-mirrored copy of the suffix rules. That copy is what produced the 2.4.1 bug where a {{log.reference}} evaluator had every seeded row silently skipped; there is now one table and no mirrors, pinned by a test asserting every judge-fillable leaf resolves.
• orq-evaluator-alignment: the grey-zone payload renders a conversation through the same parts-aware helper the judge renders with, rather than dumping raw JSON. On a Responses-API row the text sits under parts[], so the dump spent the character budget on structural noise and showed the conductor a different artifact than the judge was given.
• orq-evaluator-alignment: the recovery menu moved from step 4a to step 1a and is now offered the moment the trace scan comes up short, not several steps later after the user has already committed to a run. It gained a fourth option — bring your own examples from outside orq — alongside looking further back, using an existing orq dataset, and generating cases; the second-model option is now scoped to the flat-profile case it actually addresses.
• orq-evaluator-alignment: config.toml's section comments use the same step numbers as SKILL.md. They still numbered the recommendation and rewrite as steps 8 and 9 from an earlier draft of the flow, which is the sort of drift that sends a reader looking for a step that no longer exists.
• orq-evaluator-alignment: fetch_traces takes its dedup key from lib/content.judged_input_key rather than keeping a private copy, so the definition of "the same datapoint" is shared with the fingerprint guard by construction.
• orq-evaluator-alignment: SKILL.md rewritten for a non-specialist reader. The conductor now describes judge behaviour ("it gave a different answer 6 times out of 8") rather than the statistics that measured it, and the internal vocabulary — confuser, grey zone, instability band, open-code, conductor — is kept out of what it says to the user. Each step opens with a sentence on what is about to happen and why.

Fixed

• orq-evaluator-alignment: numeric instability could not reach 1.0, so the three output types were not on one scale despite everything downstream reading them as if they were. numeric divided the population standard deviation by the full declared scale range, but a maximal split — half the repeats at each end of the scale — has a standard deviation of exactly half the range, so the most unstable numeric judge possible scored 0.5, where boolean and categorical both reach 1.0. The denominator is now half the range, so 1.0 means "as split as this judge can be" for every type. Existing numeric scores double: the worked example (population stdev 1.5 on a 1–10 scale) moves from 0.17 to 0.33, and rows that sat just under a band edge move up one band. The bands themselves (<0.1 stable, 0.1–0.3 noisy, >0.3 unreliable) are unchanged, so a numeric run recorded before this release cannot be compared band-for-band against one recorded after it.

• orq-evaluator-alignment: serve_annotation.py could not start. Its PEP 723 block declared only fire while the module imports loguru, and uv run builds the script's environment from that block and nothing else — so the documented command in SKILL.md died with ModuleNotFoundError before serving a single page. This is the fallback the grey-zone flow falls back to, for the case where the confusers don't cluster, which is exactly when nobody has a second option. The rest of the suite could never catch it, because pytest runs with every dependency installed; tests/test_script_deps.py now parses each step script's inline metadata and its imports and compares them, so the declared set is verified in the environment that actually matters.

• orq-evaluator-alignment: the numeric agreement band ignored the scale it was measuring, which is the reverse of the argument that makes instability normalize by the declared range. numeric_agreement compared against an absolute tol defaulting to 0.5, so the gate read differently on every judge: a rewrite disagreeing with the human by 40% of a 0–1 groundedness scale passed, while one within a single point everywhere on a 0–100 scale failed. Since signal (b) exists precisely so signal (a) can't be gamed, having it go vacuous on the 0–1 scale most groundedness judges use defeats the point. The band is now derived from the evaluator's declared scale (lib.agreement.default_tolerance, 10% of the range by default), with an absolute override still available.

• orq-evaluator-alignment: one config key for the numeric band, not two. recommend.py read numeric_tolerance while retest.py and cross_model.py read numeric_tol, and neither appeared in config.toml — so setting the one that looked right moved what counted as a disagreement in one step and left the gate that signs the rewrite off sitting on its default. numeric_tol is now the single key, read by all three, and it is documented alongside numeric_tol_fraction and the four retest_min_* gates that were also read from cfg and named nowhere.

• orq-evaluator-alignment: the human's per-point numeric tolerances stopped being thrown away. validate_policy requires a tolerance on every numeric label and policy_labels carries it through, but the retest honoured it only when every point happened to share one band — otherwise several considered bands from the human were silently replaced by the configured default, and _pair_with_labels never read the field at all. numeric_agreement now takes per-pair bands and reports tol_source.

• orq-evaluator-alignment: gate (a) could be passed by rows going unmeasurable. The "before" mean was carefully recomputed over the retested rows, but the "after" was the retest run's own mean_instability — an average over whatever that run could measure. A rewrite that answers off-contract on more than half a row's repetitions drops that row below metrics.py's floor, out of the retest average, and the remaining rows average lower; the rows selected for a retest are the most unstable in the run, so that is precisely where it would happen. Both means are now taken over the rows both judges could measure, with n_rows_compared, n_lost_unmeasurable and the retest's off-contract count in retest_metrics.json, a warning when any row is lost, and a caveat naming the failure mode.

• orq-evaluator-alignment: parse_numeric accepts a quoted number in the schema ({"value": "4"}). Models quote numbers routinely; requiring a bare int/float sent those completions down the free-text path, where the surrounding explanation's own digits made the verdict line ambiguous and a judge that had answered correctly in the schema it was handed was recorded as off-contract.

• orq-evaluator-alignment: the instability report names the rows with off-contract repetitions, not just the dataset-wide total. A row can answer off-contract half the time and still band as stable on the rest, and the aggregate count could not say which row that was — which is the row a reader is about to trust.

• orq-evaluator-alignment: the grey-zone payload no longer silently overrides the count the user chose. build_queue appends low_flip_sample_size (default 5) stable spot-check rows to the queue, and assemble_payload was taking the whole queue — so "review 3 with me" put 3 + 5 = 8 examples into context, five of which had no grey zone in them by construction. Those rows are a control group for annotation, not material to open-code; they are excluded by default and reported separately.

• orq-evaluator-alignment: the aligned copy is named after its source instead of after an opaque id. GET /v2/evaluators/{id} returns an empty key for evaluators created through the current orq UI, so the old fallback produced keys like 01kzxt86ac82d1fvzw3s8wfv83-aligned-<ts>. The name is now resolved key → display_name → raw.key → id, giving groundedness-bool-testcase-aligned-<ts>. The key is what matters here: orq overwrites the display_name sent on create with the key, verified against the live API.

• orq-evaluator-alignment: the aligned copy is created in the same project as its source rather than in a new folder named after the source's id. The evaluator record carries no path on either the detail or the list endpoint, so the source's folder cannot be read back directly; the new resolve_project_key maps the id it does carry to the project key that heads a create path. That id is domain_id, not project_id: GET /v2/evaluators/{id} runs through normalizeEvalToInternalEvaluator, an explicit field allowlist that never emits project_id — so the first cut of this fix resolved nothing and every copy still landed at the workspace root. Both spellings are read, the project record is matched on any of its id fields, and the lookup pages through /v2/projects instead of reading only the first page. Falls back to the workspace root when the project can't be resolved, instead of inventing a folder.

• orq-evaluator-alignment: a free-text numeric judge no longer gets the wrong score, silently. parse_numeric fell back to the first numeric token in the completion, so 'On a scale of 1 to 5, I give this a 4' parsed as 1.0 with status ok — inside the declared scale, so nothing downstream re-checked it, and the wrong score entered mean, stdev and the instability band. It now reads the verdict the way parse_categorical does (after the last Value:/Score: label, else the last line), scrubs scale mentions (of 1, to 5, out of 5, /5) first, and surfaces a verdict line still carrying two different candidates as wrong_output_type rather than guessing between them.

• orq-evaluator-alignment: the preservation guard no longer passes a moved verdict space on a substring match. endpoint not in proposed is True for a rubric that only ever says 10, so a rewrite moving the scale from [1, 5] to [1, 10] cleared the gate while both endpoints were in fact gone; the same shape let a dropped spam label read as preserved by a rubric mentioning spammy. Both are matched on their own boundaries now.

• orq-evaluator-alignment: real judge reasoning is no longer discarded as a jury placeholder. _is_real_rationale substring-matched not available / no explanation, so 'the tool was not available at inference time, so the claim is ungrounded' was classed as no-rationale and the conductor was told there was nothing to read when there was. The generic phrases now only count as placeholders when they are the whole rationale; the distinctive jury notice still matches anywhere.

• orq-evaluator-alignment: retest.py --num_samples broke the comparison it was meant to smoke-test. The label scope stayed the full set while num_samples narrowed only the retest stability run, so original_mean covered every labelled row and retest_mean covered the first few — the subset-vs-full-run artifact the surrounding code exists to avoid. It now caps the selection itself, so both sides cover the same rows.

• orq-evaluator-alignment: the retest defaults its repeat count and temperature to whatever the original stability run actually used, read back from metrics.json, instead of to the config. Instability is estimated from N samples at a temperature and both bias the estimate, so retesting at different ones compared two different measurements; temperature was not forwarded to the retest run at all. A deliberate override to fewer repeats now marks the comparison not comparable rather than letting the sample size read as an improvement.

• orq-evaluator-alignment: label loading no longer discards half the labelling. retest/rewrite preferred grey_zone_policy.json whenever it existed, but the documented fallback runs in the other direction — try the grey-zone Q&A, find the examples don't group, open the annotation UI — so the labels collected last were silently ignored. Both files are now merged into one label set keyed by source_index, with the annotation winning a datapoint both cover. Merging rather than picking also removes a file-mtime dependency: the same run directory now scores the same way twice.

• orq-evaluator-alignment: the numeric scale is no longer sent on create. The evaluator schema has no scale field and the API drops it, so the request only looked like it carried a guarantee it never had; a numeric judge's scale lives in its rubric text, which the preservation guard is what actually protects.

• orq-evaluator-alignment: the grey-zone payload honours max_chars for real. The elision marker and the name: prefixes were added on top of the budget, so a 600-char ceiling across several variables rendered close to a thousand; both are charged against it now, and a budget too small to give every variable a character resolves to zero rather than overspending.

• orq-evaluator-alignment: cross-model disagreers are labelled and counted in the payload. The queue appends them after the instability ranking, so the token clamp drops them first — and on a judge that never wavers they are the entire signal. Each confuser carries its reason, and the budget block reports n_dropped_cross_model.

• orq-evaluator-alignment: testcases/rag-groundedness/answer_key.json no longer contradicts itself. G3c was listed both as unstable at 0.5 and as a stable-but-wrong case on the same judge; it is wrong and unstable, which is a different thing, and now sits in its own unstable_and_wrong_cases. PC3 claimed six stable-wrong cases and named seven ids, three of which (T1, T2, T4) appear in neither measured baseline and are not accounted for by either run's agreement count — it now states what was measured and flags the traps as designed-but-unconfirmed.

• orq-evaluator-alignment: the grey-zone payload no longer silently omits most of the judged input. When judge-template inversion failed, _compact_input fell back to output or query — and on a row captured in the structured Responses-API shape output is the assistant's answer, not the rendered prompt. So a groundedness judge's context and question, the only things groundedness can be checked against, never reached the conductor. The character accounting was computed over that single field, so the budget block truthfully reported "0 elided" while most of the input was missing: a silent failure the disclosure protocol structurally could not see. The fallback now carries every captured field (query, output, messages), accounts for all of them, and flags itself as input_source: "fallback" with an n_fallback_input count, because no elision number can reveal a field that was never collected.

• orq-evaluator-alignment: an exact tie no longer presents the jury's tie-break notice as if it were the judge's reasoning. Ties are where seeing both sides matters most and are exactly where no rationale exists, so the payload now carries reasoning_available / n_no_rationale and the conductor states the gap. (Per-repetition rationales remain unavailable — evaluatorq's jury layer collapses them — so this flags the limitation rather than working around it.)

• orq-evaluator-alignment: retest.py re-judged every row in traces.jsonl rather than the labelled ones, despite _build_retest_dir's docstring claiming otherwise — a 24-row run with 8 labels cost 24 × N calls, triple what the skill quoted, while agreement still scored only the 8. It is now scoped to the labelled rows, with --all_rows to opt out; because source_index is positional, the retest sub-run carries an index_map.json so labels still pair correctly, and the original mean is recomputed over the same subset so the reported drop isn't an artifact of row selection.

• orq-evaluator-alignment: metrics.py's correctness block stops computing numbers that look like a measurement and aren't. Three cases used to score a comparison that was never meaningful: an evaluator that declares a reference-family variable ({{log.reference}}, {{expected}}, {{expected_output}}) was shown reference as judge INPUT via lib.judge.make_replacements, so grading its verdict against that same value was circular, not a check — the new lib.content.reference_is_judge_input rule short-circuits it, returning n_labelled: 0 and a reason_omitted instead of a number; a categorical reference outside the judge's own declared label set was compared anyway, scoring a vocabulary mismatch as the judge being wrong, and is now excluded and counted in n_unreadable_labels; and numeric with no declared scale and no configured numeric_tol used a fixed absolute band that meant nothing on that judge's scale, and is now omitted with the same reason_omitted shape rather than computed against an arbitrary number. by_band.stable's headline claim is gated the same way: it only earns "the consistently-wrong blind spot, measured" once the labelled subset covers at least 10 rows and at least 90% of the stable band — below that floor the report captions it "partial view — do not conclude 'consistent-and-right' from this" instead of the stronger claim the coverage doesn't support yet.

• orq-evaluator-alignment: the retest no longer crashes on an unreadable dataset label, and dataset labels no longer inflate the retest cost without being asked to. _merge_dataset_labels fed an unreadable reference straight into lib.agreement's raising bool coercion, so it threw an uncaught ValueError — AFTER the judge calls it exists to score were already spent. It now parses each reference into the judge's own verdict space first, the same tolerant coercion metrics.py uses, counting what it can't read instead of crashing on it; and it skips the merge entirely for a reference-declaring evaluator, via the same reference_is_judge_input rule the correctness block uses, rather than merging judge-input back in as if it were ground truth. Dataset-merged labels were also silently re-judged alongside the human's own answers, inflating the bill past what the human actually labelled for a signal that is someone else's prior judgement, not the user's verdict; a new --with_dataset_labels flag (default off, mirroring --all_rows) keeps the default retest scoped to what the human answered, and logs how many rows were left out.

• orq-evaluator-alignment: three ways the retest's gates could pass a rewrite that hadn't earned it. success ignored the agreement before score entirely, so a rewrite could clear the 0.7 accuracy bar while scoring worse than the judge it replaces and still report PASS — agreement.regressed_vs_before now forces success: false whenever the primary metric regressed vs. the original judge. The before/after instability comparison paired the new judge's rows against every labelled row in the original run rather than the rows actually re-judged, so --num_samples (or a partial --with_low_flip retest) reported a comparison that never happened over matching rows; both sides are now the same scope, so the documented "--num_samples narrows both sides" claim is finally true. And gate (a)'s comparability check tracked repeat count but never temperature, so a retest at a different temperature — which changes how often the judge flips, not just what it says — read as directly comparable; temperature now joins the check, alongside instability.selection_bias_controlled stating outright whether --baseline_rerun isolated this run from the regression-to-the-mean that selecting the most unstable rows otherwise causes.

• orq-evaluator-alignment: the confuser queue's --count now caps what it actually promises to cap, and never double-counts a row. build_queue.py gained a third confuser class, wrong_vs_reference, in the previous release, but --count only capped the flipped/cross-model classes — a large wrong-vs-reference tail could slip past a user-chosen count uncapped. --count now caps the COMBINED confuser list (flipped, then cross-model, then wrong-vs-reference — the cap trims the tail), queue.json's meta gains n_wrong_vs_reference alongside the existing counts, the low-flip sanity sample never re-offers a row already queued under another reason, and each item carries judge_correct so the conductor can group by how the judge is wrong rather than only by what it is unsure about.

• orq-evaluator-alignment: four smaller honesty gaps at the pipeline's edges. scan_truncated was computed from the date-window-filtered trace count, so a tight trace_start_date/trace_end_date could hold few rows while the raw scan still hit trace_limit — the note then told the user a deeper scan couldn't help when it, in fact, could; it now reads the raw pre-window fetch depth. --map onto the reference field used to be silently overwritten by the expected_output derivation running after it, and an unmappable --map variable leaf (a typo like log.outout) parsed successfully and then no-op'd forever; --map onto reference now wins, and an unmappable leaf is a ValueError naming it, at parse time. create_eval.py gated creation on var_check_passed alone, so a rewrite whose verdict space had shifted — a dropped categorical label, a moved numeric scale — still created cleanly as long as the {{...}} variable set matched; it now enforces verdict_space_ok/preservation_ok too, --force bypasses all three, and approval.json records which were forced in forced_checks. And rewrite_eval.py now warns when grey_zone_policy.json/annotations.json were touched after aggregated.md was last written — the summary the rewrite is about to act on would otherwise silently predate the latest human input.

• orq-evaluator-alignment: a numeric retest with no way to size gate (b)'s tolerance now refuses before any judging, instead of silently scoring against an arbitrary band. _resolve_numeric_tol bottomed out at lib.agreement.FALLBACK_TOL (0.5) when there was no --tol, no uniform grey-zone policy band, no configured numeric_tol, and no declared scale — the exact situation metrics.py::_correctness already refuses on (reason_omitted), but the retest gate went ahead and scored against it anyway, silently making gate (b) vacuous on the 0–1 judges most groundedness rubrics use. _resolve_numeric_tol now also returns which of cli/policy_uniform/configured/scale_derived/fallback produced the band, so main can refuse on the fallback case specifically — before the retest sub-run is materialized or judged — naming the remedies (--tol, numeric_tol in config.toml, a declared scale via fetch_evaluator.py --scale_min/--scale_max, or per-point grey-zone bands). The --tol resolution chain documented in SKILL.md/config.toml no longer claims it ends at an absolute 0.5 for the retest specifically (cross_model.py's own probe is unchanged and still falls back to it).

• orq-evaluator-alignment: selection_bias_controlled now reports what the run actually did, not which flag was passed. It was set from bool(baseline_rerun), so a --baseline_rerun that found no rows measurable on both sides (baseline_mean is None, falling back to the original-run comparison a few lines later) still claimed true — the one field that exists to say whether THIS run isolated the rewrite from the selection effect was, in that case, wrong. Set from the outcome (baseline_mean is not None) instead, matching the caveat logic that already keyed off the same condition.

[2.4.1] - 2026-08-26

Added

• orq-evaluator-alignment: testcases/rag-groundedness/ — a live end-to-end test case for the skill. One orq dataset of 24 RAG-groundedness datapoints (8 stable anchors, 12 grey-zone cases across 4 engineered ambiguities, 4 consistently-wrong traps) plus four judge evaluators covering every verdict type, a provisioning/teardown script, an answer key with pass criteria, and score_run.py to grade a finished run. Unlike tests/, it exercises the real API and answers what unit tests cannot: does the skill find the ambiguity that is there, and does it admit what it cannot see. Running it on gemini-2.5-flash reproduces the documented blind spot on demand — Gwet AC1 0.961 and mean instability 0.031 on a judge that is wrong on 5 of 24, including two verbatim-restatement anchors.

Fixed

• orq-evaluator-alignment: create_datapoints sent the wrong request body. POST /v2/datasets/{id}/datapoints takes a top-level JSON array, but build_create_datapoints_body wrapped the rows as {"datapoints": [...]}, which the API rejects with 400 invalid_request_body ("expected array, received object"). This broke dataset save-back (seed_inputs.py save) entirely. Verified against the live API.
• orq-evaluator-alignment: seed.unresolved_variables did not mirror judge.make_replacements despite its docstring saying so. It had no branch for the reference | expected | expected_output leaves, so an evaluator declaring {{log.reference}} rendered correctly at judge time while every one of its dataset rows was silently skipped at pull time.

[2.4.0] - 2026-08-26

Added

• orq-evaluator-alignment: multi-type judge alignment. The measure → annotate → rewrite → retest loop now covers all four verdict spaces — boolean (flip-rate), categorical (label entropy), numeric (score spread) and string (exact-match entropy) — on one shared 0..1 instability scale. String is detect + annotate only (no rewrite/retest). The annotation UI, agreement gate, recommend/aggregate stages and the created evaluator all carry the source's verdict space through.
• orq-evaluator-alignment: grey-zone assessment stage (RES-980) — cluster the most-unstable "confuser" datapoints, drive a policy-driven Q&A, and render the resolved policy as the free-text guidance the existing rewrite step already consumes.
• orq-evaluator-alignment: a context budget for the grey-zone payload, so evaluators with very large judged inputs (transcripts, RAG contexts, documents) cluster honestly. Each confuser's input is now windowed head+tail with a per-{{variable}} fair share — a long {{query}} can no longer starve {{output}} — and carries explicit elision accounting (input_chars_shown of input_chars_original) that the conductor is required to disclose before open-coding. assemble clamps the payload to the longest prefix fitting grey_zone_max_tokens (default 60k) and reports what it dropped; build_queue projects the same number at the point the user picks how many examples to review, so the budget rides on that existing decision instead of adding a gate. New --max_tokens flag and grey_zone_top_k / grey_zone_max_chars / grey_zone_max_tokens config keys.

Changed

• orq-evaluator-alignment: run-directory artifacts (aggregated.md, new_prompt.md, and every JSON/JSONL) are written atomically (temp-then-replace) so an interrupted step can no longer truncate an existing artifact. aggregate.py and the grey-zone apply step now warn instead of silently clobbering when both write aggregated.md.
• orq-evaluator-alignment: judge model-slug resolution de-duplicates a display alias shared by two providers (first routable refId wins, with a warning) so alias→refId normalisation can't non-deterministically re-introduce the wrong-provider 403.

Fixed

• orq-evaluator-alignment: the grey-zone payload no longer silently misrepresents a large judged input. It previously concatenated every {{variable}} and cut the result head-first at 600 chars, so for a long conversation plus a short answer the conductor saw only conversation preamble — the answer the judge actually scored was elided entirely, with nothing but a trailing ellipsis to show for it. The grey_zone_top_k / grey_zone_max_chars keys the skill documented and read were also absent from config.toml, leaving the payload effectively uncapped.
• orq-evaluator-alignment: categorical and numeric instability now clamp to [0, 1] — a judge that emits labels/scores outside the declared contract can no longer push the score past 1.0 and skew downstream banding.
• orq-evaluator-alignment: creating an aligned copy of a numeric evaluator preserves its scale whether the source stores output_type as number or numeric (the latter previously dropped the scale silently). build_queue --count 0 now means "no flipped items" (low-flip sample only) instead of being folded into "all". retest fails with a clean message on a string evaluator instead of a raw ValueError.

[2.3.4] - 2026-08-25

Changed

• skills-lock.json: folder hashes are computed over a byte-ordered file list instead of localeCompare, which is ICU- and locale-dependent. Byte order also matches upstream computeSkillFolderHash, whose default .sort() is UTF-16 code-unit order, so the lock now sits closer to what npx skills computes rather than merely being deterministic. No skill content changed; 13 of 15 hashes moved because every skill with a resources/ directory sorted resources/x.md before SKILL.md under the old comparator. The hash is a skip-cache key and never an integrity check, so the only consumer-visible effect is one needless reinstall of those 13 skills on the next npx skills sync.
• install-sanity compares the installed skill name set against the skills-lock.json keys instead of asserting count >= expected. Dropping one real skill while installing one phantom left the count unchanged and the job green.

Added

• CI cross-checks all 15 skills against skills-ref, the Agent Skills reference validator. validate-skills.mjs is this repo's own reading of the spec; the spec's own implementation was the one check nobody automated, so a divergence between them surfaced through a consumer rather than a build. (PYTHONUTF8=1 — skills-ref reads SKILL.md with the locale encoding.)
• Frontmatter value shapes are validated, not just field names. A permitted name with a wrong value type makes a skill skippable under Agent Plugins §7.1 exactly as an unknown field does, and the value was previously never inspected: metadata: [a, b] passed. metadata must now be a map of key: value pairs, and name, description, license, compatibility and allowed-tools must each be a string.
• tests/skills.md is enforced as the fifth registration surface. CLAUDE.md has always listed it; nothing checked it, so a skill could ship with nothing smoke-testing it and every job green.
• Registration surfaces reject duplicates. All three collected names into a Set, which collapses a repeat, so "listed exactly once" was never enforced.
• The README.md skills table is checked label against link target, matching the AGENTS.md path list. A row pointing at another skill's SKILL.md satisfied both name diffs while every harness loaded the wrong file; a row that has lost its link is reported rather than silently skipped.
• The self-test suite asserts the committed skills-lock.json validates clean against the checkout. Nothing pinned this, so the hash-ordering change above went in stale and surfaced as a red pipeline instead of a failing test.

[2.3.3] - 2026-08-18

Fixed

• orq-red-team, orq-compare-agents, orq-simulate-agent, orq-invoke-deployment, evaluatorq: preflight existence checks now verify with the run key via REST/SDK instead of treating the MCP as authoritative. All five skills reference a shared run-key preflight doc (orq-red-team's inline copy from PR #45 replaced with a reference). orq-build-agent, orq-optimize-prompt, and orq-generate-synthetic-dataset were audited and left unchanged — they use MCP for browsing/discovery, not for run-gate existence checks.
• orq-invoke-deployment: intro text reconciled — "do NOT check the key" clarified to "do NOT prompt for it" (the new step 3 verification is intentional, not contradictory).
• orq-compare-agents, orq-simulate-agent: added Bash(curl:*) to allowed-tools. Both skills instruct the agent to verify a key with the run key via REST, but neither granted any tool able to make the call.
• orq-invoke-deployment: fixed the Get Config reference link — the get-deployment-config slug returns 404; the correct slug is get-config.

[2.3.2] - 2026-08-14

Fixed

• orq-evaluator-alignment: the judge span is now located by a normalised parent link (parent_span_id → parent_id → attributes.orq.bridge.parent_span_id). On a Responses-API trace — the exact shape 2.3.1 added support for — orq leaves parent_span_id unset and puts the link on parent_id, so the previous scoping found no child and silently fell back to "any judge span in the trace". In a trace with two evaluator calls, both rows then took the first judge span's rendered prompt, recovered content and pinned judge model. All three readers (rendered IO, judge model, hollow-guard content sources) now share one scoper, and it returns nothing rather than guessing between several unparented candidates.
• orq-evaluator-alignment: {{reference}}/{{expected}}/{{expected_output}} survive template recovery. The judge renderer mapped them to the row's reference, but the trace-side mapper did not and the fallback path hard-set reference = '', so a template like Compare {{log.output}} with {{log.expected_output}} re-judged against a blank ground truth. Both directions now share one suffix table (lib/content.py).
• orq-evaluator-alignment: a parts-shaped conversation (Responses API) renders its real text into a {{messages}} variable. The renderer read only each message's flat content, so the conversation the 2.3.1 extraction fix had just recovered was re-judged as a list of empty user: lines.
• orq-evaluator-alignment: a conversation-only evaluator (no query/output, only {{messages}}) is extracted instead of hollowed — the structured-root gate required query or output, so 2.3.1's "carries the conversation" fix could not fire on the one evaluator shape that needs it. A root carrying only a conversation still does not claim the output slot: the judge-span path runs whenever query and output are both empty.
• orq-evaluator-alignment: --force no longer overrides a shape-gap abort. It is the auth/rate-limit override, and the shape-gap message explicitly tells the operator not to force it (forcing writes empty rows that then read as perfectly stable datapoints) — it now discounts the span-detail half only.
• orq-evaluator-alignment: source_index again means "row N of traces.jsonl". 2.3.1's degraded-row filter renumbered the surviving rows, so annotations carried across a rerun with a different --include_degraded would land on different datapoints.
• orq-evaluator-alignment: when the degraded filter empties the set, the error names the filter (and --include_degraded) instead of telling the operator to run fetch_traces.py first — the file was full and fine.
• orq-evaluator-alignment: fetch_evaluator.py --with_traces exits non-zero when the chained trace fetch persists no datapoints. It caught the abort, logged remedies and returned normally, so automation read a hollow run as a successful step 1 and failed later on a missing traces.jsonl.
• orq-evaluator-alignment: run_experiment.py reads new_prompt.md as utf-8-sig like every other human-editable artifact; a BOM from a Windows editor reached the judge prompt on the retest path.
• orq-evaluator-alignment: hollow_debug.json's span inventory prints the normalised parent alongside the raw fields — it previously showed the Responses-API judge span as unparented in the very dump meant to explain the shape.

Changed

• CI runs the skills' pytest suites (skill-tests job). They existed and were green only on whoever last ran them locally, so every invariant they pin was unprotected. A skill opts in by shipping tests/requirements.txt.

[2.3.1] - 2026-08-13

Fixed

• orq-evaluator-alignment: trace extraction now reads the orq Responses API span shape (span.responses, with text under messages[].parts[].content) and no longer treats a reference-only root span as content. Previously every datapoint from a Responses-API evaluator came back hollow and the scan aborted with a misleading auth/rate-limit guess.
• orq-evaluator-alignment: the hollow-datapoint guard now distinguishes a span-detail fetch failure (auth/rate-limit) from an unrecognised span shape. It tells the two apart by intersecting the spans a row's content is actually read from — the root span and this eval span's own judge spans (scoped as _judge_io scopes them, so a repeated evaluator's 429'd judge span isn't borrowed) — with the spans whose detail fetch failed, so a 429 on a content-source span isn't misfiled as an extractor bug; it writes hollow_debug.json with the offending span shape on a shape-gap abort instead of guessing.
• orq-evaluator-alignment: _structured_io now carries the root span's conversation messages instead of hardcoding None, for any row it already returns (one carrying a query or an output). An evaluator whose judge template has a {{messages}}/{{history}}/{{conversation}} variable was re-judging with a blank conversation, and because the row's output was non-empty nothing flagged it — a silent quality drop behind a green pipeline. (A conversation-only evaluator — no query or output at all — is fixed in 2.3.2; this gate still rejected it.)
• orq-evaluator-alignment: a datapoint whose query/output extracted cleanly is no longer marked degraded merely because one of its spans' detail fetch was rate-limited. The stability step was dropping such usable rows with a false "empty output, cannot be meaningfully re-judged" reason; only genuinely-empty rows are degraded now.
• orq-evaluator-alignment: Windows robustness — the cosmetic run-dir rename is guarded against OneDrive/WinError 32 locks, and the artifacts feeding evaluator creation (new_prompt.md, aggregated.md, and every JSON/JSONL read) are read as utf-8-sig so an editor-written BOM no longer crashes reads or corrupts the created prompt's first line. (The retest path's own read of new_prompt.md was missed; fixed in 2.3.2.)
• orq-evaluator-alignment: the stability step now skips degraded/hollow datapoints by default (they waste judge calls and flatten the flip metrics); --include_degraded keeps them.

[2.3.0] - 2026-08-11

Added

• Root plugin.json conforming to Agent Plugins 1.0.0 — one portable manifest ($schema, closed field set) any spec-conformant client can load from the repo root. Skills ship from the fixed skills/ location; mcp.json stays as-is for now (a 1.0.0 client disables MCP for the plugin and still loads skills; conformant MCP config is a follow-up).

• CI validates the root manifest with ajv against tests/schemas/agent-plugins-1.0.0.plugin.schema.json, a vendored copy of the published 1.0.0 schema, and asserts every tracked symlink resolves inside the plugin root (§4.1 path containment). A nested manifest carrying the 1.0.0 $schema is rejected — the repo root must be the only Agent Plugins root.

• tests/scripts/validate-skills.test.sh covers every invariant the validator enforces, not just the stale lock hash: each case breaks one thing in a clean fixture and asserts both a non-zero exit and the message naming the check. A git-backed fixture means the tracked-file checks (§4.1 containment, stray skills, one plugin root) are exercised too — previously they were verified only by hand during review, which is how two of them shipped passing while not checking.

Removed

• disallowed-tools from all 14 skills that carried it. It is not an Agent Skills frontmatter field — the field set there is closed (name, description, license, compatibility, metadata, allowed-tools) — and Agent Plugins §7.1 requires a client to skip a skill that does not conform to that spec. skills-ref, the reference validator the spec links to, reported all 14 as errors, so this release would otherwise have advertised a portable plugin from which a conformant client loads one skill out of fifteen. This loosens the tool restrictions added in 2.2.3: delete_* orq tools were removed from the pool while those skills were active, and are now merely un-pre-approved, so they prompt instead of being unavailable. No skill's body calls them. Restoring the guarantee needs a mechanism the Agent Skills spec actually has.

Fixed

• CI validates every skill against the Agent Skills spec, not just against this repo's own conventions: closed frontmatter field set, name pattern and length, description and compatibility limits. The suite passed all 15 skills while 14 of them were non-conformant, because it only ever checked what this repo had thought to check.
• A git ls-files failure no longer skips the tracked-file checks in silence. The skip exists for a non-git tree (a test fixture); when .git is present and git fails anyway — a corrupt index, detected dubious ownership under a containerised runner — the run now fails and quotes git, instead of reporting success for three checks that never ran.

Changed

• .agents/plugins/marketplace.json source.path now points at the repo root (./) instead of the removed plugins/orq copy.
• docs/install-codex.md: the personal install used an absolute source.path, which Codex silently ignores — it resolves source.path relative to the marketplace root and requires a ./-relative path inside it. The clone now lives under $HOME and the entry is ./.codex/plugins/orq.
• Manifest version sync now covers the root plugin.json alongside the three per-harness manifests.
• Install docs now name the repository orq-ai/assistant-plugins throughout; several still pointed at the pre-rename orq-ai/orq-skills, which worked only via GitHub's redirect.

Removed

• plugins/orq/ — a duplicate Codex surface wired to the repo root via symlinks that escaped its own plugin root (spec §4.1 violation). The repo root is now the single orq skills plugin root; plugins/trace-hooks/ remains a separate, independently versioned Claude Code plugin. It was the documented target of the Codex personal install (cp -rL plugins/orq …), so docs/install-codex.md moves to the repo root in the same release; existing installs are unaffected, which is why this is MINOR rather than MAJOR.

[2.2.3] - 2026-08-07

Added

• Curated allowed-tools on five read-dominant skills (orq-analyze-trace-failures, orq-optimize-prompt, orq-run-experiment, orq-build-evaluator, orq-build-agent) — the orq* wildcard (which matched no tool and pre-approved nothing) replaced with explicit read/search orq tools for prompt-free lookups. Writes are not pre-approved: create_*/update_*/invoke_* and shell commands still prompt on these five skills. Each skill states this rationale under its title.
• disallowed-tools on the same five skills disables every delete_* orq tool outright while the skill is active — these skills never delete, so the tools are removed from the pool, not merely left prompting.
• allowed-tools on orq-evaluator-alignment, the last skill in the suite that declared none. It drives the orq API through its own bundled uv run scripts/*.py toolkit rather than orq tools, so its allowlist is narrow (Bash(uv run:*) plus file and question tools); the delete_* set is disallowed for consistency.
• Finished the orq* sweep across the remaining nine skills: the dead wildcard is replaced with the orq tools each skill's body actually calls, and disallowed-tools now covers every skill except orq-manage-skills (which legitimately deletes) and orq-generate-synthetic-dataset (which legitimately deletes datapoints, so only that one delete stays available — and only behind a prompt).
• Scoped Bash to the commands each skill actually runs (Bash(orq:*), Bash(eq:*), Bash(curl:*), …), and dropped it entirely from the four skills with no shell examples. Unscoped Bash in allowed-tools pre-approved every shell command without a prompt, which was a wider grant than the delete_* tools this release set out to close.

Changed

• orq-manage-skills: create_skill, update_skill, and delete_skill removed from allowed-tools (reads stay pre-approved) so every write and delete prompts; its delete workflow now uses the repo's ## Destructive Actions + AskUserQuestion convention with per-entity confirmation and no bulk deletes without listing every item first.

[2.2.2] - 2026-08-06

Added

• Evaluatorq v1.10 features now referenced: LLM-jury and pairwise judging (llm_jury, llm_jury_pairwise, PairwiseComparator), eq dashboard run browser, eq sim ui, CrewAI and Pydantic AI target wrappers, redteam --strategy/--delivery-method/--executive-summary flags, and the ORQ_WORKSPACE/ORQ_UI_BASE_URL dashboard deep-link variables.

Fixed

• Evaluatorq-family skills re-verified against the current evaluatorq release (1.11.x) (RES-1220):
  • CLI flags aligned to 1.11: eq sim generate writes via --datapoints/-d; eq sim simulate reads via --input/-i and writes via --results/-r; redteam report flags are --report/--artifacts-dir/--report-md/--report-html (the pre-1.11 --save-report/--output-dir/--export-md/--export-html no longer exist); Python red_team(artifacts_dir=) likewise.
  • target_callback= removed everywhere — it does not exist in the library (target= is the only name; passing it raises TypeError).
  • simulate()/generate_and_simulate() report kwargs documented (report=, executive_summary=, save=, orq_results_path=); eq sim subcommand list completed (from-traces, upload-dataset, validate with validate-dataset as deprecated alias); OWASP table carries ASI10 Rogue Agents; async scorer example uses await orq.evals.invoke_async(...) with env-derived credentials; orq-ai-sdk added to the install table.
  • orq-red-team: credential-priority rule corrected — ORQ_API_KEY (gateway) wins when both keys are set, not OPENAI_API_KEY; the "uv run + .env credential trap" section, conflict guidance, and troubleshooting rows rewritten around the verified routing order. Removed the nonexistent [ui] extra (eq redteam ui ships inside [redteam]; openai/typer are core deps).
  • evaluatorq: eq sim quick references corrected — the target flag is --target agent:<key> (no --agent-key), datapoint-driven runs use eq sim simulate (not run), and the generate→export example now includes the required simulate step. Dashboard stage-file name fixed (03_summary_report.json).
  • orq-simulate-agent: simulate()/generate_and_simulate() examples no longer pass agent_key= (a TypeError — the parameter is target=); red_team import path corrected to evaluatorq.redteam; raw-model targets use OpenAIModelTarget (the llm: prefix is rejected); run store documented as CWD-relative .evaluatorq/ with $EVALUATORQ_DIR override.
  • orq-compare-agents: the my.→api. staging URL rewrite no longer exists (removed in evaluatorq v1.10.0) — ORQ_BASE_URL is used verbatim with default https://my.orq.ai; the obsolete workaround was replaced with a historical note.

[2.2.1] - 2026-08-06

Changed

• orq-cli: command map re-verified live against the 4.13 CLI and rewritten to describe the 4.13 surface only — no version deltas or cross-version caveats. Covers the annotation-queues group (eval corrections / unified annotations), per-trace drill-down (traces get / list-spans / get-span), and notes there is no experiments group. The map is scoped to the supported surface; some CLI groups are intentionally not covered. (RES-1210)
• orq-cli: caveats re-tested live on 4.13 — plain strings accepted by generated flags (double-JSON-quoting workaround deleted); no client-side enum validation (documented as such: read the server error).
• orq-cli: the global projection flag is -j/--jmespath — 4.13 removed -q/--query, and body fields no longer shadow globals (colliding body fields get a body- prefix; a body query field is plain --query, full-text search). Every example updated; the shadowing section replaced with the --query-is-not-a-projection trap and the -q-muscle-memory error.
• MCP tool-name rot fixed across resources: evaluator_get → get_llm_eval/get_python_eval (4 files); phantom list_registry_keys row removed from orq-invoke-deployment; ## Companion Skills heading case normalized.
• orq-cli: "MCP tools or the CLI?" decision table extended with the coverage split — MCP-exclusive (experiments, docs/entity search) vs CLI-only (schedules, identities, projects, API keys, webhooks, knowledge bases, memory stores, files).
• All 12 MCP-primary skills now cross-reference orq-cli for anything that must run again without an agent present (CI, cron, scripts, bulk), pointing at the canonical decision table. (RES-1163)

Fixed

• orq-red-team: stale MCP references mcp__orq-mcp-global__agent_get/agent_list corrected to mcp__orq-workspace__get_agent/search_entities — the old server and tool names no longer exist.

[2.2.0] - 2026-07-31

Added

• orq-cli skill — drive the orq command-line interface end to end: verify the install (npm i -g @orq-ai/cli or install.sh, which lands in the often-not-on-PATH ~/.orq/bin), authenticate via OAuth session or ORQ_API_KEY, scope to the right workspace, discover commands with --help, and run them with --json plus -q JMESPath projections. Covers the profile model, the auth whoami / workspace list / doctor JSON shapes, the four request-body input paths (typed flags, stdin, --from-file, --example) and bartolo shorthand, the orq request escape hatch, and a symptom→cause troubleshooting table. Companion resources/command-map.md carries the full v4.12.15 command tree, global flags and their env-var twins, verified response field names for traces/spans/agents, and JMESPath recipes — including resolving the active workspace key (orq auth whoami --json -q active_workspace_key --raw) for app deep-links, with a warning not to shell out to the CLI from library request paths. Registered in agents/AGENTS.md, README.md, tests/skills.md, and skills-lock.json. (RES-1140)

[2.1.1] - 2026-07-22

Fixed

• orq-evaluator-alignment — corrected the judge_model router slug format documented in SKILL.md. It previously stated the router requires <provider>/openai/<model> with a literal openai/ segment "always required, whatever the provider". That form 404s on the router (anthropic/openai/claude-haiku-4-5 → 404) and every user following it with a non-OpenAI provider hit a bare 404 mid-run. The correct form is the plain <provider>/<model> single-prefix slug (e.g. anthropic/claude-haiku-4-5, google/gemini-2.5-flash) — verified live against the Orq router (chat-completions and responses endpoints, openai/anthropic/google all route) and consistent with the agent config and the MCP create_llm_eval tool. Added a regression note to tests/skills.md. (RES-1145)

[2.1.0] - 2026-07-02

Added

• orq-evaluator-alignment skill — align, calibrate, or improve an existing binary Pass/Fail LLM-as-a-judge (orq evaluator) so its verdicts match human judgment. Measures judge self-consistency (flip-rate) via repeated runs, surfaces the most ambiguous datapoints for human annotation, rewrites the judge prompt from the labels, and creates the new evaluator only after human approval. Complements orq-build-evaluator (build from scratch) and orq-optimize-prompt (fix via prompt tweaks). Every step script is self-contained via PEP 723 inline dependencies, so uv run scripts/<name>.py builds its own environment with no repo checkout or uv sync. Registered in agents/AGENTS.md, README.md, tests/skills.md, and skills-lock.json.

[2.0.0] - 2026-06-24

Changed

• BREAKING — renamed the Claude Code marketplace from assistant-plugins to orq-claude-plugin (.claude-plugin/marketplace.json name). Install commands change from claude plugin install <plugin>@assistant-plugins to @orq-claude-plugin. The GitHub repo path (orq-ai/assistant-plugins) used by claude plugin marketplace add and npx skills add is unchanged, so Codex/Cursor/Warp/Gemini installs are unaffected — only Claude Code's @<marketplace> handle changed. Existing @assistant-plugins installs must re-add the marketplace. Updated install docs in README.md, plugins/trace-hooks/README.md, plugins/trace-hooks/CLAUDE.md (dev symlink path), and plugins/trace-hooks/tests/README.md; the orquesta-web docs (claude-code.mdx) were aligned to the already-published @orq-claude-plugin instructions in the changelog/tutorial/skills pages.

[1.0.0] - 2026-06-11

Changed

• BREAKING — namespace every skill under the orq- prefix to prevent collisions with similarly-named skills from other plugin marketplaces and make the suite discoverable as a set. Renamed 11 skill directories and their frontmatter name: analyze-trace-failures, build-agent, build-evaluator, compare-agents, generate-synthetic-dataset, invoke-deployment, manage-skills, optimize-prompt, run-experiment, setup-observability, simulate-agent → orq-*. orq-red-team already carried the prefix; evaluatorq is left as-is (its name already reads "orq"). Updated all cross-references in agents/AGENTS.md (path list + <available_skills>), README.md, tests/, skills-lock.json, and inter-skill companion references.
• BREAKING — renamed the /manage-skills slash command to /orq-manage-skills (commands/manage-skills.md → commands/orq-manage-skills.md, frontmatter name) to match its 1:1 skill.

[0.5.2] - 2026-06-11

Fixed

• orq-red-team: the "Verify the target exists" preflight now exports the resolved ORQ_API_KEY on success, so the verify call and the subsequent eq redteam run use the same key. Previously a key present only in ./.env would pass the verify curl (which sourced it explicitly) but the run — eq reads ORQ_API_KEY from the environment and does not auto-read ./.env when run directly — would get an empty key and fail deep with a cryptic 401/404.

[0.5.1] - 2026-06-10

Changed

• orq-red-team: the "Verify the target exists" preflight now checks via REST/SDK with the key the run actually uses (ORQ_API_KEY from the env, else the project .env) instead of the MCP, whose separately-configured key is often in a different project — an MCP miss isn't proof the target is absent. Covers both agent: (GET /v2/agents/{key}) and deployment: (POST /v2/deployments/get_config) targets, and falls back to asking the user when no key resolves.

[0.5.0] - 2026-06-08

Added

• evaluatorq: new skill for writing evaluatorq evaluation scripts (Python + TypeScript) and operating the eq CLI. Covers single-agent evaluation, custom scorers, built-in evaluators. Routes to red-team skill for eq redteam and simulate-agent skill for eq sim.
• evaluatorq/resources/cli-reference.md: full flag reference for eq redteam and eq sim, output file schemas, and common usage patterns.

[0.4.0] - 2026-06-06

Added

• orq-red-team: eq discovery ladder before installing — probe local/cheap options first (PATH → project .venv/bin/eq → uv run --package evaluatorq orqkit workspace → python3 -m evaluatorq) and use the first hit; install only as a last resort, preferring a project-local venv (uv pip install into .venv) over a global uv tool install, and avoiding global pip (PEP 668 / --break-system-packages breakage). Every resolved invocation is quote-clean and reusable as a $EQ prefix; the preflight CLI check honors it via ${EQ:-eq}.
• orq-red-team: document the uv run + .env credential trap — uv injects an env-file only when opted in (UV_ENV_FILE set or --env-file passed); it does not auto-read ./.env from a bare uv run (verified on uv 0.11.19). When such an env-file holds OPENAI_API_KEY, uv re-loads it after a shell unset, flipping routing to direct OpenAI and breaking gateway model strings (openai/gpt-5-mini → 401 Incorrect API key). Includes a default-vs---no-env-file detector that also surfaces UV_ENV_FILE (presence only, never prints the key), a decide-don't-auto-strip guide that surfaces the conflict and lets the user choose (a key in the env usually means they want it), Fix A (env -u OPENAI_API_KEY uv run --no-env-file … — strip the key and block uv from re-adding it, no temp file), Fix B (run eq off PATH), and a note that plain env -u without uv run is unaffected.
• orq-red-team: pre-run "Verify the target agent exists" step — check the --target key via orq MCP agent_get/agent_list, REST GET /v2/agents/{agent_key} (curl), or SDK agents.retrieve, so a wrong key fails fast instead of deep in the run with Agent not found. Documents the project-scoping caveat (MCP key may differ from the CLI key; a hit confirms existence, a miss is conclusive only when the checking credential shares the agent's project — verified live).
• orq-red-team: troubleshooting rows for the uv run/.env 401, the mid-run Agent not found, and the discovery-first eq: command not found fix.
• orq-red-team (resources/python-sdk.md): document OrqResponsesTarget — the hosted-agent target wrapping orq's Responses v3 API. Covers the import path (evaluatorq.openresponses.target), constructor signature, a red_team() example, and require_orq=True gateway routing. Notes it's usually built for you by the openresponses backend when you pass "agent:<key>"; hand-build only for a custom client/instructions/timeout/retry.
• orq-red-team: "Plan the run — decide parameters with the user" guided flow before the first invocation — step through mode (dynamic/static/hybrid, with an upside/downside table), datapoint budget (--max-dynamic-datapoints / --max-static-datapoints as the main cost lever, with smoke-test vs assessment guidance and the per-category multiplier), vulnerability scope (an agent-surface → OWASP-ASI/LLM category map, noting the framework auto-prunes non-applicable categories so over-picking is cheap), and delivery (one-off CLI vs a reusable script for the fix → re-run loop, CI, and baking in the .env/uv credential handling), then confirm the choices and state the coverage gap before assembling the command.

[0.3.0] - 2026-06-06

Changed

• Rename the red-team skill to orq-red-team for clearer invocation and to namespace it under orq. Skill directory skills/red-team/ → skills/orq-red-team/, frontmatter name, and all references in README.md, agents/AGENTS.md, and tests/skills.md. (Treated as MINOR rather than MAJOR: 0.2.0 was never released/tagged, so the old name has no external consumers.)

[0.2.0] - 2026-06-05

Added

• red-team: resources/python-sdk.md progressive-disclosure reference for the evaluatorq.redteam Python API — covers red_team(), OpenAIModelTarget / the agent:<key> string target / custom AgentTarget, a raw-model worked example (the case the CLI cannot do), and programmatic RedTeamReport handling.
• red-team: document external-framework targets in resources/python-sdk.md — LangGraphTarget ([langgraph]), OpenAIAgentTarget ([openai-agents]), and CallableTarget (bundled, the escape hatch for any async def(prompt) -> str), plus LangChain/Vercel AI SDK pointers. Covers red-teaming a non-orq agent, which the CLI cannot do.
• red-team: document generate_recommendations=True and report.focus_area_recommendations (SDK-only LLM remediation) in both SKILL.md and resources/python-sdk.md.
• red-team: add "Acting on results — next steps" guidance for coding assistants — how to mine report.results[] (filter vulnerable, read attack.category/attack_technique, the transcript, and evaluation.explanation), prioritize by summary.by_technique/by_category, map failure patterns to concrete fixes, and close the re-run feedback loop. jq recipes in SKILL.md; the Python equivalent plus focus_area_recommendations handling in resources/python-sdk.md.
• simulate-agent skill: run multi-turn agent simulations using evaluatorq's first-class primitives (simulate(), generate_and_simulate(), wrap_simulation_agent()). Covers the real Persona schema (patience / assertiveness / politeness / technical_level scalars, communication_style, background, optional emotional_arc and cultural_context), Scenario schema (goal, criteria-driven judge termination, starting emotion, conversation strategy, edge-case flag), three target shapes (agent_key, target_callback via from_orq_deployment / from_chat_completions, custom AgentTarget), and where outputs land (OTel spans auto-emitted to orq.ai, SimulationResult in memory, auto-uploaded Experiments via evaluatorq() routing, JSONL export). Resources: persona-scenario-template.md, simulation-loop.md, redteam-mode.md. RES-732.

Fixed

• red-team: correct ASI01 label — it is Agent Goal Hijacking, not prompt injection (prompt injection is LLM01). Reframed the worked example and category guidance, and added the full OWASP-ASI (ASI01–10) / OWASP-LLM (LLM01–09) name mapping.
• red-team: correct the credential model — routing is decided by which env var is set (OPENAI_API_KEY → direct OpenAI with bare model names; else ORQ_API_KEY → orq gateway with provider-prefixed names like openai/gpt-5-mini), not by the model string. OPENAI_API_KEY wins if both set; ORQ_API_KEY always required for agent:/deployment: targets.
• red-team: use openai/gpt-5-mini in examples and drop the backwards "switch to gpt-4o" troubleshooting advice (the default gpt-5-mini is newer).
• red-team: remove invented framework labels ("OWASP Agentic 2026" / "OWASP LLM 2025"); use the real OWASP-ASI / OWASP-LLM identifiers.
• red-team: fix install instructions to pip install 'evaluatorq[redteam]' (and note the [ui] extra for the dashboard).

Changed

• red-team: invocation preflight checks credentials before any eq redteam run — hard-fail if no LLM credential at all (OPENAI_API_KEY or ORQ_API_KEY), and check-and-warn for ORQ_API_KEY. Document that ORQ_API_KEY is not strictly required (raw-model runs work with OPENAI_API_KEY alone) but is needed for orq agent:/deployment: targets, gateway LLM routing, and uploading results to orq (experiment_url). The agent halts only when an orq-agent target is requested without the key.
• red-team: trim the flag table to first-run essentials and defer the full set to eq redteam run --help; document the deployment:<key> target form, the eq redteam validate-dataset pre-flight, and the --system-prompt flag.
• red-team: add a Constraints note (and --no-cleanup-memory flag row) that dynamic runs against a memory-backed agent write entities into its memory store (cleaned up unless --no-cleanup-memory); no-op for memory-less agents, raw models, and static mode.

[0.1.0] - 2026-06-04

Added

• red-team: new skill for invoking the orq red teaming library — adaptive attacks, dataset runs, hybrid mode, OWASP Agentic/LLM coverage, and ASR reporting.
• manage-skills skill — CRUD workflow for the orq.ai Skills entity (formerly Prompt Snippets), backed by /v2/skills. Covers list, get, create, update, soft-retire (tag as retired), and delete via the *_skill MCP tools. Includes authoring guidance (display_name, description, tags, project_id, path) and disambiguates the platform Skill entity from this repo's code-assistant Orq Skills and from the unrelated A2A AgentCard.skills array.
• manage-skills: documents both {{skill.<display_name>}} (canonical) and {{snippet.<display_name>}} (backward-compatible alias, falls back to the Skill whose display_name matches) as the template placeholders for consuming Skills inside prompts and agent instructions.
• manage-skills: reference-scan-before-delete workflow — paginates search_entities, fetches each candidate's body with get_deployment / get_agent / get_skill, and substring-matches both {{skill.<display_name>}} and {{snippet.<display_name>}} to surface consumers before any destructive operation. Defaults to tagging with retired (soft-retire) when references are found.
• manage-skills: rename-breaks-references warning on display_name updates — runs the same reference scan before any rename and offers to fan out updates in the same session.
• manage-skills: documents GET /v2/skills cursor pagination (limit / starting_after / ending_before) and the lack of server-side filters; pushes project_id / tags / display_name filtering to the client.
• manage-skills: anti-pattern guidance against +NEVER+ / "you MUST refuse" prose constraints in instructions — recommends MCP tool gates for hard guardrails.
• manage-skills: error-handling guidance for create_skill AlreadyExists (offers either a renamed create or update_skill against the existing Skill).
• /manage-skills slash command — routes to list / get / create / update / retire / delete phases.

Fixed

• red-team: rewrite skill to target the real evaluatorq package (orqkit/packages/evaluatorq-py) and eq redteam CLI instead of the legacy research/projects/red-teaming path.
• red-team: replace non-existent redteam run adaptive/dataset/hybrid subcommands with the actual eq redteam run --mode dynamic|static|hybrid interface.
• red-team: fix all CLI flags — --category repeatable (not --categories comma-separated), --max-dynamic-datapoints/--max-static-datapoints (not --max-attacks), --generated-strategy-count (not --generated-count), --parallelism default 10 (not 5), --output-dir (not --out).
• red-team: remove non-existent redteam report summarize command; replace with eq redteam runs / eq redteam ui <path>.
• red-team: fix default model to gpt-5-mini; add OpenAI gpt-4o as worked example model.
• red-team: fix env var section — document auto-detection order (OPENAI_API_KEY → direct OpenAI; ORQ_API_KEY → orq gateway); remove incorrect Azure credential guidance.
• red-team: fix output file naming — auto-named redteam-report-<target>-<ts>.json in .evaluatorq/runs/; use --save-report <path> for explicit path.
• red-team: add authorization guardrail — require explicit user confirmation before attacking any deployment.
• red-team: fix tests/skills.md scenarios to use correct eq redteam run --mode dynamic invocations.
• agents/AGENTS.md: remove trailing blank line after red-team <available_skills> entry.

[0.0.2] - 2026-04-21

Added

• invoke-deployment: document three deployment invocation patterns — variable substitution (inputs), message appending (messages), and mixed — with Python and curl templates for each.
• invoke-deployment: Phase 1 Step 3 now fetches GET /v2/deployments/<key>/config to discover {{variable}} placeholders before invoking.
• invoke-deployment: anti-pattern entry for passing inputs to a deployment with no matching {{variable}} placeholders (silently ignored).

Changed

• invoke-deployment: Phase 1 marked as one-time setup — discovery steps do not belong in production invocation flows.
• invoke-deployment: clarify inputs only substitute when matching {{variable}} placeholder exists in the prompt template.

Fixed

• invoke-deployment: replace insecure curl -sk with curl -s in deployment config fetch example (no TLS bypass).

[0.0.1] - 2026-04-21

Added

• invoke-deployment skill — invoke orq.ai deployments, agents, and models via Python SDK, Node.js SDK, or curl. Covers prompt variable substitution, multi-turn agent conversations via task_id, AI Router calls with provider/model format, and streaming.
• setup-observability skill — instrument LLM applications with orq.ai tracing. AI Router mode, OpenTelemetry/OpenInference mode, and the @traced decorator for custom spans.
• compare-agents skill — cross-framework agent comparisons using evaluatorq from orqkit. Compare orq.ai, LangGraph, CrewAI, OpenAI Agents SDK, and Vercel AI SDK head-to-head.
• Codex and Cursor plugin manifests (.codex-plugin/, .cursor-plugin/) plus Codex marketplace entry.
• tests/scripts/validate-plugin-manifests.sh — validates plugin JSON, field values, and symlink integrity.
• Smoke test scenarios in tests/skills.md for every skill.

Changed

• README install instructions expanded to cover 5 tools: Claude Code, Cursor, Codex, npx skills CLI, and manual clone.
• Python code templates now use os.environ["ORQ_API_KEY"] instead of os.environ.get() / os.getenv() to fail fast on missing key.
• Renamed instrument-app skill to setup-observability.
• AI Router base URL standardized to https://api.orq.ai/v2/router across all skills.