orq-ai/orq

Write and run evaluatorq evaluation scripts (Python or TypeScript) for a single agent or deployment — custom scorers, built-in evaluators, and dataset-driven evaluation. For CLI workflows, use the companion skills: orq-red-team for eq redteam adversarial testing and orq-simulate-agent for eq sim multi-turn user simulation. Do NOT use when comparing multiple agents head-to-head (use orq-compare-agents) or when running orq.ai-native experiments only (use orq-run-experiment).

Read production traces, identify what's failing, and build failure taxonomies using open coding and axial coding methodology. Use when debugging agent or pipeline quality, investigating "why are my outputs bad?", or before building any evaluator — error analysis must come first. Do NOT use when you already have identified failure modes and need evaluators (use orq-build-evaluator) or datasets (use orq-generate-synthetic-dataset).

Design, create, and configure orq.ai Agents with tools, instructions, knowledge bases, and memory stores. Use when building new agents, attaching KBs or memory, writing system instructions, selecting models, or setting up RAG pipelines. Do NOT use for debugging existing agents (use orq-analyze-trace-failures) or comparing agents across frameworks (use orq-compare-agents).

Create validated LLM-as-a-Judge evaluators following best practices — binary Pass/Fail judges with TPR/TNR validation for measuring specific failure modes. Use when you need to automate quality checks, build guardrails, or measure a specific failure mode identified during trace analysis. Do NOT use when failures are fixable with prompt changes (use orq-optimize-prompt) or when failure modes are unknown (use orq-analyze-trace-failures first).

Drive the orq command-line interface — check the install, authenticate, select a workspace, and run read and write commands against any orq.ai resource (traces, agents, deployments, evals, prompts, datasets, projects, skills). Use when a task needs shell access to orq.ai, when a script or CI job must read workspace data as JSON, or when the active workspace key has to be resolved. Do NOT use for writing application code that calls orq.ai (use orq-invoke-deployment) or for guided evaluation workflows (use orq-run-experiment).

Run cross-framework agent comparisons using evaluatorq from orqkit — compares any combination of agents (orq.ai, LangGraph, CrewAI, OpenAI Agents SDK, Vercel AI SDK) head-to-head on the same dataset with LLM-as-a-judge scoring. Use when comparing agents, benchmarking, or wanting side-by-side evaluation. Do NOT use when comparing only orq.ai configurations with no external agents (use orq-run-experiment instead).

Align, calibrate, or improve an existing LLM-as-a-judge (orq evaluator) so its verdicts match human judgment — boolean, categorical, or numeric judges. Use when the user wants to "align my evaluator", "improve my eval", "my judge keeps changing its mind", "find ambiguous cases", or "annotate an evaluator" — i.e. they have an LLM judge that disagrees with human labels or is inconsistent. Measures judge self-consistency as one 0..1 instability score via repeated runs, groups the least reliable examples by what makes them hard and asks a few questions instead of making the user label every row, rewrites the judge prompt from those answers, and creates the new evaluator only after the human approves. If the evaluator ID isn't given, ask for it after triggering. Do NOT use to build an evaluator from scratch (use orq-build-evaluator) or to fix failures with prompt tweaks (use orq-optimize-prompt).

Generate and curate evaluation datasets — structured generation via dimensions-tuples-NL, quick from description, expansion from existing data, plus dataset maintenance through deduplication, rebalancing, and gap-filling. Use when creating eval data, expanding test coverage, or cleaning datasets. Do NOT use when sufficient real production data exists (use orq-analyze-trace-failures instead). Do NOT use for evaluator creation (use orq-build-evaluator).

Invoke orq.ai deployments, agents, and models via the Python SDK or HTTP API. Use when a user wants to call a deployment with prompt variables, invoke an agent in a conversation, or call a model directly through the AI Router. Do NOT use for creating or editing deployments/agents (use orq-optimize-prompt or orq-build-agent). Do NOT use for running evaluations (use orq-run-experiment).

Manage orq.ai Skills (the platform entity, formerly called Snippets) end-to-end — list, get, create, update, and delete Skills, plus authoring guidance (display name, description, tags, project scoping, path placement), and how Skills get consumed (the {{skill.<display_name>}} template placeholder inside prompts and agent instructions). Use when the user wants to create, audit, edit, retire, or hook up orq.ai Skills.

Analyze and optimize system prompts using a structured prompting guidelines framework — AI-powered analysis and rewriting. Use when a prompt needs improvement, experiment results show quality gaps, or you want a structured review of an existing system prompt. Do NOT use when production traces show failures (use orq-analyze-trace-failures first to identify patterns). Do NOT use to build evaluators (use orq-build-evaluator).

Invoke the evaluatorq red teaming CLI to run adversarial attacks against deployed agents or static datasets. Use when asked to "run a red team", "red team this deployment", "test my agent for vulnerabilities", "OWASP red team", or "check how my agent handles adversarial inputs". Do NOT use when you only need to build evaluators (use orq-build-evaluator) or analyze existing trace failures (use orq-analyze-trace-failures).

Create and run orq.ai experiments — compare configurations against datasets using evaluators, analyze results, and generate prioritized action plans. Use when evaluating LLM agents, deployments, conversations, or RAG pipelines end-to-end. Do NOT use without a dataset and evaluators. Do NOT use for cross-framework comparisons with external agents (use orq-compare-agents).

Set up orq.ai observability for LLM applications. Use when setting up tracing, adding the AI Router proxy, integrating OpenTelemetry, auditing existing instrumentation, or enriching traces with metadata.

Run multi-turn agent simulations using evaluatorq's first-class simulation primitives (simulate(), generate_and_simulate(), wrap_simulation_agent()). Drive an agent under test with a simulated user LLM, scored by a built-in JudgeAgent that decides per turn whether the goal was achieved or rules broken. Use when generating realistic multi-turn data for experiments, stress-testing conversational agents, or producing seed transcripts for dataset curation. Do NOT use when you have enough real production conversations (use orq-analyze-trace-failures). Do NOT use for adversarial red-teaming sweeps (use evaluatorq's built-in red_team() directly, see resources/redteam-mode.md).