Skip to content

joao-aires/ai-development

v0.15.0MIT

Provider-independent AI application design, evaluation programs, testing methods and operations; models, context, tools, harnesses and protocols.

AI development plugin: application method, evaluation and operations

The ai-development plugin carries provider-independent guidance for building and assessing AI-enabled applications. It covers the application method, evaluation method, evaluation program and AI operations; it does not select a framework, provider or evaluation SDK.

What it owns

SkillOwns
AI developmentTask and system boundary, context, tools, harnesses, protocols and interaction design.
AI testingWhat to evaluate, acceptance at each boundary, case and grader design, and what the evidence can establish.
AI evaluationThe evaluation program: decision and claim, dataset and slice governance, baselines, gates, production sampling, red-team coverage and reporting.
AI operationsOperating signals, quality monitoring, provider limits, latency/cost, drift, rollout and recovery.

The evaluation pair is deliberately split. ai-testing is the method for choosing and interpreting evaluation evidence. ai-evaluation is the activity for running that method as a program across a change, release or product claim. Neither replaces application verification or the delivery decision.

How it composes

NeedUse
Build or change an AI applicationAI development
Decide what to evaluate and what the evidence meansAI testing
Plan, gate and report an evaluation programAI evaluation
Implement a selected Python/DeepEval suiteai-deepeval-testing when installed and chosen
Verify application behavior, contracts and persisted journeystest when installed
Implement traces, metrics and logsobservability when installed
Make the delivery decisionrelease when installed

Boundaries

  • Evaluation evidence supports a decision; it does not authorize merge, deployment or external publication.
  • A component score, simulated user or recorded output establishes a narrower claim than a live end-to-end journey.
  • Provider and SDK choices stay with the project. The plugin does not require a hosted evaluation service, synthetic data or a particular framework.
  • Durable artifacts and homes belong to the project's artifact system. Evaluation results update the existing change or verification record rather than creating a new artifact type.

Use the skill descriptions and the project's accepted requirements to select guidance; this README is orientation, not a required sequence.