Skip to content

cari-daal/ascend-op-eval

v0.4.0

Measure PyTorch or Triton operators, compare correctness and speedup, and recover evaluations.

GitHub

Ascend Op Eval plugin

This repository contains a client plugin for an independently deployed Ascend Op Eval HTTP service. plugins/ascend-op-eval/ is its location within this repository; ascend-op-eval/ is the plugin root. Metadata and skill content have different roles. Keep this package separate from the service source and Docker deployment files; distribute the contents of plugins/ascend-op-eval/ as the plugin:

plugins/ascend-op-eval/
├── plugin.json                  # Portable identity and metadata (canonical)
├── .codex-plugin/plugin.json    # Codex compatibility manifest
├── .claude-plugin/plugin.json   # Claude Code manifest
└── skills/ascend-op-eval/
    ├── SKILL.md
    └── scripts/ascend_op_eval.py

The root manifest follows OpenAI's portable plugin format. The hidden manifests provide the documented client compatibility locations; their skills: "./skills/" paths resolve from the plugin root. Keep all three manifests' shared metadata consistent. Skills and their scripts stay under the root skills/ directory, as specified by the Claude Code manifest reference.

The structure also follows actual open-source examples: OpenAI's Boltz CLI plugin uses .codex-plugin/plugin.json beside its skills, and Anthropic's example plugin uses .claude-plugin/plugin.json beside its components. No generic .plugin/ directory or additional wrapper is needed.

The skill bundles a Python 3.10+ standard-library CLI. No service package, Torch or Ascend installation is required on the client machine.

For pipe-utilization feedback use benchmark --method profiler. The service returns per-case kernel_metrics from the same timing capture; the CLI preserves those fields in its JSON output. Event timing remains the default. See the response semantics.

This example connects to a local service listening on 127.0.0.1:11907. For remote or containerized clients, set the URL to the address reachable from the client. Compose uses the service's configured private host address; the CLI itself has no default URL.

Configure the service URL and caller capability-file path:

export ASCEND_OP_EVAL_CLIENT_URL=http://127.0.0.1:11907
export ASCEND_OP_EVAL_CLIENT_CAPABILITY_FILE=/path/to/capability
python3 plugins/ascend-op-eval/skills/ascend-op-eval/scripts/ascend_op_eval.py verify \
  --definition /path/to/definition.json --workloads /path/to/workloads.jsonl \
  --triton /path/to/triton_impl.py --save-handle evaluation-handle.json

The client accepts the official definition and workload files directly. Workloads can be JSONL or a JSON array. The definition contains axes, ordered named inputs, outputs, optional name, and optional Python reference. A workload has uuid, axes, and named input specifications, for example:

{"uuid":"add-1024","axes":null,"inputs":{"x":{"type":"random","shape":[1024],"dtype":"float32"},"y":{"type":"random","shape":[1024],"dtype":"float32"}}}

The request seed defaults to 200 (--seed overrides it). A candidate is supplied with --triton; custom input generator source uses --custom-inputs. verify requires both the definition reference and candidate. benchmark and profile also support a single implementation. For Torch alone, omit --triton. For Triton alone, pass --triton-only alongside --triton to remove the reference from the submitted definition. Files are read as data and never executed locally.

For custom workloads, declare custom_inputs_entrypoint in the definition. With --triton-only, the client preserves the original reference source as custom_inputs_code for that factory; the submitted definition omits its mathematical reference. An explicit --custom-inputs file takes precedence.

triton_eval.py --version reports this client's version with no service, token or service package involved; the three plugin manifests declare the same version. version --url ... --capability-file ... reports the deployment's service version, implemented API contracts and build.

The client prints one JSON resource to stdout and progress to stderr. It retains HTTP admission, polling and recovery behavior: --save-handle records the URL, evaluation ID, idempotency key and session ID; use get EVALUATION_ID against the original URL to recover. Retry an ambiguous submission with unchanged files, options, session and idempotency key. Client timeout does not cancel accepted work.

Comparison benchmarks cache Torch baseline timing by default. Pass --no-baseline-cache to measure a fresh reference baseline. Single implementation benchmarks measure fresh. Cache switches apply to comparative timing.

A successful single implementation measurement reports correctness_verified: false; preserve that limitation when interpreting its timing or profile result. See the skill for recovery and result interpretation.

evaluate --checks correctness performance profile --profile-mode trace submits a combined request. Timing and kernel-duration trace share candidate preparation; profiler timing can reuse its capture. Add --wait-for performance to return completed timing while trace continues, or use progress EVALUATION_ID to read the current case measurements immediately. Performance snapshots may have complete: false and pending_case_indices even though they appear under completed_checks; --wait-for performance keeps waiting in that case. Existing get returns the final resource.

After a timeout, retained per_case numbers remain available with termination_reason: "timeout"; other interrupted runs use "interrupted". The resource's timed_out state is not a failed correctness verdict. Earlier correctness evidence survives, and unavailable correctness stays unknown. Interpret partial speedups only over their measured cases, not as a full-task score. Terminal JSON and the CLI's infrastructure/timeout exit status are preserved. See partial-result semantics.

Use explicit --correctness-cases, --performance-cases, and --profile-cases for combined requests. Shortcut commands accept --case-indices. Correctness supports --stop-on first_failure; subset passes are labeled incomplete. --correctness-from EVALUATION_ID explicitly requests validated reuse of an unchanged candidate's original complete correctness observation. The service refuses mismatched, stale or inaccessible evidence without silently rechecking.

Timing controls (--warmup, --repeats, --measurement-method, baseline-cache flags) also apply to evaluate. Deep profile controls include --metric-groups and --no-per-core; inspect subcommand help for the exact accepted values. These controls require a matching updated service; operator source and input formats are unchanged.

Native AscendC projects

Use --native-project path/to/native.json instead of --triton. This manifest contains an explicit list of UTF-8 files relative to its own directory:

{
  "kind": "native",
  "files": ["build.sh", "kernels.asc", "extension.cpp", "entry.py"],
  "build": ["bash", "build.sh"],
  "entrypoint": "entry:run",
  "build_timeout_seconds": 300,
  "kernel_names": ["my_kernel"],
  "event_timing": false
}

The client packages those file contents without compiling or importing them. Paths must be relative POSIX paths without traversal or symlinks; source limits are 128 files and 2 MiB. The manifest is limited to 1 MiB. The client requires POSIX no-follow file APIs for this packaging mode. The worker must provide a compatible native toolchain and dependencies, and the entrypoint returns tensors.

python3 "$ASCEND_OP_EVAL_CLI" benchmark \
  --definition definition.json --workloads workload.jsonl \
  --native-project project/native.json --warmup 3 --repeats 5

Native timing defaults to profiler (native_profiler_kernel_sum_v1), with manual warmup, one capture per measured call and the mean of per-call kernel sums. It does not claim CANNBench equivalence. Event timing additionally requires event_timing: true to declare correct stream coverage. Use --candidate-only for explicitly unverified single-source measurement. Baseline cache and --correctness-from reuse are unsupported for native projects in this release. Build diagnostics are returned in result.native_build.