Spectra

Product screenshots and demo clips go stale the moment the UI changes, and re-shooting them by hand costs an afternoon. Spectra drives your running app across web (Chrome DevTools Protocol), macOS (accessibility bridge), iOS and watchOS (simulators) to capture screenshots, videos, and narrated walkthroughs, then files every asset in a searchable tagged library. You get publish-ready visuals for docs, launch posts, and App Store previews without reshooting.

Start here

Type /spectra and describe what you want in plain language — "grab a screenshot of the settings screen", "record a 30-second demo of onboarding", "find the captures I tagged launch". That one command routes to every workflow below; you never have to pick a subcommand.

To report a bug or request a feature: /spectra:submit-feedback.

Install

Via Claude Code plugin marketplace (recommended)

/plugin marketplace add tyroneross/spectra
/plugin install spectra@spectra

Manual install (development / library use)

Requirements: Node.js 22+, macOS (for native features), Xcode CLI tools (for Swift compilation)

git clone https://github.com/tyroneross/spectra.git
cd spectra
npm install
npm run build

Native bridge (macOS/iOS/watchOS automation)

# Bare ~/.spectra/bin helpers are development-only and require an explicit mode.
export SPECTRA_HELPER_MODE=development

# Compile the Swift binary to ~/.spectra/bin/spectra-native
npm run build:native

# Optional: compile the SwiftUI test fixture
npm run build:test-app

macOS permissions required:

• Installed Spectra.app: grant Spectra access in System Settings → Privacy & Security → Accessibility and Screen & System Audio Recording (named Screen Recording on older macOS versions).
• Explicit development mode: macOS may attribute bare ~/.spectra/bin helpers to the terminal or launcher. Use this only for local development, not installed-app or release verification.

Dashboard (web UI)

cd web-ui
npm install
cd ..
npm run serve    # → http://localhost:4300

Claude Code Plugin

Spectra is a Claude Code plugin. Install via the marketplace (see above) or locally for development:

# From another project, point Claude Code at the Spectra directory
claude --plugin-dir /path/to/spectra

Or add to .claude/settings.json:

{
  "plugins": ["/path/to/spectra"]
}

Slash Commands

Marketing content loop

/spectra:marketing runs an evidence-backed creative loop above Spectra's capture and polish tools:

brief → diagnose → three concepts → weighted selection → claim evidence
→ storyboard → capture/render → 15-dimension audit → repair (maximum two)

Plan-only requests stop at PLAN_READY. Produced work reaches READY_TO_TEST only at 65/75 or higher with no unsupported claim, missing proof, broken CTA/destination continuity, unreadable media, or technical-output blocker. Campaign artifacts are stored under .spectra/campaigns/<slug>/ when the host can write files.

The agent is included in the npm package and uses current Spectra operations; it does not add a second capture runtime.

Routing: Host-Routed vs Standalone

The primary Spectra path is host-routed. A host agent such as Claude Code, Codex, or another LLM coding host reads spectra_snapshot, plans the next action, and calls Spectra MCP tools to execute and capture. Spectra does not need a model API key for this path.

The standalone Spectra.app fallback is for sessions launched outside a host agent. In that case the app uses WalkthroughPlanner.swift and AnthropicClient.swift to plan locally, with the user's key stored in Keychain.

Use host-routed walkthroughs when working inside a coding agent. Use the standalone path when the menu-bar app is the only planner available.

MCP Tools

spectra_connect

Start a new UI automation session.

Examples:

target: "https://myapp.vercel.app"       → web (CDP)
target: "Finder"                          → macOS native (AX)
target: "pid:4242"                        → one exact macOS process
target: "sim:iPhone 16 Pro"              → iOS simulator
target: "sim:Apple Watch Series 10"      → watchOS simulator

Targeting one instance when two are running. An app name is ambiguous while two instances of the same app are up — a live app and an isolated test instance, say — and the native helper resolves the name to whichever instance it finds first, usually the live one. Pass pid (or the pid:<n> target form, which sets the same field) to make the binding exact: every AX snapshot and action goes to that process only, and startRecording records a window owned by that pid or fails with recording_failed — it never falls back to the app name. pid:<n> still resolves a readable app name for the session label.

Pid-bound recording additionally requires a native helper that reports the windowId/pid recording selectors from its capabilities RPC. Before starting such a recording the daemon performs that handshake and refuses when it is absent, because a helper without the selectors would quietly record by app name. A helper embedded in an installed Spectra.app takes precedence over the locally built one (ensureBinary()), so a bundle predating this feature must be rebuilt and re-embedded — or point SPECTRA_APP_BUNDLE_HELPERS_DIR at a directory holding a rebuilt helper. AX snapshot/act need no handshake.

spectra_connect { "target": "Easy Terminal", "pid": 4242 }
spectra_connect { "target": "pid:4242" }

Returns: { sessionId, platform, target, name }

spectra_snapshot

Get the current accessibility tree snapshot.

Returns: Serialized AX tree — compact element list with roles, labels, bounds, actions.

spectra_act

Perform an action on a specific element.

Returns: { success, snapshot } — updated snapshot after the action.

spectra_step

Natural language navigation — describe what to do, Spectra finds the element and executes.

Returns: { resolved, elementId, action, confidence, snapshot } — plus optional screenshot.

Uses the Jaro-Winkler resolution engine to fuzzy-match intents to AX tree elements.

spectra_capture

Take a screenshot or manage video recording. Supports intelligent framing modes.

Capture modes:

• full — standard full-page screenshot
• element — crops to a single element by ID
• region — crops to a detected region by label (run spectra_analyze first to see available regions)
• auto — automatically frames the most important content on screen

Production presets:

• docs — clean 16:9 auto-framed screenshots and lossless 30 fps recordings for documentation.
• demo — clean 16:9 full-screen screenshots and smooth 60 fps H.264 recordings for product walkthroughs.
• social — clean 9:16 auto-framed screenshots and H.264 recordings for short social clips.
• app-store — stable clean full-screen captures for marketplace-style product shots.

Explicit capture options override preset defaults, so preset="demo" with fps=30 records at 30 fps while keeping the rest of the demo defaults.

Visual cleanup (clean: true):

• Hides scrollbars (web)
• Cleans simulator status bar — 9:41, full battery, full signal (iOS/watchOS)
• Removes cursor artifacts (web)

Returns: { path, format, preset?, crop?, label?, cleanApplied }

spectra_demo

Turn a raw screen recording into a polished, shareable demo clip. Applies a spotlight focus effect (sharp focal region, dimmed/blurred periphery), burns in lower-third captions — a full-width bar with a numbered step chip and a one-line description — and merges multiple captioned segments into one mp4. Audio handling is opt-in; screen recordings are stripped by default.

Actions:

• scan — scene-change analysis of a recording: per-minute activity + contiguous active ranges, so you can pick the segments worth showing.
• polish-clip — polish a single segment: spotlight focus, optional zoom, optional lower-third caption.
• polish-script — polish multiple segments and merge them into one mp4, each with its own numbered lower-third caption (the multi-scene captioned-walkthrough format).
• run-script — drive a live browser via CDP through scripted demo beats (search/click/navigate/scroll/hold) while capturing.
• auto-ramp — derive zoom ramps automatically from scene detection when explicit click telemetry is absent.

Caption banner is a fixed reference style: full-width #050709@92% bar, #27AFE8 rounded step chip, #F8FAFC text. Write one caption per beat — present-tense and outcome-focused ("Claude reads the inbox — green light to build").

Returns: { path, ... } — the rendered mp4 (or scan report for action=scan).

Privacy scan before publishing

Marketing recordings of terminal apps routinely show API keys, tokens, emails, and home-directory paths. Before sharing a clip, sample its frames, OCR them, and match against a library of secret/PII patterns:

spectra scan-frames <video-or-image> [--fps 1] [--range-ms <start:end>]... [--out <report.json>] [--allow <regex>]... [--json]

--allow <regex> suppresses known-safe matches (e.g. your own home path or an intentionally shown demo account email). Findings only ever carry a redacted preview (first 4 chars + … + last 2) — the full match is never written to the report or the terminal.

Use --fps 30 with repeated --range-ms values to inspect every frame in intentionally sharp windows without OCR-scanning fully blurred sections. Reports store only the input filename and temporary frame filename, never their absolute paths.

This is a pre-publish gate, not proof of absence. OCR misses low-contrast or small text, and the pattern library only covers known secret shapes. A clean verdict means "nothing obvious found," not "guaranteed safe" — still eyeball the frames before publishing.

spectra_analyze

Score the current screen — element importance, regions of interest, UI state.

Returns:

{
  "state": "populated",
  "stateConfidence": 0.833,
  "regions": [
    { "label": "Navigation", "score": 0.712, "bounds": [0, 0, 1280, 60], "elementCount": 8 },
    { "label": "Form", "score": 0.645, "bounds": [200, 300, 400, 250], "elementCount": 5 }
  ],
  "topElements": [
    { "id": "e3", "role": "heading", "label": "Welcome", "importance": 0.891 },
    { "id": "e7", "role": "button", "label": "Get Started", "importance": 0.856 }
  ],
  "totalElements": 42
}

UI states detected: loading, error, empty, populated, focused, unknown

Importance scoring uses 6 weighted heuristics from UEyes CHI 2023 eye-tracking research:

• Semantic role (30%) — headings, buttons, links score highest
• Position (20%) — top-left bias, above-fold bonus
• Interactivity (15%) — elements with actions
• Label quality (15%) — meaningful text labels
• Content density (10%) — elements surrounded by related content
• Visual prominence (10%) — larger elements score higher

spectra_discover

Auto-navigate and capture an entire app. BFS crawls through links, buttons, and tabs — captures and frames each screen.

How it works:

1. Takes initial screenshot + AX snapshot
2. Scores elements by importance
3. Detects UI state (populated, loading, error, etc.)
4. Auto-frames and saves the best region
5. Finds navigable elements (links, buttons, tabs, menu items)
6. BFS crawls to each, repeating steps 2-4
7. Deduplicates screens by role+label fingerprint (stable across React/Angular)
8. Detects and flags sensitive screens (password fields, API keys)
9. Writes manifest with all captures

Security:

• Same-origin only by default
• Never interacts with password/credential fields
• Flags sensitive screens in manifest (skips capture)

Returns: { screens, captures, sensitive, manifestPath, outputDir }

Output: {outputDir}/screen-*.png, {outputDir}/framed-*.png, {outputDir}/manifest.json

spectra_session

Manage active sessions.

Library API

Spectra exports a full programmatic API for use by other tools:

Drivers

import { CdpDriver, NativeDriver, SimDriver } from 'spectra'

// Web — Chrome DevTools Protocol
const web = new CdpDriver({ browser: { headless: true } })
await web.connect({ url: 'https://myapp.vercel.app' })
const snap = await web.snapshot()    // → Snapshot { elements, platform, timestamp, metadata }
const buf = await web.screenshot()   // → Buffer (PNG)
await web.navigate('https://myapp.vercel.app/about')
await web.act('e4', 'click')         // → ActResult { success, snapshot }
await web.disconnect()

// macOS — Accessibility bridge
const mac = new NativeDriver()
await mac.connect({ appName: 'Finder' })

// iOS/watchOS — Simulator
const sim = new SimDriver()
await sim.connect({ simulator: 'iPhone 16 Pro' })

Intelligence — Importance Scoring

import { scoreElements, findRegions } from 'spectra'
import type { Viewport, ImportanceScore, RegionOfInterest } from 'spectra'

const viewport: Viewport = { width: 1280, height: 800, devicePixelRatio: 2 }
const scores: ImportanceScore[] = scoreElements(snapshot.elements, viewport)
// → [{ elementId: 'e3', score: 0.891, factors: [...] }, ...]

const regions: RegionOfInterest[] = findRegions(scores, snapshot.elements)
// → [{ label: 'Navigation', score: 0.712, bounds: [0,0,1280,60], elements: ['e1','e2',...] }]

Intelligence — Change Detection

import { perceptualHash, hashDistance, diffSnapshots, detectChange } from 'spectra'

// Fast visual comparison (dHash — 64-bit gradient hash)
const hash1 = perceptualHash(screenshotBuffer1)  // → bigint
const hash2 = perceptualHash(screenshotBuffer2)
const distance = hashDistance(hash1, hash2)        // → number (0 = identical, >10 = different)

// Semantic comparison (AX tree structural diff)
const diff = diffSnapshots(snapshot1, snapshot2)
// → { changed: true, score: 0.35, type: 'significant', details: [{kind:'added',...}] }

// Combined pipeline: dHash pre-filter → structural diff
const change = detectChange(buf1, buf2, snap1, snap2, 0.15)
// → ChangeResult { changed, score, type, details }

Intelligence — State Detection

import { detectState, createStateTriggers } from 'spectra'
import type { UIState, StateDetection, StateTriggerOptions } from 'spectra'

const state: StateDetection = detectState(snapshot)
// → { state: 'populated', confidence: 0.83, indicators: ['e5', 'e8', ...] }

// CDP state triggers — inject loading/error/empty UI for capture
const triggers = createStateTriggers({
  conn: cdpConnection,       // from driver.getConnection()
  sessionId: 'target-id',
  platform: 'web',
})
// → StateTrigger[] — each has trigger() and restore()

for (const t of triggers) {
  await t.trigger()          // inject simulated UI state
  // ... capture screenshot ...
  await t.restore()          // restore original page
}

Intelligence — Smart Framing

import { frame, autoFrame } from 'spectra'
import type { FrameOptions, FrameResult } from 'spectra'

// Auto-frame to best content region
const result: FrameResult = frame(screenshotBuffer, scores, elements)
// → { crop: [x, y, w, h], buffer: Buffer, label: 'Navigation' }

// Frame a specific element
const elementFrame = frame(buf, scores, elements, {
  target: 'element',
  elementId: 'e7',
  padding: 16,
  aspectRatio: 16 / 9,
})

// Get multiple crop suggestions
const frames: FrameResult[] = autoFrame(buf, scores, elements)
// → top 5 regions sorted by importance score

Intelligence — Navigation

import { crawl, discoverByScroll } from 'spectra'
import type { NavigationGraph, CrawlOptions } from 'spectra'

const graph: NavigationGraph = await crawl(driver, {
  maxDepth: 3,
  maxScreens: 50,
  scrollDiscover: true,
  changeThreshold: 0.15,
  allowExternal: false,
  allowFormSubmit: false,
})
// → { nodes: Map<id, ScreenNode>, edges: NavigationEdge[], root: string }

// Scroll to discover lazy-loaded content
const newScreens = await discoverByScroll(driver, 20)

Media — PNG Codec

Zero-dependency PNG decode/encode built on node:zlib:

import { decodePng, encodePng, cropImage, resizeNearest, toGrayscale } from 'spectra'
import type { RawImage } from 'spectra'

const raw: RawImage = decodePng(pngBuffer)
// → { width, height, data: Uint8Array (RGBA, 4 bytes/pixel) }

const cropped = cropImage(raw, 100, 50, 400, 300)   // x, y, w, h
const small = resizeNearest(raw, 9, 8)                // nearest-neighbor
const gray: Uint8Array = toGrayscale(raw)             // 1 byte/pixel
const encoded: Buffer = encodePng(cropped)             // → valid PNG buffer

Media — Capture Cleanup

import { prepareForCapture, restoreAfterCapture } from 'spectra'
import type { CleanOptions, CleanState } from 'spectra'

// Prepare environment for clean screenshots
const state: CleanState = await prepareForCapture(cdpConnection, sessionId, 'web', {
  hideScrollbars: true,   // Emulation.setScrollbarsHidden
  hideCursor: true,       // CSS injection
  cleanStatusBar: true,   // xcrun simctl status_bar (iOS/watchOS)
  viewport: { width: 1280, height: 800 },
})
// state.applied → ['scrollbars', 'cursor', 'viewport']

// ... take screenshots ...

await restoreAfterCapture(state)  // reverses all cleanup (reverse order, fault-tolerant)

Media — Video Probe + Poster Frames

import { buildProbeArgs, buildPosterFrameArgs, probeVideo } from 'spectra'

// Build FFprobe arguments for metadata extraction.
const probeArgs = buildProbeArgs('/tmp/output.mp4')
// -> ['-v', 'error', '-select_streams', 'v:0', ...]

const metadata = await probeVideo('/tmp/output.mp4')
// -> { durationMs, width, height, fps, codec }

// Build FFmpeg arguments for a poster frame.
const posterArgs = buildPosterFrameArgs('/tmp/output.mp4', '/tmp/poster.jpg', {
  atSeconds: 1,
  maxWidth: 1280,
})

Media — Production Bundles

import { createProductionBundle } from 'spectra'

const bundle = await createProductionBundle([
  {
    id: 'capture-1',
    path: '/absolute/path/to/capture.png',
    type: 'screenshot',
    preset: 'demo',
    caption: 'Dashboard overview',
  },
], {
  outDir: '/absolute/path/to/.spectra/productions/demo-bundle',
  title: 'Demo launch assets',
})

// Writes masters/, derivatives/, manifest.json, quality-report.json, and README.md.
// PNG screenshots get thumbnail derivatives. Videos are packaged as masters
// with quality checks until probe/transcode derivatives are explicitly added.
console.log(bundle.manifest.quality.status)

Resolution Engine

import { resolve } from 'spectra'

const result = resolve(snapshot.elements, 'click the submit button', {
  bias: 'spatial',        // 'semantic' | 'spatial'
  threshold: 0.4,
})
// → { elementId: 'e12', confidence: 0.87, method: 'jaro-winkler' }

Session Management

import { SessionManager } from 'spectra'

const manager = new SessionManager()
const session = await manager.create({ name: 'my-capture', platform: 'web' })
await manager.addStep(session.id, { intent: 'click login', elementId: 'e3', action: 'click' })
const sessions = manager.list()

Serialization

import { serializeSnapshot, serializeElement, normalizeRole } from 'spectra'

const compact = serializeSnapshot(snapshot)   // → compact string representation
const role = normalizeRole('AXButton', 'macos')  // → 'button' (cross-platform)

Core Types

interface Element {
  id: string
  role: string                              // normalized: 'button', 'link', 'heading', etc.
  label: string
  value: string | null
  enabled: boolean
  focused: boolean
  actions: ActionType[]                     // 'click' | 'type' | 'clear' | 'select' | 'scroll' | 'hover' | 'focus'
  bounds: [number, number, number, number]  // [x, y, width, height]
  parent: string | null
}

interface Snapshot {
  platform: Platform        // 'web' | 'macos' | 'ios' | 'watchos'
  elements: Element[]
  timestamp: number
  metadata: SnapshotMetadata
}

interface Driver {
  connect(target: DriverTarget): Promise<void>
  snapshot(): Promise<Snapshot>
  act(elementId: string, action: ActionType, value?: string): Promise<ActResult>
  screenshot(): Promise<Buffer>
  navigate?(url: string): Promise<void>
  getConnection?(): { conn: unknown; sessionId: string | null }
  close(): Promise<void>
  disconnect(): Promise<void>
}

Project Structure

spectra/
├── src/
│   ├── core/           # Types, session manager, resolve engine, serialize, normalize
│   ├── cdp/            # Chrome DevTools Protocol (connection, browser, 7 domain wrappers)
│   ├── intelligence/   # Capture optimization
│   │   ├── spatial.ts      # Shared: edgeDistance, regionLabel, boundingBox, clusterElements
│   │   ├── types.ts        # 18 shared interfaces
│   │   ├── importance.ts   # 6-factor element scoring (UEyes CHI 2023)
│   │   ├── change.ts       # dHash perceptual hash + structural diff
│   │   ├── states.ts       # UI state detection + CDP triggers
│   │   ├── framing.ts      # Smart crop with aspect ratio enforcement
│   │   └── navigation.ts   # BFS crawl + scroll discovery
│   ├── mcp/            # MCP server + 8 tool handlers
│   ├── media/          # PNG codec, capture, cleanup, video pipeline
│   └── native/         # Swift bridge, native driver, simulator driver
├── native/swift/       # Swift source for native binary (AXBridge, SimBridge)
├── web-ui/             # Next.js dashboard (browse, manage, export captures)
├── commands/           # Claude Code slash commands
├── skills/             # Claude Code skills
├── tests/              # Vitest test suite
├── artifacts/          # Capture output (gitignored)
└── .spectra/           # Session data, playbooks, archive (gitignored)

Scripts

Environment Variables

Platforms

Codex

This package now ships an additive Codex plugin surface alongside the existing Claude Code package. The Claude package remains authoritative for Claude behavior; the Codex package adds a parallel .codex-plugin/plugin.json install surface without changing the Claude runtime.

Package root for Codex installs:

• the repository root (.)

Primary Codex surface:

• skills from ./skills when present
• MCP config from inline mcpServers metadata when present

Install the package from this package root using your current Codex plugin install flow. The Codex package is additive only: Claude-specific hooks, slash commands, and agent wiring remain unchanged for Claude Code.

Hybrid story studio

Combine real app captures with generated HTML motion scenes or imported renders, using one frame clock for narration, captions and sound. Claude, Codex and other agents use the same async spectra_demo compose/status actions or spectra demo CLI. Existing app capture, polish styles, spotlight, zoom and voice/audio workflows remain available. See story studio, story-builder skill and runnable example.