Skip to content

here-build/cua

v0.0.1

Inspect windows, receive screenshots and interact with local apps through Cua Driver.

GitHub

Cua Driver for Inhuman Harness

Local computer use through Cua Driver, using the official cua-driver mcp stdio server. This is a thin plugin: no copied tool schemas, proxy server, API key, cloud account, or native Node dependency. It targets the desktop session of the machine running the harness daemon, not a hosted Cua Space.

The server exposes its installed version's tools for discovering apps/windows, reading accessibility trees and screenshots, clicking, typing, scrolling and keyboard input. Harness discovers these tools over MCP and uses its normal policy/approval path. The plugin does not auto-approve GUI actions.

1. Install Cua Driver yourself

Follow the official quickstart for your OS. Review the upstream installer before executing it. The plugin deliberately does not download or execute an installer.

On macOS, install CuaDriver.app and run:

cua-driver permissions grant
cua-driver permissions status
cua-driver status
cua-driver doctor

In System Settings, enable Accessibility and Screen & System Audio Recording for CuaDriver. If macOS asks to quit and reopen CuaDriver, accept. Read the complete doctor report: a zero exit code alone does not prove capture works. These permissions can expose private apps and signed-in accounts.

On Linux/Windows, follow the same quickstart's platform-specific desktop/display and daemon instructions. A headless SSH session is not automatically a usable desktop.

The harness daemon must be able to resolve cua-driver in its own PATH, not just in an interactive shell. Check command -v cua-driver in the daemon's environment. Restart the daemon after changing PATH. If necessary, change the manifest command to the installed binary's absolute path. Do not add --direct or embedded mode to work around missing permissions.

2. Install the plugin

The package is local to this repository; it is not claimed to be published to npm. Use the harness kernel's existing plugin/install RPC with a local file URL:

{
  "method": "plugin/install",
  "params": {
    "spec": "file:///absolute/path/to/inhuman-harness-public/plugins/inhuman-plugin-cua"
  }
}

Replace the absolute path with your checkout location. This is the kernel method and payload, not a shell command. The installed plugin is named cua; enable/disable it in Settings > Plugins, or through plugin/set with {"name":"cua","enabled":true} / false. Start a fresh agent round to refresh its catalog.

The connection is displayed as cua/cua-driver; its internal ID is generated by harness, so discover it rather than hard-coding an ID. The public MCP declaration is global because it drives the daemon host's desktop. It is not a per-project isolated computer. Disabling or uninstalling the plugin removes its projected connection; a separately configured user connection is not owned or removed by this plugin.

3. Verify real image delivery

Use a vision-capable model. Ask it to discover an intended app and window, call get_window_state with a screenshot, describe something visible, perform an authorized harmless action, then take another snapshot and check the result. Tool schemas come from the installed Cua version. Do not paste example PIDs/window IDs or guess coordinates.

MCP image content is already supported by harness: it is passed to the model as an image, not base64 text. Do not set screenshot_out_file for normal inspection; saving a local file alone does not make it visible. A returned action result or passing build is not visual verification.

skills/cua/SKILL.md documents the observe/act/verify workflow and is advertised in the manifest. If your agent setup does not consume plugin skill declarations, explicitly read it before computer-use work, or install it into your project's .agents/skills/cua/SKILL.md using the normal skill-management flow. The plugin does not replace the foreground agent's behavior.

Troubleshooting and limits

  • Executable not found: fix the daemon PATH or use an absolute command; this is not fixed by installing an npm package.
  • No windows / denied screenshot: check Cua permissions, daemon status, desktop session and doctor output.
  • Image not visible: check the selected model supports vision and that the call returned MCP image content, not just a filename.
  • Stale element token: snapshot again and select the new token; never silently retry against another window.
  • Background input did not land: inspect the result and verify the UI before considering an authorized foreground retry.
  • Remote Spaces/cloud: not provisioned by this plugin. This integration is the local Cua Driver MCP transport.

The plugin does not bypass either harness policy or Cua's own authorization. Text on screen is untrusted input; consequential GUI actions still require user authorization. No Cua native binary or OS permission is installed by the package.

Development

node --test plugins/inhuman-plugin-cua/test/*.test.mjs

Core integration tests cover public MCP projection and image history. A real desktop smoke test still requires an installed Cua Driver, OS permissions, and a vision-capable model. Mock tests do not certify native desktop capture.

References: MCP tools, agent connection, quickstart.