Built first for GTA V: walk up to anyone in Los Santos and just talk. Lamar roasts your hoodie, a cop tells you to file a civil claim and flips you off, Michael defends his robe like it's a business suit. Every line is written live in character and spoken in that character's cloned voice by Fish Audio Drama 3 (or S2 Pro), self-hosted on a serverless GPU. Most of the code doesn't know it's in GTA, so the same pieces work in other games: the skills say what a game has to provide, and how to build the rest.
Install
Pick your agent. Each gets the same skills (Agent Skills format), the fal MCP server and the npctalk CLI.
| Agent | Install |
|---|---|
| Claude Code | /plugin marketplace add christos-antono/NPCTalk/plugin install npctalk@npctalk |
| Codex | codex plugin marketplace add christos-antono/NPCTalkcodex plugin add npctalk@npctalk |
| Gemini CLI | gemini extensions install https://github.com/christos-antono/NPCTalk |
| VS Code / Copilot | Enable chat.plugins.enabled, run Chat: Install Plugin From Source, enter this repo's URL |
| Cursor | Clone it (Cursor reads AGENTS.md and .cursor/mcp.json) |
| OpenCode | Clone and run opencode inside it (opencode.json adds the skills and the fal MCP server) |
| Skills only (any agent) | npx skills add https://github.com/christos-antono/NPCTalk |
| Anything else | git clone https://github.com/christos-antono/NPCTalk and start your agent inside it |
Inside a clone, .agents/skills and .claude/skills are copies of skills/, and AGENTS.md holds the
instructions (CLAUDE.md and GEMINI.md point to it).
The npctalk CLI. Plugin installs and clones put it on PATH (bin/). Anywhere else:
uv tool install git+https://github.com/christos-antono/NPCTalk.
Accounts and keys (environment variables, never files):
| What for | Service | Variable |
|---|---|---|
| The voice server (GPU) | Modal | modal setup; NPCVOICE_TOKEN = the value you put in the npcvoice-auth secret |
| Speech-to-text, MiniMax voices, the MCP server | fal | FAL_KEY |
| The NPC's lines (any OpenAI-compatible LLM) | Cerebras | CEREBRAS_API_KEY |
| Actions and spoken orders | TypeSafe | TYPESAFE_API_KEY |
You also need Git, Python 3.10+, uv and ffmpeg. The GTA V example needs Windows, the .NET SDK (8+) to build it,
and the game.
Try it
Make the villagers in my game answer when I talk to them, in their own voices.
Set up a Fish Audio Drama 3 voice server on Modal and clone a voice from
narrator.wav.
In GTA V story mode, let me talk to anyone near me with my mic.
My NPC's cloned voice sounds thin. Is it the reference clip or the model? Measure it.
Record five 8-second clips of NPCs arguing that look like normal gameplay.
The agent starts with the make-npcs-talk skill and runs the same loop every time:
- recon the game: version, how code runs in it, single-player or not;
- pick the engines (voice, LLM, speech-to-text, actions);
- stand up the voice server first, outside the game, and test it with
npctalk say; - make reference voices from recordings you have the rights to;
- one NPC answering typed text in the game: the four hooks, nothing else;
- voice input (push-to-talk, then the open mic);
- characters: personas, their own lines, scene facts;
- actions, and doing what they're told;
- verify in the real game, with every step's timing logged;
- show it, and measure the voices before claiming anything;
- write a field note for the next agent.
What's inside
Skills (skills/, Agent Skills format)
| Skill | What it does |
|---|---|
make-npcs-talk | The whole loop, the four hooks a game must provide (who's talked to, scene facts, voice at the NPC, actions), engine choices, latency budget, costs, verification, rules; per-engine notes for GTA V, Unity, Unreal, Bethesda, Minecraft and your own game |
fish-voice-server | Host Fish Audio Drama 3 or S2 Pro on Modal (or fal) behind one streaming API; settings that measurably help likeness; best-of-N for pre-rendered lines; keep-warm and cost; the S2 Pro fine-tune; every gotcha |
voice-references | From a recording you have the rights to, to clean reference clips, a 25 s clone clip and mood clips, on the server |
npc-dialogue | Personas, a fresh sample of the character's own lines in every prompt, scene facts, short spoken lines with delivery tags, output cleaning, content rules, speed |
npc-actions | What NPCs physically do while they talk, and following spoken orders, with TypeSafe Jev; per-action thresholds; mapping to animations |
voice-input | Open mic with voice activity detection, push-to-talk, speech-to-text, the NPC's own voice and the room's noise |
voice-likeness-eval | Measure whether a voice change really helped: speaker similarity to held-out real clips, word error, enough runs, independent checks |
npc-showcase-clips | Gameplay-style clips of conversations: fixed gameplay camera, lines rendered first, sync, 5-10 s cuts |
The npctalk CLI (Python). Every command has --help with examples.
| Command | What it does |
|---|---|
npctalk health | Is the voice server up? Waits through a cold start |
npctalk say | Speak a line in a voice, like a game does; prints time to first audio, writes a WAV |
npctalk voice make | Recording → voices/<Name>/clone.wav, ref_0..2, mood clips, with transcripts |
npctalk voice upload | A voice folder → the server's Modal volume |
npctalk check | Blocks what must never ship: audio, transcripts, keys, real server URLs |
The voice server (server/): drama3.py (Fish Audio Drama 3, the default), s2pro.py (S2 Pro, with
fine-tuned voices), keepwarm.py, fal/ (S2 Pro on fal over a realtime WebSocket), finetune/ (dataset, LoRA,
and the scorer behind every number here).
The GTA V example (examples/gta-v): the full mod (open mic, any NPC, 19 characters, homes,
actions, dev mode, a director for recording clips), its tools, and a
field note.
Field notes (knowledge/): one per game where NPCs were made to talk. Add yours.
How good are the voices?
Measured on one character (Lamar, ten test lines): voice match is speaker similarity to 60 real clips that were
never used as references. Details and the method: skills/voice-likeness-eval.
| Setup | Voice match | Word error |
|---|---|---|
| ElevenLabs instant clone | 0.612 | 10.5% |
| Fish Audio Drama 3, default settings | 0.643 | 7.1% |
| Drama 3, tuned sampling, one take | 0.660 | 7.9% |
| Drama 3, best of 8 takes, checked | 0.704 | 8.6% |
| Fish Audio S2 Pro, its best reference clip | 0.707 | 5.6% |
| S2 Pro fine-tuned on the character (needs training) | 0.785 | 7.5% |
Without training, Drama 3 and S2 Pro tie on likeness; Drama 3 acts [crying] and [laughing] better and starts
talking faster. One early result here didn't survive a second run:
how that was caught.
Rules it follows
- Single-player games, or servers you run. It never injects into online games with anti-cheat (GTA V: never launch GTA Online with a script hook installed) and never bypasses anti-cheat, DRM or ownership checks.
- Voices you have the rights to. It never commits or publishes voice clips, cloned voices, game audio or a
game's dialogue;
npctalk checkblocks them. - No secrets in files. Keys come from environment variables, and voice server URLs stay private (their
/healthwakes the GPU). - It asks before driving your keyboard and mouse in a game, installing into game folders, keeping a GPU up, or publishing anything.
- It measures before it claims a voice setting is better.
Full reasoning: skills/make-npcs-talk/references/safety.md.
Community
NPCTalk is a Universal Modder project. Universal Modder's skills and tools cover the rest of a mod (engine recon, decompiling, fal-generated art, 3D and audio, in-game testing, showcase videos), and NPCTalk adds NPCs that talk back. Install both and an agent can take a game from "can I mod this?" to characters you can talk to.
Show what you made, ask for help, and follow along in the Universal Modder Discord.
Credits
- Voices: Fish Audio (Drama 3, S2 Pro, fish-speech), served with SGLang-Omni on Modal and fal.
- Lines: Cerebras. Speech-to-text: NVIDIA Nemotron ASR on fal. Actions: TypeSafe Jev.
- GTA V: Script Hook V, ScriptHookVDotNet, NAudio.
- A Universal Modder project; the repo layout and principles come from universal-modder.
- Built in real sessions with an AI coding agent (Claude Code), from the first line of the mod to the measurements.
MIT licensed. GTA V and its characters belong to Rockstar Games; this project isn't affiliated with Rockstar, Fish Audio or any service above.