christos-antono/npctalk

Host Fish Audio's open-weights voice models (Drama 3 and S2 Pro) on a serverless GPU (Modal, or fal) behind one small HTTP API a game can call. POST /speak takes text and a voice and streams 16-bit PCM back with first audio in ~0.4 s, cloned zero-shot from a reference clip, with [tags] like [angry] or [laughing] for delivery. Covers deploying with SGLang-Omni, the API, preparing voices at startup, keep-warm and cold starts, scaling to zero and cost, sampling settings that measurably help likeness, picking the best of several takes for pre-rendered lines, an optional S2 Pro fine-tune, and the setup gotchas. Use when a game, mod or app needs realtime cloned character voices, the user mentions Fish Audio, Fish Speech, S2 Pro, Drama 3 or a self-hosted TTS server, or a voice server is slow, cold or failing.

Make NPCs in a game talk back. The player speaks or types, the NPC answers in character, in a cloned voice, about 1-2 seconds later, and can act on what was said. Covers the whole loop and the order to build it in (who is being talked to, what the scene is, speech-to-text, an LLM in character, Fish Audio voices on a GPU, streamed playback from the NPC, subtitles, mouth movement, actions), the four hooks a game has to provide, latency budgets, costs, testing in the real game, and the rules (single-player, voices you have the rights to, nothing of the game shipped). Use when the user wants NPCs, characters, villagers, companions or enemies in any game to hold conversations, react to the player's voice, or be voiced by Fish Audio (S2 Pro or Drama 3) and an LLM.

Make NPCs physically react to a conversation and follow spoken orders. Pick an action (punch, slap, hug, kneel and beg, flee, flip off, square up, nothing...) from what the player said and what the NPC replied, with TypeSafe's Jev returning a typed choice and a probability in ~0.1 s, per-action thresholds, a repeat cooldown, orders ("get on your knees") that override mood and make the LLM rewrite the line, and mapping action names onto the game's animations with checks at load. Use when NPCs should act out what they say, obey commands, or when actions picked by an LLM are too random, too timid or too frequent.

Write what NPCs say with an LLM so they sound like themselves and like the game. Covers persona files, a fresh random sample of the character's own lines in every prompt, scene facts (place, time, weather, what the player holds or drives, danger), per-NPC conversation history, short spoken output with one delivery tag for the voice model, doing what the player tells them, fast models with reasoning off, and cleaning the output (refusals, name prefixes, length, slur rules). Use when NPC replies sound generic, robotic, too long, preachy or out of character, refuse, break the game's tone, or come back too slowly, or when adding a new character's persona.

Record short clips of NPC conversations that look like real gameplay, ready to post. Covers scripted exchanges at chosen places and times, the gameplay camera held still behind the player, both lines rendered before the clip rolls so it plays back to back, the game's own HUD and subtitles with nothing of the mod on screen, timeline markers for cutting, screen and audio sync, 5-10 s cuts in 16:9 (and 9:16), pinning lines to re-voice an approved take, and re-taking clips that run long. Use when the user wants demo videos, clips or a montage of talking NPCs.

Let the player talk to NPCs out loud. Covers an always-on open mic with voice activity detection against the room's noise floor, push-to-talk and typing as fallbacks, ignoring the NPC's own voice coming out of the speakers, dropping coughs, clicks and filler words, fast speech-to-text (fal's Nemotron ASR, ~0.4 s, audio sent inline), and routing what was heard to the right NPC. Use when building or debugging mic input for a game, when the open mic triggers on noise or misses speech, when NPCs answer things the player didn't say, or when "nobody answers".

Measure how much a cloned voice sounds like the real character, and whether a change really helped. Covers speaker-embedding similarity (ECAPA) against the centroid of held-out real clips, word error rate with Whisper, comparing per line with paired t-tests, how many runs it takes to beat sampling noise, checking a take-picker with an independent speaker model, and simulating picking strategies on many scored takes. Use when comparing voice engines or settings (Fish Audio Drama 3 vs S2 Pro vs ElevenLabs, sampling, reference clips, best-of-N, fine-tuning), or before telling anyone that one voice setup is better than another.

Make the reference clips a zero-shot voice cloner needs for each character (Fish Audio Drama 3 or S2 Pro, MiniMax, ElevenLabs-style cloning) from recordings the user has the rights to. Split into lines, pick clean 6-11 s runs, transcribe them, drop screams and noise, build a ~25 s clone clip plus single reference clips and mood clips (angry, drunk), check them, and upload them to the voice server. Use when adding a character's voice, when a cloned voice sounds off, thin or noisy, or when the user asks how to clone a voice for an NPC.