social-video-agent

An agent-native, local-first editor for turning existing recordings into Reels and Shorts. Codex makes editorial decisions from an inspectable transcript; FFmpeg performs frame-accurate cuts and audio, and Remotion adds a planned motion-design layer. Sources are never modified, and no paid AI API key is required.

source → local transcription → edit plan → EDL + motion plan → FFmpeg base → Remotion → QA

The project is an alpha. The FFmpeg pipeline, default Remotion compositor, captions, local faster-whisper backend, profiles, and QA are implemented. WhisperX alignment and diarization remain optional future integrations.

Windows 11 + Codex + WSL2

This is the Tier 1 setup. ChatGPT Desktop runs on Windows, while Codex and every media tool run inside WSL2. Do not install a second copy of Python, FFmpeg, or Node natively on Windows for this workflow.

1. Install WSL2 from an Administrator PowerShell terminal, then restart if Windows requests it:

   wsl --install
   

2. In the ChatGPT desktop app open Settings → Agent environment, choose Windows Subsystem for Linux, select the desired distribution if offered, and restart the app.

3. Open WSL and clone the repository into the Linux filesystem:

   mkdir -p ~/projects
   cd ~/projects
   git clone https://github.com/bartlomiejborzucki/social-video-agent.git
   cd social-video-agent
   ./scripts/wsl/bootstrap.sh
   social-video-agent doctor
   

4. Add or open ~/projects/social-video-agent in Codex and ask:

   Edit this video into a vertical Reel.

Microsoft recommends keeping Linux-tool projects in the WSL filesystem. Do not use /mnt/c/Users/.../social-video-agent as the normal repository location: Git, Python environments, node_modules, frame extraction, and render caches are substantially slower across the mounted Windows filesystem boundary.

Windows media paths

Source and final media may still live on Windows drives:

C:\Users\User\Videos\Mój film.mp4
↕
/mnt/c/Users/User/Videos/Mój film.mp4

Both forms are accepted inside WSL. Pasted drive-letter paths are converted with wslpath; spaces and Polish characters are passed as ordinary subprocess arguments, never through a shell.

social-video-agent inspect 'C:\Users\User\Videos\Mój film.mp4'
social-video-agent edit '/mnt/c/Users/User/Videos/Mój film.mp4' \
  --output '/mnt/c/Users/User/Videos/output/'

For a source under /mnt/c or /mnt/d, high-I/O intermediate data is stored under ~/.cache/social-video-agent/workspaces/ in Linux. Only the requested final file is copied to the Windows destination. Override the cache root with SOCIAL_VIDEO_HOME.

Two supported runtime modes

The editing engine is always the Linux one. What changes is where the agent runs:

runtime_mode: auto (the default) decides from the real platform of the agent process, whether wsl.exe runs, whether the chosen distribution reports WSL 2, and whether the engine is installed inside it. It never reads a terminal preference: which shell an editor opens says nothing about where the agent runs. Pin it with runtime_mode: in the project config or SOCIAL_VIDEO_RUNTIME_MODE, and name a distribution with wsl_distribution: or SOCIAL_VIDEO_WSL_DISTRIBUTION when more than one is installed.

In the hybrid mode the only supported bridge is scripts/windows/social-video-agent.ps1. It passes arguments as an array (never a command string), uses no Invoke-Expression and no sh -lc, propagates stdout, stderr and the exit code, and refuses to fall back to ffmpeg.exe, Windows Python or Windows Node — a run that mixed Windows and Linux binaries would not be the run that was reviewed. If WSL or the engine is missing it names which one and stops; it never installs anything. There is still no native Windows execution engine, Docker service, MCP server, or cloud backend.

social-video-agent doctor reports the agent side and the engine side separately, and distinguishes a missing WSL install, a broken one, a WSL 1 distribution, an ambiguous choice of distribution and a missing engine inside a working distribution. Capabilities that only the agent can see — its native image tool, a Canva connection — are reported as exactly that.

Remotion is enabled by default

Bootstrap installs the exact Node packages from package-lock.json, downloads Chrome Headless Shell through Remotion's official command, type-checks the composition, and runs a real offline MP4 smoke render. doctor treats Node, Remotion, and its browser as required. Keep both the repository and node_modules in the WSL filesystem.

Remotion uses a source-available license rather than Apache-2.0. Stage 0 stops before project discovery or media processing until the user records one of:

• free_license_eligible — after confirming the current free-license terms;
• company_license_confirmed — after obtaining the applicable Company License.

See the current Remotion license. The CLI does not decide legal eligibility. Users who explicitly do not want Remotion may select --renderer ffmpeg, but that is an opt-out from the default visual pipeline.

Declare once per project rather than once per edit:

social-video-agent remotion-license attest free_license_eligible \
  --project-root . --accept-terms
social-video-agent remotion-license status --project-root . --json
social-video-agent remotion-license refresh --project-root . --accept-terms
social-video-agent remotion-license revoke --project-root . --reason "yearly review"

The statement is stored in .social-video/remotion-license.json — beside the project config, never inside it, because config.yaml is branding and this is a licensing statement. It records the declaration, the acknowledgement, the date, the component versions and the terms URL, and nothing else: eligibility is never derived from company size, revenue or any other data, and there is no field in which such data could be supplied. --accept-terms is the user's own act; an assistant must ask and use the answer given.

workflow init and render take an explicit --remotion-license flag first (a one-edit declaration), then the edit's own workflow-state.json when resuming, then the project's stored declaration; with none of those they stop and print how to record one. The CLI asks again after a revocation, a change of terms URL, a new release line, or a year. See the declaration migration.

Project-aware editing

social-video-agent starts from the project, not generic editing defaults. When invoked from a repository it performs a bounded local discovery pass before editorial planning. It looks for explicit social-video.yaml configuration, applicable AGENTS.md, brandbooks, video and tone-of-voice guidelines, fonts, logos, templates, and relevant design assets. It skips .git, dependencies, virtual environments, builds, caches, models, renders, and temporary frames; it does not recursively ingest the whole repository.

The target project root is distinct from the installed plugin/skill root. All findings are saved per edit under:

edit/context/project-context.json
edit/context/project-context.md
edit/context/context-sources.json

Explicit user instructions have highest priority, followed by project-specific social-video config, project instructions, video guidance, brand guidance, templates, the general design system, previous outputs, and finally skill defaults. Claims carry a source and confidence; heuristic colors or web fonts are not silently promoted to video rules.

Inspect or refresh discovery deterministically:

social-video-agent context inspect . --workspace edit
social-video-agent context refresh . --workspace edit

Initialize and validate a versioned, executable project contract:

social-video-agent config init .
social-video-agent config validate . --workspace edit

A config written before 0.4 stays valid discovery context. To compile it into a contract, config migrate lists every value that needs a human decision -- punch_in_intensity: restrained has no one correct number, and a margin of 0.15 meant 15% when margins were fractions but means a hairline now that they are percentages. Renames are applied for you; editorial values are never guessed:

social-video-agent config migrate .

.social-video/config.yaml (or social-video.yaml) may specify only the fields the project needs, for example:

schema_version: 1
brand_name: Example
font: Lato
font_file: assets/fonts/Lato-Bold.ttf
caption_style:
  font_weight: bold
  text_color: "#FFFFFF"
  background_color: "#28BCA5"
  background_style: rounded_box
  corner_radius: 24
  outline_color: "#394463"
  outline_or_shadow: none
  font_size_pct: 3.6
  max_lines: 2
  max_words_per_cue: 4
  max_chars_per_cue: 24
  position: lower_safe_zone
  bottom_margin_pct: 22
editing_profile: calm-expert
music_policy: none
sfx_policy: none
audio_cleanup_policy: measured
default_aspect_ratio: "9:16"
default_resolution: "1080x1920"
default_fps_policy: "30"
delivery_output: exports/social

Validation resolves local assets and writes edit/brand-contract.json. Stage 2 renders from this contract; it does not merely copy branding into context. Rounded backgrounds are drawn deterministically by Remotion. Stage 4 requires both technical QA and edit/qa/qa-brand.json.

Voice cleanup is measured, not applied

audio_cleanup_policy: measured is the default, and it is not a one-button "enhance voice". Before mixing anything under the speech the render measures this recording: the per-window RMS distribution (the noise floor is the 10th percentile, speech the 90th, their difference the usable SNR), energy below 60 Hz where no voice lives, a narrow band at 50 and 60 Hz for mains hum, 5-9 kHz for sibilance, and peak level for clipping. Only what crosses its own threshold is applied, and each repair has a ceiling:

A well-recorded voice therefore comes out untouched, and the manifest records that every check was made and every one was below its threshold. Clipping is reported rather than repaired, because reconstructing a flattened waveform is invention. Harmonics of a mains hum are left alone for the same reason: notching 100 or 120 Hz thins the voice.

Every render says what it changed and how to undo it:

rendered /project/edit/final/final.mp4
audio was changed by measured voice cleanup:
  - high-pass 80 Hz: energy below 60 Hz is -12.1 dB under the voice, above the -15 dB threshold
  - notch 50 Hz: a narrow band there is -13.1 dB under the voice and concentrated enough to be a tone rather than rumble
  - denoise 10 dB: the noise floor is -46.9 dBFS, only 9.3 dB under speech
  to undo: re-render with --no-audio-cleanup, or set audio_cleanup_policy: none in .social-video/config.yaml

The measurement, the applied steps, the skipped steps with their reasons and the ceilings all land in the render manifest, and brand QA checks the repair against them. Source media is never modified, so the untouched audio is always one re-render away.

Captions are never shortened to fit

The values above are the reference geometry for a 9:16 short, and they are the defaults for any key a config omits:

Layout is measured in Python against the project font's own metrics and handed to the compositor as explicit lines, so the browser never wraps, clamps or ellipsizes anything. When a cue does not fit, the order of resort is: wrap at word boundaries, then shrink that cue's font (down to 72% of the contracted size), then fail with the cue and its text named. Nothing is ever truncated, and config validate refuses a geometry whose own text cannot be drawn.

Brand QA checks the geometry that was drawn, not the one that was asked for: whether any cue lost text, whether any gained an ellipsis the transcript did not have, whether the widest line stayed inside the box, whether a cue was longer than the contract allows, and whether the applied style matches the contract field by field.

Private local brand assets may remain gitignored. Discovery reads them locally but never copies them into this plugin or uploads them.

The root, cache, and state decisions are recorded in ADR-003.

Guided multi-model workflow

The default interactive workflow persists state in edit/workflow-state.json, so changing model or starting a new Codex conversation does not lose progress:

Astra and Claude Fable 5.1 are not normal steps. They are suggested only for a genuinely difficult narrative reconstruction or deep technical impasse. Claude recommendations apply only when that provider is available in the user's host; the skill never claims to switch provider or model automatically. Model names are mapped from stable conceptual tiers in one configuration file, so future model-name changes do not alter workflow schemas.

Initialize and inspect a workflow:

social-video-agent workflow init interview.mp4 --project-root . \
  --remotion-license free_license_eligible --language pl
social-video-agent workflow status edit --language pl
social-video-agent workflow complete 1 --workspace edit --language pl

At each boundary, including entry to Stages 4 and 5, the agent validates artifacts, saves state, stops, shows the OpenAI recommendation and Claude alternative, explains why, and gives a short continuation prompt. It does not claim to switch the user's model automatically. Ask “Where are we?” or “Continue social-video-agent with Stage 2” in a new conversation to resume.

Use --workflow-mode continuous or say “Do everything with the current model” to run all stages without handoff stops. “Just make it quickly” does the same, but context discovery, artifacts, source safety, and QA remain mandatory. Budgets are economical, balanced, and quality; Astra is never selected automatically.

Example:

User: Use social-video-agent to turn interview.mp4 into a Reel.

Agent: I found docs/brandbook.pdf, docs/video-guidelines.md, and assets/logo.svg. I will use the calm educational profile and project typography. Stage 1 will create the editorial plan without rendering.

Agent after Stage 1: Stage 1 complete. Switch to Terra and send: “Continue social-video-agent with Stage 2.”

Before that first discovery, the agent explains the Remotion license gate and records the user's declaration. Stage 2 writes both edl.json and motion-plan.json. The motion plan is project-aware and reviewable: it may use restrained hook typography, lower thirds, callouts, and a designed end card, but every element needs a timeline interval and reason. It does not add random zooms, transitions, or motion merely to appear busy. A clean edit with no extra graphic is valid when that serves the material better.

What works

• Local transcription with faster-whisper, downloaded on first transcription and cached under ~/.cache/social-video-agent/models/.
• Frame-exact output length on ffmpeg 6 and 7, checked by CI on both.
• Horizontal and vertical sources, compatible CFR 30/29.97/60/59.94 exports, multiple audio tracks, sample-derived AAC timestamps, and deterministic EDL rendering.
• Face-aware 9:16 framing with restrained movement.
• ASS captions with preflight glyph checks, including Zażółć gęślą jaźń.
• Project-aware Remotion motion design with exact pinned dependencies, a durable motion-plan.json, H.264/AAC social export, and real render smoke QA.
• Profiles for talking heads, education, podcasts, stories, landscape, square, and faster social editing.
• Mechanical QA for duration, audio, clipping, silence at cuts, black frames, caption bounds, the opening hook, and per-platform safe zones.
• Composed brand covers, optional credential-gated cover/end-card plates from OpenAI or Gemini image APIs, and validated publishing metadata.
• Licence-gated music beds with sidechain ducking and hand-placed sound effects.
• Speaker-aware framing by correlating mouth movement with the speech envelope.
• One long recording into several standalone shorts, each in its own workspace.
• Content-addressed transcription caching and immutable source files.

CLI

The canonical command is social-video-agent; social-video remains as a compatibility alias.

social-video-agent doctor
social-video-agent context inspect . --workspace edit
social-video-agent workflow init INPUT --project-root . \
  --remotion-license free_license_eligible --language en
social-video-agent workflow status edit
social-video-agent inspect INPUT
social-video-agent transcribe INPUT
social-video-agent pack WORKSPACE
social-video-agent plan INPUT --profile talking-head --goal '45 second Reel'
social-video-agent edit INPUT --profile talking-head
social-video-agent render WORKSPACE --quality preview --output /path/to/preview.mp4
social-video-agent qa WORKSPACE --platform reels
social-video-agent apply-editorial-qa WORKSPACE
social-video-agent platforms
social-video-agent shorts list WORKSPACE
social-video-agent shorts create WORKSPACE --reframe face
social-video-agent image status
social-video-agent image prompt --prompt '...'
social-video-agent image plate WORKSPACE --kind cover_plate --prompt '...' --allow-cloud-image
social-video-agent image register WORKSPACE --file /tmp/plate.png --prompt '...'
social-video-agent image capabilities WORKSPACE --chosen-source video_frame --reason '...' 
social-video-agent cover WORKSPACE --title 'Nikt ci tego nie powie'
social-video-agent deliver WORKSPACE --output DEST --with-captions --cover --publish

Generated plates are optional and off by default, and there are four possible sources. Two of them — a native image tool the host gives its agent (ChatGPT/Codex ImageGen) and a Canva MCP connection — belong to the agent: no Python process can call them, and they need no API key of the user's own. The other two are the OpenAI and Gemini image APIs, which this CLI calls itself when OPENAI_API_KEY or GEMINI_API_KEY is set.

image status therefore reports cli_can_generate and says plainly that it checked local_api_integrations_only, with native_imagegen: unknown_to_cli. A missing API key is not evidence that imagery is unavailable; only the agent can settle that, by looking at its own tool list. Nothing is inferred from a subscription in either direction.

When the agent uses its own tool it takes the guarded prompt from image prompt, calls the tool, and hands the file to image register, which copies it into the workspace, hashes it and records the provenance — naming the tool that really drew it and leaving the model empty when the tool reports none. Consent is required either way, because the prompt still leaves the machine, and image_generation_policy: none blocks every route. Preference order: the user's own material or a real frame, then Canva when a template is called for, then the native tool, then an API with a key, then a locally composed background.

Every word in the finished video is rendered locally by Remotion. Image tools draw backgrounds, illustrations and textures — never captions, headlines, CTAs or logos, which is the only way Polish diacritics, the brand contract and the safe area all survive. See generated-visuals.md and Canva MCP.

social-video-agent doctor checks this project's WSL/Linux environment, FFmpeg, Python dependencies, ASR, fonts, Node, locked Remotion packages, Chrome Headless Shell, plugin files, and workspace. OpenAI's separate codex doctor checks Codex itself; the commands are complementary.

GPU/CUDA is optional. CPU mode is supported and setup never installs NVIDIA drivers, CUDA, Docker, or Whisper models. Models download only when transcription first needs one.

Linux and macOS

The pipeline remains cross-platform. Install Python 3.10–3.13, uv, FFmpeg with libass, fontconfig, a font with the required glyphs, Node.js 20+, and npm, then run:

uv sync --extra dev
npm ci
npx remotion browser ensure
uv run social-video-agent doctor

The WSL bootstrap is intentionally Linux-only and its automatic package installation targets Ubuntu/Debian. Unsupported distributions receive the exact prerequisite list instead of an attempted apt command. macOS users should verify that their FFmpeg build contains the subtitles filter.

Codex plugin and canonical skill

The one canonical skill is skills/social-video-agent/SKILL.md. Plugin metadata follows the current Codex layout:

.codex-plugin/plugin.json
.agents/plugins/marketplace.json
plugins/social-video-agent/
skills/social-video-agent/

Install from GitHub:

codex plugin marketplace add bartlomiejborzucki/social-video-agent
codex plugin add social-video-agent@social-video-agent

For local development and update instructions, see docs/plugin-installation.md. The plugin/skill provides workflow knowledge; it does not contain FFmpeg, a virtual environment, node_modules, or model weights. ChatGPT Work does not execute this local media runtime; use Codex with its agent environment set to WSL2.

Build a lightweight skill upload artifact with:

./scripts/build-skill.sh

This creates dist/social-video-agent-skill.zip without runtime dependencies or private media.

Development and release checks

uv sync --extra dev
uv run pytest
uv run ruff check .
uv run mypy
uv run python scripts/validate_distribution.py
social-video-agent doctor
./scripts/release-check.sh

Tiny media fixtures are generated deterministically with FFmpeg. Automated Linux CI covers unit tests, plugin/skill/package validation, 30/60 fps rendering, paths with spaces and Polish Unicode, caption burn-in, and source immutability. GitHub-hosted Linux is not WSL: actual /mnt/c behavior and WSL detection remain a manual release gate in docs/testing/windows-wsl2-acceptance.md.

A dedicated CI job installs Node 24, the locked Remotion graph and Chrome Headless Shell, type-checks the React composition, renders a synthetic Polish fixture, verifies CFR/AAC timing, runs technical QA, and fully decodes both streams. It still does not pretend that generic Linux is a real /mnt/c test.

The social delivery contract is MP4 with H.264/avc1, yuv420p, BT.709, compatible constant frame rate, AAC-LC stereo at 48 kHz with continuous sample timestamps, faststart, 720×1280 preview, and 1080×1920 final. Rendering first finishes and fully decodes a private file in Linux cache, then atomically publishes preview.mp4 or final.mp4; a partial MP4 never appears under the destination name. Technical QA always writes qa-report.json.

Stage 5 publishes durable, hashed variants without changing editorial state:

social-video-agent deliver edit --output ./delivery \
  --with-captions --no-captions --srt --vtt --poster

delivery-manifest.json records file paths, sizes, formats, QA statuses and SHA-256 values, plus the identical EDL hash from before and after delivery.

Full branded workflow example

social-video-agent config init .
# Edit .social-video/config.yaml and point font_file/logo_file at local assets.
social-video-agent config validate . --workspace edit
# Once per project; later edits and sessions reuse this declaration.
social-video-agent remotion-license attest free_license_eligible \
  --project-root . --accept-terms
social-video-agent workflow init interview.mp4 --project-root . --workspace edit \
  --language pl
# Stages 1-4 create/approve edit-plan.json, edl.json, captions, preview and final.
social-video-agent qa edit --output edit/final/final.mp4
social-video-agent workflow complete 4 --workspace edit --language pl
social-video-agent deliver edit --output ./delivery \
  --with-captions --no-captions --srt --vtt --poster --resolution 720x1280
social-video-agent workflow complete 5 --workspace edit --language pl

If a privacy-safe picture ends before its audio, validation requires an explicit EDL ending strategy. It will not silently create a multi-second still. See the artifact contract.

Privacy and licenses

Media processing is local by default. Models are downloaded once on first use and are not committed. Cloud transcription, if added explicitly, must disclose that media leaves the machine.

The project is Apache-2.0 and incorporates attributed MIT-licensed work from browser-use/video-use. See UPSTREAM.md, NOTICE, and THIRD_PARTY_NOTICES.md.

Known limitations

• The legacy edit command remains a continuous mechanical path; normal skill use now creates a project-aware staged plan and stops at guided handoffs.
• Candidate selection for multi-short workflows is editorial and stays with the agent; shorts create only materialises what was already chosen.
• Platform reserved zones are conservative estimates of the feed UI, not published specifications. Verify on a real device before a campaign.
• Music and effects are never sourced or licence-cleared by this tool. It mixes a local file the project already holds the rights to, and records that claim.
• Speaker framing is audio-correlated mouth motion, not neural active-speaker detection. It declines to choose rather than guessing when two faces move with the audio equally; LR-ASD remains the next step recorded in the audit.
• Generated plates are backgrounds only and need the user's own API key. There is no image generation on a host without one, by design.
• WhisperX alignment and speaker diarization are declared optional dependencies but are not connected to the pipeline.
• Remotion currently composites a deliberately small vocabulary of project-aware hook, lower-third, callout, and end-card graphics. It does not automatically invent bespoke illustration, 3D work, or brand animation. Professional quality still depends on the source, project guidance, Stage 1 decisions, and Stage 3 supervising-editor review.
• Legacy workspaces resume with the FFmpeg renderer for compatibility. New workflows default to Remotion and require the Stage 0 declaration.
• Face-aware framing follows the most prominent face; --reframe speaker adds audio correlation on top of it.
• Actual Windows 11 /mnt/c acceptance must be recorded for each release; generic Linux CI is not equivalent.

See CHANGELOG.md, CONTRIBUTING.md, and SECURITY.md for public project policies.