ai-cost-latency-control

Controlling cost and latency of LLM features in a TypeScript/NestJS chat backend - budgets per user, tenant, and request, token accounting from provider usage fields, prompt caching and stable prefixes, model routing between small and large models, max output tokens, timeouts, fallbacks, retries with jitter and idempotency, time-to-first-token as the primary UX metric, parallel tool calls, semantic and response caching caveats, cost dashboards and alerts, and degradation modes. Provider-agnostic. Load when estimating or capping spend, reducing response latency, adding retries or fallbacks, or setting up cost monitoring.