self-hosted-llm-batch-inference
This skill should be used when a startup without a dedicated ML platform team needs recurring, latency-tolerant offline inference for a self-hosted 1B to 8B generative language model on AWS while minimizing GPU spend and recovering automatically from Spot interruption. It prescribes AWS Batch backed by Amazon ECS Managed Instances Spot, with model artifacts staged in Amazon S3, deterministic input shards, vLLM offline inference on one GPU, bounded retries, and output reconciliation before success. Covers sizing from the real prompt and model footprint, choosing networking without an avoidable idle cost floor, and verifying that results are complete and GPU capacity is gone. Not for online endpoints, training, embeddings, other inference engines, multi-GPU or distributed inference, or one-off experiments where manual rerun is acceptable.
- License
- Apache-2.0
Pinned to revision 11ee755b7c8f, so it is the text this page describes rather than whatever the author pushed since.
Files
- skills/self-hosted-llm-batch-inference/SKILL.md
- skills/self-hosted-llm-batch-inference/references/prepare-model-and-input.md
- skills/self-hosted-llm-batch-inference/references/run-with-aws-batch.md
- skills/self-hosted-llm-batch-inference/references/verify-and-optimize.md
Every link opens the file at its source, pinned to the revision this page describes.