llm-deployment

Deploy and serve large language models efficiently. Covers vLLM, Text Generation Inference (TGI), Ollama, llama.cpp, ONNX Runtime, TensorRT-LLM, model quantization (GPTQ, AWQ, GGUF, INT4/INT8), KV-cache optimization, continuous batching, speculative decoding, PagedAttention, tensor parallelism, pipeline parallelism, API gateway setup, streaming responses, multi-model serving, GPU memory management, auto-scaling for LLMs, Docker/Kubernetes deployment, and cost-performance optimization. Use when deploying LLMs, optimizing inference speed, reducing serving costs, or setting up LLM infrastructure.

Version

1.0

License

Apache-2.0