llm-distillation

Distill knowledge from large teacher LLMs into small student models via sequence-level (response-based) knowledge distillation. Covers teacher selection and licensing checks, generating distillation data with Bedrock serverless teachers (DeepSeek-R1, Claude via converse), reasoning distillation (keeping <think>/chain-of-thought for math/code/puzzle domains, stripping it for simple QA), sampling and temperature choices, batch generation with retry/backoff, cost estimation, data curation (dedup, decontamination, verifiable-reward filtering, LLM-judge filtering), producing TRL-ready JSONL for SFT/QLoRA, and evaluating distillation quality with student-vs-teacher relative gates. Use when creating a small specialized model from a large model's outputs, building R1-style distilled reasoners, or preparing teacher-generated training data.

Version

1.0

License

Apache-2.0