Task-Specific Knowledge Distillation: Matching Large Teacher Accuracy with 10x Fewer Parameters
Knowledge distillation (KD) enables training compact student models that match large teacher model accuracy. We conduct a systematic empirical study comparing standard KD (Hinton et al., 2015), feature-level matching, attention transfer, and combined approaches. Through experiments on classification tasks with 10x parameter reduction (2M teacher → 200K student), we demonstrate that combined distillation achieves 98.8% of teacher accuracy versus 92.8% without distillation. We analyze the effectiveness of different loss functions, calibration techniques, and architectural constraints. Our results show feature-level KD provides 0.3% additional benefit over standard KD, while attention transfer contributes minor improvements. Combined approaches achieve best results with <2% accuracy degradation. These findings enable practical deployment of efficient models with minimal quality loss, critical for mobile and edge inference.