Seonglae ChoDistillation from teacher network to student network (less parameter)
Pre-trained Teacher network → Student network
- Create and transfer dataset (reusable and identical to existing training methods)
- Train using KL Divergence loss between logit distributions from KL Divergence (more accurate training possible with code below)
Implementing KD begins with training the teacher model to its full capacity. Next, the student model is trained using a specific loss function.
This loss function is based not only the hard labels of the training data but also on the soft outputs (probabilities) generated by the teacher model. These soft outputs convey the teacher's confidence across various classes, offering a more nuanced understanding than hard labels alone. The process typically utilizes a temperature parameter to soften the probabilities, making the distribution more informative and easier for the student to learn from.
Cross Entropy (G. Hinton etal.)
import torch import torch.nn.functional as F temperature = 5.0 alpha = 0.5 def distillation_loss(student_logits, teacher_logits, targets, temperature): soft_loss = F.kl_div(F.log_softmax(student_logits / temperature, dim=1), F.softmax(teacher_logits / temperature, dim=1), reduction='batchmean') * (temperature ** 2) hard_loss = F.cross_entropy(student_logits, targets) return soft_loss + hard_loss teacher_logits = teacher_model(input_data) student_logits = student_model(input_data) loss = distillation_loss(student_logits, teacher_logits, targets, temperature) loss.backward() optimizer.step()
Knowledge Distillation Notion
Knowledge Distillation Usages
Geoffrey Hinton, Jeff Dean
NIPS 2014 Geoffrey Hinton, 오리올 비니알스, Jeff Dean
