In this deep dive video, we zoom in on model distillation, an advanced technique to build high-performance small language models at a reasonable cost.

We first explain what a model distillation is. Then, we introduce two popular techniques for distillation, logits distillation, and hidden states distillation. We study in detail how they work, and how they’re implemented in the Arcee DistillKit open-source library. Finally, we look at two Arcee models built with distillation, Arcee SuperNova 70B and Arcee SuperNova Medius 14B.