Distillation, or, How to Steal a Model

Linear Digressions by Ben Jaffe and Katie Malone

This week we’re covering model distillation: the technique of using a large "teacher" model's outputs to train a smaller, cheaper "student" model that mimics it. They cover the two big reasons labs do this — making light...View more