As I will be starting an industrial ph.d. soon on the topic of model compression and edge deployment I thought it would be nice to familiarise myself a bit more practically with model compression. I sought out courses and found some on deeplearning.ai. They have one on Fundamentals and a successor that goes a bit more in Depth.
I liked the first one as an approachable introduction to the subject. It was also pretty short so it was an easy investment to make. One thing that I miss is that you don't really have to be actively engaged. It would be nice with some more hands-on exercises that forces you to think through the concepts.
I'll be on the lookout for more practical exercises throughout the summer to get more familiar with quantization, pruning and knowledge distillation.
FUTO recently appeared in my YouTube recommendations and I think that the algorithm is pretty spot-on in this case. It's an organization that works against the pattern of computers being users of people, and not the other way around. An interesting point during the talk by Peter Wang is that since it has been shown that the internal representations of knowledge inside language models all converge, it could be argued that the more we train these models the more it should be viewed as public, common knowledge. Much like how digits of Pi isn't intellectual property.