Deserved. word2vec is a beautiful peice of science. The hypothesis behind it was so amazing. I kinda feel like everything else in LLMs (including the attention mechanism itself) is just engineering around the representation learning breakthrough that word2vec demonstrated
And with applications far outside the original scope, which I think is one of the hallmarks of foundational discoveries. As soon as you have that one puzzle piece a whole lot of other stuff comes into focus.
The idea that a word is defined by the company it keeps was decades old. Mikolov’s insight was that you could learn it at scale with a very simple model, and that meaning would fall out as geometry. Obvious in hindsight, foundational in practice.
I still in awe how could someone understand well in math, is it pure nature or nurture. Could any average possible to grasp the beauty of math, starting from high school level math understanding.
5 comments