The 2017 paper that every AI chatbot is built on
Originally published at denisostapenko.com . It gets to me when people turn a language model into a miracle. Go read how it works, it's all out there. So one day I sat down and went through the papers it stands on. There aren't many. The whole industry, every chatbot, every gadget with "AI" stuck on it, rides on a short chain of work, and almost all of it fits in twenty years. You can walk it fast. And once you do, the miracle evaporates, and the fear that's off the mark goes with it, and so do the inflated expectations. Here's the chain, link by link. The idea is old: language is prediction The thought that language can be predicted isn't new. Claude Shannon, back in 1948, in the paper that started information theory, measured how predictable an English letter is when you know the ones before it. He showed text is full of redundancy, that knowing the start, the next character is easier to guess than you'd think. There's the seed of everything ...