AI Before AI
Eons ago (i.e. before 2012), “AI” referred to an assortment of techniques that people hoped would get computers to be intelligent, even if it didn’t involve any “learning” by the machine. For example, Deep Blue was considered a breakthrough in AI in the 1990s, even though it consisted of hand-coded chess heuristics combined with a powerful computer for searching through hundreds of millions of moves per second. My textbook on artificial intelligence from 2012 (published in 2009) described many different “hand-coded” techniques to have computers solve problems, and had only few chapters on what AI soon came to mean.
Machine Learning & Neural Nets
Machine Learning (ML) - Give a machine enough structured data with labels, and it can learn patterns from that data using simple methods such as linear regression. Early ML techniques go back many decades, but it was in the 2000s that large tech companies really began to use them, such as in recommendation systems on Amazon and Netflix. These systems are generally efficient and it’s often straightforward to interpret how they work. For more on machine learning (and on neural networks), see my previous post:
Deep Learning & Artificial Neural Networks (ANNs) - ANNs use multiple layers of “artificial neurons” to learn more complex patterns than what simpler ML techniques can learn. “Deep learning” just means artificial neural networks with multiple hidden layers. Here’s a simple Google visualization of a multi-layered neural network (from my previous post):
ANNs were invented at the beginning of the computer age, but there wasn’t initially enough data and computation power for them to be practical. In 2012, this changed with AlexNet, which demonstrated that deep learning could classify images better than any alternative. Over the next few years, tech companies started to use ANNs for image and face recognition, language translation, speech recognition and more.
Eventually the word “artificial” was dropped and they were just called “neural nets”. When people said “AI” between 2013 and 2019, they meant machine learning, especially deep learning. To be “AI”, the computer had to “learn”, not follow hard-coded rules.
Large Language Models
Many approaches were tried for creating generally intelligent computers, such as training computers to play games. These approaches were successful at making superhuman AIs at playing games and also had applications in other domains, but this didn’t end up being the key to a more general intelligence. That key turned out to be language. Language allows people to communicate with computers and it also allows computers to think more generally. As Scott Alexander said about GPT-2 (in 2019) “Learn language really well, and you understand reality”.
Before 2017, many had tried to train computers to learn language using a sequential technique where the computer processes sentences word-by-word. This approach was slow to train and and the AI struggled to maintain longer contexts. In 2017, researchers at Google published Attention is All You Need, which contained the key to teaching machines language. They demonstrated how a new type of neural network called a “transformer” can process words simultaneously and keep track of how each word relates to every other word. By giving the AI training data of existing text and asking it to predict the next “token” (word, usually), the AI was able to detect and learn the patterns of language. Given enough data and using large enough AI models (“Large language models”), AI was eventually able to master language. The “transformer” architecture became the key to modern AI, including domains beyond language such as image generation.
When ordinary people say “AI” nowadays, they are referring to LLMs, along with other transformer-based ML systems. While modern AI has also been trained with other techniques beyond "next-token” prediction and can do more than just output text, this ability to understand language turned out to be the key to building a general intelligence.