What Is an LLM? From AI to GPT · Build an LLM From Scratch

Before we write a single line of PyTorch, it pays to have the whole journey on one map: what a large language model (LLM) actually is, why the old way of building language software gave way to it, and the exact two-step recipe — pretrain, then fine-tune — that the rest of this book implements from scratch.

1.1 What is an LLM?

An LLM is a deep neural network trained on enormous amounts of text to understand and generate human-like language. The word large is doing double duty: it refers both to the model’s size — tens to hundreds of billions of parameters (GPT-3 has 175 billion) — and to the data it was trained on: hundreds of billions of words.

LLMs sit inside a nest of older, broader ideas. Artificial intelligence contains machine learning; machine learning contains deep learning ; and LLMs are a particular, very large kind of deep-learning model. They are also a form of generative AI — software that produces new content (here, text) rather than only classifying existing content.

AI ⊃ machine learning ⊃ deep learning ⊃ LLMs. LLMs also belong to generative AI (dashed) — they generate text.

Why deep learning changed everything

The break with classical natural-language software is that LLMs need no manual feature engineering. In traditional machine learning, a human expert had to decide, up front, which measurable properties of the input the model should look at. For a spam filter that meant hand-picking features: the frequency of trigger words like “prize”, “win”, “free”; the count of exclamation marks; whether the text is ALL CAPS; the presence of suspicious links. Deep learning throws that step away — the network learns its own features directly from raw text.

The extra amber box — hand-designed features — is exactly what deep learning removes.

1.2 Applications of LLMs

Because a single pretrained model captures so much about language, one LLM can be pointed at a remarkable range of tasks: drafting and completing text, machine translation, sentiment analysis, summarization, question answering, and powering chatbots and virtual assistants. The same core network that autocompletes a sentence can, with the right prompt or a little fine-tuning, answer a support ticket or classify an email. This versatility is why LLMs have become a general-purpose tool rather than one-task software.

1.3 The two stages: pretraining and fine-tuning

Building and using an LLM happens in two distinct training stages — the backbone of this entire book.

Pretraining . The model is trained on a huge pile of ordinary, unlabeled text (web pages, books, Wikipedia) by playing one endless game: predict the next word. Because the “answer” — the next word — is already sitting in the text, no human has to label anything; this is self-supervised learning . The result is a broadly capable foundation model with text-completion and few-shot abilities.
Fine-tuning . The foundation model is then trained a little more on a smaller, labeled dataset to specialize it. The two most common flavors — both built later in this book — are instruction fine-tuning (on instruction–response pairs, Chapter 7) and classification fine-tuning (on text–label pairs, Chapter 6).

The book in one picture: pretrain a foundation model on unlabeled text (Stage 2), then fine-tune it into a classifier or assistant (Stage 3).

1.4 Using LLMs for different tasks — the transformer

Almost every modern LLM is built on the transformer , the architecture introduced in the 2017 paper “Attention Is All You Need.” The original transformer was designed for machine translation and has two halves:

an encoder that reads the entire input sentence and compresses it into context-carrying embedding vectors, and
a decoder that generates the translated sentence one word at a time, consulting those vectors.

Walk through how the original transformer translates “This is an example” into German, one numbered stage at a time:

Input text to be translated

The source sentence "This is an example" enters the encoder side.

1 / 8

The original encoder–decoder transformer translating one sentence (book Figure, p. 5).

Key idea

The single most important part of a transformer is the self-attention mechanism: it lets the model weigh how relevant every token is to every other token in the sequence. Self-attention is what Chapter 3 builds from scratch, and it is the reason transformers handle long-range context so well.

BERT vs. GPT: two children of the transformer

You do not always need both halves. Two hugely influential model families each kept only one:

BERT (Bidirectional Encoder Representations from Transformers) uses the encoder and is trained by masked-word prediction: random words are hidden and the model fills them back in. It sees text in both directions, which makes it strong at classification and understanding.
GPT (Generative Pretrained Transformer) uses the decoder and is trained by next-word prediction, generating text one token at a time — it is autoregressive . This is the design we build in this book.

Same transformer backbone, opposite training games: BERT unmasks; GPT continues.

Interestingly, GPT models — trained only to predict the next word — turn out to be capable at tasks they were never explicitly taught, from translation to summarization. That surprising generality, which grows with scale, is the “emergent behavior” we look at next in §1.5–1.8.

📝 Check yourself: what an LLM is

0 / 4

1.What is the single biggest practical difference between deep learning and traditional machine learning for a task like spam detection?
2.In the two-stage recipe, what exactly is a 'foundation model'?
3.The book builds a GPT-style model. How does GPT differ architecturally from BERT?
4.Pretraining an LLM is described as 'self-supervised'. Where do the training labels come from?