Building a ... (Part 1 )

...Tiny GPT From Scratch (CPU Only, No Excuses) and learn how a model works in AI

Every few years it's worth reminding yourself how the thing everyone's suddenly obsessed with actually works under the hood. So I built one. Not a wrapper around an API, not a fine-tune, not a prompt-engineering exercise — an actual GPT-style transformer, written from scratch in PyTorch, trained on my own laptop with nothing but an Intel integrated GPU doing the graphics and the CPU doing all the real work.

Meet tiny-gpt: a character-level language model that learns to write Shakespeare-ish text, one letter at a time, in about five minutes flat.

Why bother, when Claude and GPT-4 already exist?

Because using a large language model and understanding one are two completely different skills, and I wanted the second one just as sharp as the first. There's a real difference between "I can call an API" and "I know exactly what happens between the input text and the output text" — and that difference matters when you're the one people are trusting to build AI-driven systems for their business.

So the goal here wasn't to build something impressive. It was to build something small enough to fully understand, end to end — every matrix multiplication, every design decision — while still being a genuine, working transformer and not a toy simplification that skips the hard parts.

The approach

The whole thing is built around one deliberate constraint: it has to train in a few minutes, on ordinary hardware, with no GPU acceleration at all. That single constraint shaped everything else:

1. Character-level, not word-level. The model doesn't know what a "word" is. It only knows individual characters — letters, punctuation, line breaks — and has to learn everything else, including where words even begin and end, purely from patterns in the text.
2. A real transformer, just a small one. Multi-head self-attention, causal masking, feedforward layers, residual connections, layer normalization — the exact same architecture behind every large language model out there, just scaled down to roughly 800,000 parameters instead of hundreds of billions.
3. CPU-only by design. No CUDA, no discrete GPU, nothing exotic. If it can't train reasonably fast on an ordinary business laptop, it doesn't belong in this project.

Training data is the classic "Tiny Shakespeare" set — about a megabyte of his plays, stitched together. Small enough to make CPU training realistic, structured enough that you can actually watch the model's output evolve from random noise into something that at least looks like language.

What's actually happening inside it

Strip away the jargon and a GPT-style model is doing something fairly intuitive: for every character it's seen so far, it's asking "given everything before this point, what character is most likely to come next?" — and it keeps asking that question, one character at a time, feeding its own answer back in as new context for the next guess.

The "attention" mechanism is what lets it weigh which earlier characters actually matter for predicting the next one. Early in a word, it might lean heavily on the last couple of letters. At the start of a new line, it might lean on the fact that the previous line ended with a colon, which in Shakespeare usually means a character's name is about to follow. None of that is hand-coded — the model discovers all of it purely from exposure to the text.

Walking through the code

The whole project is three files, and each one has exactly one job.

model.py — the transformer itself

This is the actual neural network architecture, built from the ground up in PyTorch, no shortcuts.

1. CausalSelfAttention — the core attention mechanism. For every character in the input, it computes how much every earlier character should influence the prediction. "Causal" means a strict one-way mask: position 5 can look at positions 1–5, never at position 6 onward. That's what makes the model autoregressive — it predicts left to right and never gets to cheat by peeking ahead. "Multi-head" means this happens several times in parallel (4 heads here), each one free to learn a different kind of pattern to focus on.
2. FeedForward — a small two-layer network that runs after attention in every block. Attention figures out what to look at; this is where the model does the actual "thinking" about what it means.
3. TransformerBlock — bundles one attention layer and one feedforward layer together, each wrapped with layer normalization and a residual (skip) connection. Those skip connections are what let you stack many blocks on top of each other without training collapsing.
4. TinyGPT — the full model. Stacks 4 TransformerBlocks on top of token and position embeddings, so the model always knows both which character it's looking at and where it sits in the sequence, then finishes with a linear layer that scores every possible next character. This is also where generate() lives — the method that produces new text by feeding the model's own output back in as its next input, one character at a time.

train.py — teaches the model

Loads the training text, builds a character-to-number lookup table, and then just repeats one loop, over and over: grab a random chunk of text, ask the model to predict the next character at every position in it, measure how wrong those predictions were, and nudge every weight in the model a tiny step toward "less wrong." Six hundred passes through that loop, with progress printed every hundred steps, and at the end everything needed to bring the model back to life later — its weights, its vocabulary, its exact size settings — gets saved into a single checkpoint file.

generate.py — puts the trained model to work

Loads that checkpoint back in, rebuilds the model to the exact same shape it was trained with, and calls its generate() method to produce new text starting from a prompt you give it. A --temperature setting controls how safe versus adventurous its character choices are, and --top-k keeps it from ever picking something wildly out of left field.

That's the entire project. No hidden framework doing the heavy lifting behind the scenes — just under three hundred lines of Python, and every one of them earns its place.

Does it actually work?

Real numbers, from an actual run on my own hardware — no rounding, no estimates: training ran for 600 steps and finished in 247 seconds, a little over four minutes, with the loss dropping steadily from 2.64 down to 2.26 as it went. Here's a genuine, unedited sample the model produced afterward, starting from a blank prompt:

Tle thy woull Sourst trof wersendes hur:

Atand hou, yo I whe hell ce ar wind uth ron.

FUESIUK:

$ t aveo mup, hord ind Car utheie s;

ARThover moou allof lerastit our lisecepe;

thout thour usis st cor rsa chin;

Whas chat t flis yon ffr'dit ve o these theve sugucke.

KICENF CENG'

BRD:

Andear I sea

Not coherent Shakespeare, obviously. But look at what's already there without anyone telling it to be: character names in capitals, colons after names exactly where a script would put them, apostrophes landing in plausible spots, line breaks that read like verse rather than a wall of text. All of that structure was learned purely from four minutes of exposure to raw text, on a laptop CPU, by a model with under a million parameters. That's the whole point — not the output quality, but watching real structure emerge from something this small and this fast.

The point was never the output quality. The point was building something where I can point at any single line of code and tell you exactly what it does and why it's there — no black boxes, no "it just works," no vendor magic.

Where this fits

This isn't a product and it's not part of any client work. It's a personal project, built purely to keep the fundamentals sharp — the same instinct that makes a good mechanic take an engine apart every so often even after decades of turning wrenches. Understanding a transformer from the inside out is exactly the kind of grounding that makes the difference when you're advising a business on where AI genuinely helps versus where it's just noise.

How to optimize it

Well, this mambo-jambo text is hard to understand if you are not an AI. God knows what out AI thought when it created this text. Honestly, in the current setup the code was created, the most important ingredient is time (If you dont have excessive GPU Performance,... and even then).

But this something for another "Building a..." episode.