Build a small language model

Let's compare what we're building here with current state-of-the-art language models.

Feature	Our small model	GPT-6 Astra	Claude Opus 5.5
What it does	Trained on simple addition examples	General-purpose, multimodal	General-purpose, multimodal
Parameters (numbers adjusted during training)	413,440	Approx. hundreds of billions to trillions	Approx. hundreds of billions to trillions
Training time	Approx. 51 seconds in our recorded run	Approx. several months	Approx. several months
Text it can use at once (context window)	32 tokens	1,050,000 tokens	1,000,000 tokens
Where it runs	Your browser or computer	Hosted servers	Hosted servers
Downloadable weights (learned parameter values)	Yes	No public download	No public download

Language models behind tools like ChatGPT generate text one token at a time. They learn patterns from training examples, then use those patterns to predict what comes next. A token can be a word, part of a word or a character.

In this tutorial I use addition to build a small model you can run here. A calculator follows an addition rule. My model learns from examples such as 1+1=2 and predicts one character at a time. That means it can also get the answer wrong.

Run 1+1=, then try 1+1. Compare what it writes next.

Try the finished model

Runs on your device. First load is about 14 MB.

Improved model

Your input

Exact input: "1+1="

Model output

No output yet

Input + output

1+1=

Run to see a prediction.

Model and input details

Printable ASCII is passed verbatim. BOS is added once. All 98 tokens remain available. Context: 32 tokens. Run budget: 16 output tokens, capped by remaining context. EOS ends generation. Unexpected PAD or BOS tokens are visible.

Improved model. Epoch 140. 1.74 MB of weights. The WASM runtime is downloaded on first use.

Start building

Code and training steps