Chapter 1 of 10

Predict one character

I want the model to continue text such as 1+1= with an answer. It will choose one character, add it to the input and repeat until it decides to stop.

Try the prediction loop

Choose each character yourself, then end the answer. We'll build the network later.

Starting textYour next character

1+1=

Append your chosen character to the starting text.

Start with the source

I use Python with PyTorch, a library that handles the calculations and works out how to adjust weights during training. I define the network myself and start with random weights.

Set up the project, then follow the code in each chapter. Commands below run from the course folder with the Python environment activated.

course/
  scripts/prepare_data.py   # create addition examples
  tiny_gpt/tokenizer.py     # characters to numbers
  tiny_gpt/model.py         # the network
  tiny_gpt/train.py         # learn the weights
  tiny_gpt/generation.py    # predict new characters

You can run the supplied files as you learn, or write your own versions from the displayed source. For now, the loop only appends a supplied character:

prefix = "1+1="     # text so far
next_character = "2"  # your choice in this exercise
prefix = prefix + next_character
print(prefix)        # 1+1=2

Technical details

The completed model will choose each character from its output scores, then add it to the input for the next prediction.

Code and training steps