Training a Tiny LLaMA From Scratch
January 10, 2024
A complete LLaMA-style language model can fit into 500,000 parameters and train overnight on a CPU. Brian Greenforest built that machine from scratch in PyTorch on TinyStories, turning the modern Transformer stack into something one engineer can inspect end to end.
The run makes architecture, data, loss, optimization, and sampling tangible through a generated story produced by the resulting weights.
The Whole Model Fits on One Line
The configuration used dimension 103, five layers, eight attention heads, one key-value head, vocabulary size 103, multiple-of-four alignment, sequence length 240, and dropout 0.2.
Those choices expose grouped-query attention, residual blocks, normalization, token representation, autoregressive masking, and next-token training at a scale that invites direct study.
Train the Architecture, Then Read Its Voice
An overnight CPU run produced recognizably story-shaped text with characters, actions, emotional words, and imperfect grammar. The output makes learned statistical structure visible without hiding behind a giant pretrained checkpoint.
Students and engineers can reproduce the run, inspect every tensor, change one architectural choice, and build genuine intuition for how a LLaMA-family model learns.
Run Training a Tiny LLaMA From Scratch in a Four-Layer Transformer
The complete training run connects this mathematical argument to executable code, data flow, and a working small model.
LinkedIn status when archived: Visible to anyone on or off LinkedIn.
Bizarre fun training a LLaMA-2 FROM SCRATCH 500k parameters on the TinyStories dataset (trained overnight on CPU in PyTorch: ModelArgs(dim=103, n_layers=5, n_heads=8, n_kv_heads=1, vocab_size=103, multiple_of=4, max_seq_len=240, dropout=0.2))
"Once upon a time, there was a very cold! The road had been to be thoughtful and see the animals remembered to eat her new friends. One day, while playing, Tom took a big pot in his friend when he decided to see the sun to let Lucy be dangerous of happy plants. Tom jumped on..."
This comment was also preserved verbatim from Brian Greenforest’s LinkedIn data export or the public post page.
Comment 1 · (2024-01-11 00:23:10 UTC)
Step 36780: training loss 0.7645, validation loss 0.7675.
"In the beginning, it had a wide fly in the sky when the sunshine stepped draw and clapped on its hair. Soon, a little tree saw that she learned that she wrote always gone. She takes nuts to each other, you got a new selfish. You're brave and askfully quiet."
Comments added by Brian Greenforest on LinkedIn
This comment was also preserved verbatim from Brian Greenforest’s LinkedIn data export or the public post page.
Comment 1 · (2024-01-11 00:23:10 UTC)
View the LinkedIn post