This was a small LLaMA-style training run: about 500k parameters on TinyStories, trained overnight on CPU in PyTorch.

The value is that model shape and output stay together. The page preserves the architecture settings and a generated sample so the run can be read as a concrete artifact, not just a claim that a model was trained.

Small-model runOriginally posted 2024-01-10; expanded here around the training artifact.

Article focus: TinyStories, 500k parameters, CPU training, PyTorch, architecture settings, and generated sample text.

The Run

The model used TinyStories and trained overnight on CPU in PyTorch. The reported size was about 500k parameters.

The preserved settings were: dim 103, 5 layers, 8 heads, 1 key/value head, vocabulary size 103, multiple_of 4, maximum sequence length 240, and dropout 0.2.

Sample Output

Once upon a time, there was a very cold! The road had been to be thoughtful and see the animals remembered to eat her new friends. One day, while playing, Tom took a big pot in his friend when he decided to see the sun to let Lucy be dangerous of happy plants. Tom jumped on...

The sample is useful because it shows the level of the run directly: local grammar and story-like texture appear, but the model is still very small and the generated sequence does not yet maintain strong coherence.