Open The Complete Run
A GitHub pull request carries the training log and script in one file. It records the architecture, training command, inference command, loss progression, and generated output from the same run.
Model Shape
From Initialization To Generative Behavior
The command initializes a new model from scratch with a custom 361-token vocabulary, batch size 64, 128-token sequence length, learning rate 3e-4, weight decay 0.1, a 2,500-iteration warmup, and a target maximum of 100,000 iterations.
Training starts at loss 13.0864 with validation loss 13.0707. By step 53,130, logged values occupy roughly the 0.75 to 0.86 range as the model begins producing recognizable story structure.
The inference command loads ./outminimagic23/ckpt.pt and ./data/tok361.model, then runs with --max_new_tokens=10240, --top_k=200, and --temperature=0.8.
The Pseudo-Inverse Unembedding
The implementation computes a pseudo-inverse of the embedding matrix, uses it for unembedding, and propagates gradients through that transformation. This change makes the output projection depend directly on the learned embedding geometry.
Story Structure Emerges
The samples form characters, actions, simple causal turns, and moral-like endings. Repetition and unstable references reveal the pressure of the compact vocabulary and model shape, while the recognizable narrative structure shows what 834,644 trained parameters can organize.
Why The Run Matters
The entire path—from tensor dimensions and tokenizer size through optimization settings, checkpoint selection, loss progression, and output—fits on one engineering surface. That density makes the run useful for teaching, adaptation, and direct experimentation with architecture changes.
Use the source to reproduce the run, change the vocabulary or model geometry, and watch each decision alter the learning curve and generated structure.