Start With Probability, Dynamics, and Optimization
Joint and conditional probability, Bayes rules, maximum entropy, mutual information, experimental design, Gaussian mixtures, PCA, ICA, hidden Markov models, autoregression, Q-learning, Pontryagin, Euler-Lagrange, dynamic programming, and Hamilton-Jacobi-Bellman equations establish the modeling foundation.
Kernels, convolution, central-limit behavior, nonlinear embeddings, Hilbert problems, topological spaces, and tensor contraction explain how representations carry complex relationships.
Follow the Architecture Into Training and Hardware
RNNs, LSTMs, attention, cross-attention, causal masking, KV caches, RoPE, normalization, GLU-family activations, dropout, initialization, momentum, minibatches, adaptive spans, sparse attention, grouped-query attention, tokenizers, temperature, top-k sampling, and logits define the modern model path.
FMA, GEMM, Hadamard products, bfloat16, quantization, binarized weights, memory movement, and fused kernels determine how that mathematics runs. Students can use this map as a multi-year path from first principles to original Transformer and AI-chip work.
Run The Long Mathematical Path to Transformers in a Four-Layer Transformer
The complete training run connects this mathematical argument to executable code, data flow, and a working small model.
