Optimizer History Around Attention and AdamW
Transformer history is usually told as an architecture story, but optimizer choice is part of the mechanism. LeNet-era systems and later ImageNet training made heavy use of SGD with momentum; transformer work arrived in a period where Adam-style methods were already central to large neural-network training.
AdamW matters because decoupled weight decay changes how regularization is applied during optimization. That is not a minor implementation detail when training stability, scale, and reproducibility are the actual problem.
The useful reading is not that every newer optimizer replaces the older ones. It is that architecture, normalization, initialization, learning-rate schedules, batch size, regularization, and optimizer dynamics form one training system.
When a model works, the article should keep those parts connected. Attention was not deployed in isolation; it survived inside a training recipe.