Hessian-Free Methods Did Not Vanish
Hessian-free optimization is part of the history of training deep and recurrent networks. Martens' work and later work with Sutskever showed that second-order structure could be useful when ordinary gradient descent struggled with curvature and long dependencies.
Those methods did not vanish because they were conceptually wrong. They became less visible as large-scale training practice standardized around first-order methods, better initialization, normalization, hardware-friendly batches, and optimizers such as AdamW.
Second-order ideas still matter when curvature, conditioning, and stability determine whether training moves efficiently. Some later work combines those ideas with modern optimizer practice instead of treating them as separate eras.
The useful takeaway is historical continuity: transformer-era training did not erase earlier optimization work. It selected pieces that fit the data scale, hardware, and engineering constraints of the moment.