Autoregression, Nonlinear Processes, and Embedding
August 30, 2023
A nonlinear process can become tractable when a model embeds its history into a state where simpler prediction works. Brian Greenforest connects linear autoregression over reconstructed state with the longer mathematical lineage behind embedding.
Whitney’s 1936 theorem established broad geometric embedding results; Takens’s 1981 theorem showed how delayed observations can reconstruct the state of a dynamical system.
Create State From a History of Measurements
A scalar time series hides the full system that generated it. Delay coordinates collect several past values into one vector, revealing a state space whose geometry can preserve the original dynamics.
Autoregression can then operate on that representation. The apparent linear predictor sits one level above a nonlinear process and one level below the observation sequence.
Connect Classical Embedding to Learned Embeddings
Transformer embeddings serve a different formal role, yet both traditions search for a representation where relationships become easier to compute.
Mathematicians and ML researchers can trace that lineage precisely, compare delay-coordinate and learned spaces, and design models that preserve more of the generating dynamics.
Run Autoregression, Nonlinear Processes, and Embedding in a Four-Layer Transformer
The complete training run connects this mathematical argument to executable code, data flow, and a working small model.
LinkedIn status when archived: Visible to anyone on or off LinkedIn.
I just came up with an idea of use a linear autoregression to model the nonlinear autoregression generating function of a heavily nonlinear process (yes, three degrees of indirection!) and found out it was the generalization of the Hilbert's 13th problem caled "Embedding Theorem" invented in 1936 (sic!)
I suspect the word "embdedding" in the transformer model came not from k-NN, but from back then.
Fair not is that it wasn't possible to use until 1981.
#transformer #attention #gpt #llm #embedding #vectordatabase #vectorsearch #nonlinear #autoregression #linearregression