Attention, Dynamic Weights, and Mutual Information
July 21, 2023
Attention became transformative because it lets a model compute new weights from the current input. That dynamic weighting connects modern Transformers with Gaussian mixture models, mixture-density networks, and soft windows developed across earlier decades.
Brian Greenforest traces the lineage toward mutual information as a richer language for relationships between learned features.
Dynamic Weights Turn Context Into Computation
A fixed layer applies the same parameters to every example. Attention builds a context-dependent weighting pattern, letting each token or feature select the other information most useful at that moment.
Mixture models and density networks already learned input-dependent coefficients, while soft windows moved focus across structured sequences. Transformers scaled the principle and made it central.
Measure the Information Behind the Weight
Mutual information quantifies how much knowing one variable reduces uncertainty about another. It can reveal dependencies that simple correlation or geometric distance misses.
ML researchers, information theorists, and quantum foundations researchers can connect attention maps with information measures and develop models that expose why a dynamic relationship matters.
Open the Dynamic-Weight Research
The linked research connects attention, sample-dependent weights, and mutual information to the mathematical mechanism discussed here.
LinkedIn status when archived: Visible to anyone on or off LinkedIn.
Of course, attention came from Gaussian mixture models and density networks, soft windows, that was realizing the concept of dynamic weights--historically--as evangelized by Hinton since the 1980s, and we just started to re-understand why we use Transformer at the first place? The concept of mutual information is the best upgrade for quantum mechanics that came from rather sloppy and surprising origin in computational linguistics (you'd expect Shannon at least, or rigorous mathematics?) :-)
https://lnkd.in/g_auZKeZ
Comments added by Brian Greenforest on LinkedIn
This comment was also preserved verbatim from Brian Greenforest’s LinkedIn data export or the public post page.
Comment 1 · (2023-07-21 05:08:54 UTC)
Related to the mechanism how measurements become tokens: https://arxiv.org/abs/2202.02076
Comments added by Brian Greenforest on LinkedIn
This comment was also preserved verbatim from Brian Greenforest’s LinkedIn data export or the public post page.
Comment 1 · (2023-07-21 05:08:54 UTC)
View the LinkedIn post