Attention as Dynamic Weighting
Attention is useful to read as dynamic weighting: the model computes which parts of the current context should contribute to the next representation. That connects naturally to older ideas such as soft windows, density networks, mixture models, and adaptive routing of information.
The practical value is that the weights are input-dependent. A fixed set of parameters can still produce a changing computation path as context changes.
Why Mutual Information Belongs Nearby
Mutual information gives one way to ask what a representation preserves about another variable. In attention systems, that question matters because the mechanism is selecting and mixing context at every layer.
This note does not turn mutual information into a universal answer. It is a useful lens for asking what the attention path is carrying, discarding, or making easier for later layers to use.
Useful Scope
The original post linked arXiv:2212.10559. The durable point is narrower than the original wording: attention should be explained through mechanism, history, and information flow rather than treated as a black-box transformer ingredient.