Intrinsic Metrics and Mutual Information
April 18, 2023
Data gains meaning when a system can compare states from within the phenomenon itself. Intrinsic metrics define distance through internal geometry, while mutual information measures how strongly variables share structure.
Brian Greenforest develops those ideas into a seven-part path from differential geometry and information theory to nonlinear tensor methods, emergent features, and machine-learning applications.
Build Geometry From Information
The opening chapters connect intrinsic metrics with differential geometry and nonlinear tensor algebra, then develop mutual information for nonlinear systems, feature selection, and clustering.
Kernel methods extend the comparison into nonlinear spaces. A kernel can measure dependence between feature vectors even when ordinary Euclidean distance hides their relationship.
Let Abstract Bits Produce New Concepts
The next stage treats mutual information as a foundation for emergent features. Machine-learning systems can derive comparison, distance, order, and other concepts from measurements that encode the changing state of a system.
Case studies can compare this approach with traditional methods and reveal applications across science and engineering. Researchers can develop the mathematics, run the studies, and turn the full outline into a book and working research program.
Run Intrinsic Metrics and Mutual Information in a Four-Layer Transformer
The complete training run connects this mathematical argument to executable code, data flow, and a working small model.
LinkedIn status when archived: Visible to anyone on or off LinkedIn.
"Intrinsic Metrics and Mutual Information: From Abstract Measurements to Machine Learning"
Chapter 1: Introduction
Motivation for the book
Overview of the key concepts: intrinsic metrics, mutual information, nonlinear tensor algebra, etc.
Chapter 2: Intrinsic Metrics and Differential Geometry
Review of differential geometry and intrinsic metrics
The relationship between intrinsic metrics and nonlinear tensor algebra
Applications of intrinsic metrics in machine learning
Chapter 3: Mutual Information and Information Theory
Overview of information theory and mutual information
Calculation of mutual information for nonlinear systems
Applications of mutual information in feature selection and clustering
Chapter 4: Nonlinear Tensor Algebra and Kernel Methods
Overview of nonlinear tensor algebra
Application of kernel methods to nonlinear tensor algebra
Use of kernel functions to define the measure of linear dependency between feature vectors in a nonlinear space
Chapter 5: From Abstract Measurements to Emergent Features
Use of mutual information as an abstract foundation for emergent features
Development of machine learning experiments that rely on abstract bits containing information about system states
Analysis of the resulting emergent concepts of comparison, distance, order, etc.
Chapter 6: Applications of Intrinsic Metrics and Mutual Information in Machine Learning
Case studies of using intrinsic metrics and mutual information in machine learning tasks
Comparison with traditional machine learning techniques
Future directions and potential limitations of the approach
Chapter 7: Conclusion
Summary of key points
Implications of the approach for the field of machine learning and beyond
Suggestions for further reading and research