Global Low-Rank, Local Full-Rank: The Holographic Encoding of Learned Algorithms
arXiv:2602.18649v1 Announce Type: new Abstract: Grokking -- the abrupt transition from memorization to generalization after extended training -- has been linked to the emergence of low-dimensional structure in learning dynamics. Yet neural network parameters inhabit extremely high-dimensional spaces. How can a low-dimensional learning process produce solutions that resist low-dimensional compression? We investigate this question in multi-task modular arithmetic, training shared-trunk Transformers with separate heads for addition, multiplication, and a quadratic operation modulo 97. Across three model scales (315K--2.2M parameters) and five weight decay settings, we compare three reconstruction methods: per-matrix SVD, joint cross-matrix SVD, and trajectory PCA. Across all conditions, grokking trajectories are confined to a 2--6 dimensional global subspace, while individual weight matrices remain effectively full-rank. Reconstruction from 3--5 trajectory PCs recovers over 95\% of f
arXiv:2602.18649v1 Announce Type: new Abstract: Grokking -- the abrupt transition from memorization to generalization after extended training -- has been linked to the emergence of low-dimensional structure in learning dynamics. Yet neural network parameters inhabit extremely high-dimensional spaces. How can a low-dimensional learning process produce solutions that resist low-dimensional compression? We investigate this question in multi-task modular arithmetic, training shared-trunk Transformers with separate heads for addition, multiplication, and a quadratic operation modulo 97. Across three model scales (315K--2.2M parameters) and five weight decay settings, we compare three reconstruction methods: per-matrix SVD, joint cross-matrix SVD, and trajectory PCA. Across all conditions, grokking trajectories are confined to a 2--6 dimensional global subspace, while individual weight matrices remain effectively full-rank. Reconstruction from 3--5 trajectory PCs recovers over 95\% of final accuracy, whereas both per-matrix and joint SVD fail at sub-full rank. Even when static decompositions capture most spectral energy, they destroy task-relevant structure. These results show that learned algorithms are encoded through dynamically coordinated updates spanning all matrices, rather than localized low-rank components. We term this the holographic encoding principle: grokked solutions are globally low-rank in the space of learning directions but locally full-rank in parameter space, with implications for compression, interpretability, and understanding how neural networks encode computation.
Executive Summary
This study delves into the phenomenon of 'grokking' in neural networks, where a learning process abruptly transitions from memorization to generalization. By analyzing the dynamics of shared-trunk Transformers trained on multi-task modular arithmetic, the authors demonstrate that grokking trajectories reside in a low-dimensional global subspace, while individual weight matrices remain full-rank. This finding challenges the prevailing view of localized low-rank components and introduces the holographic encoding principle. The study's results have significant implications for compression, interpretability, and understanding neural network computation. The findings suggest that neural networks may be more resilient to overfitting than previously thought, as their low-dimensional learning process can produce solutions that resist low-dimensional compression.
Key Points
- ▸ Grokking trajectories in neural networks reside in a low-dimensional global subspace.
- ▸ Individual weight matrices remain full-rank despite low-dimensional learning process.
- ▸ The holographic encoding principle challenges the localized low-rank components view.
Merits
Strength
The study's use of multi-task modular arithmetic provides a controlled environment to investigate grokking dynamics.
Demerits
Limitation
The study's focus on a specific type of neural network architecture (shared-trunk Transformers) may limit generalizability to other architectures.
Expert Commentary
This study represents a significant advancement in our understanding of neural network dynamics and computation. The holographic encoding principle has far-reaching implications for the development of more efficient, interpretable, and compressible neural networks. While the study's focus on a specific architecture may limit generalizability, the findings are robust and suggest that neural networks may be more resilient to overfitting than previously thought. Future research should build upon this work to explore the holographic encoding principle's implications for a broader range of neural network architectures and applications.
Recommendations
- ✓ Future studies should investigate the holographic encoding principle in other neural network architectures and applications.
- ✓ Developing more interpretable neural network architectures should be a priority to better understand computation and overfitting.
