On the Mechanism and Dynamics of Modular Addition: Fourier Features, Lottery Ticket, and Grokking
arXiv:2602.16849v1 Announce Type: new Abstract: We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual neurons learn single-frequency Fourier features and phase alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual neurons produce noisy signals, the phase symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify
arXiv:2602.16849v1 Announce Type: new Abstract: We present a comprehensive analysis of how two-layer neural networks learn features to solve the modular addition task. Our work provides a full mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. While prior work has identified that individual neurons learn single-frequency Fourier features and phase alignment, it does not fully explain how these features combine into a global solution. We bridge this gap by formalizing a diversification condition that emerges during training when overparametrized, consisting of two parts: phase symmetry and frequency diversification. We prove that these properties allow the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. While individual neurons produce noisy signals, the phase symmetry enables a majority-voting scheme that cancels out noise, allowing the network to robustly identify the correct sum. Furthermore, we explain the emergence of these features under random initialization via a lottery ticket mechanism. Our gradient flow analysis proves that frequencies compete within each neuron, with the "winner" determined by its initial spectral magnitude and phase alignment. From a technical standpoint, we provide a rigorous characterization of the layer-wise phase coupling dynamics and formalize the competitive landscape using the ODE comparison lemma. Finally, we use these insights to demystify grokking, characterizing it as a three-stage process involving memorization followed by two generalization phases, driven by the competition between loss minimization and weight decay.
Executive Summary
This article provides a comprehensive analysis of how two-layer neural networks learn to solve the modular addition task. The authors propose a mechanistic interpretation of the learned model and a theoretical explanation of its training dynamics. They formalize a diversification condition that enables the network to collectively approximate a flawed indicator function on the correct logic for the modular addition task. The authors also explain the emergence of these features under random initialization via a lottery ticket mechanism and demystify grokking as a three-stage process. This work contributes to a deeper understanding of neural network dynamics and has implications for the development of more efficient and robust neural networks.
Key Points
- ▸ Two-layer neural networks learn single-frequency Fourier features and phase alignment to solve the modular addition task.
- ▸ A diversification condition emerges during training, consisting of phase symmetry and frequency diversification, allowing the network to collectively approximate the correct logic.
- ▸ The lottery ticket mechanism explains the emergence of these features under random initialization.
- ▸ Grokking is characterized as a three-stage process involving memorization followed by two generalization phases.
Merits
Strength in mechanistic interpretation
The authors provide a detailed and mathematically rigorous explanation of the neural network's behavior, shedding light on the underlying mechanisms that enable it to learn and generalize.
Theoretical contribution
The work provides a theoretical framework for understanding the training dynamics of neural networks, which can be applied to a wide range of problems and network architectures.
Demerits
Limited generalizability
The analysis is focused on a specific task (modular addition) and network architecture (two-layer neural network), which may limit the generalizability of the findings to other problems and networks.
High technical requirements
The work assumes a high level of mathematical and computational expertise, which may make it inaccessible to researchers without a strong background in neural networks and optimization.
Expert Commentary
The article presents a rigorous and insightful analysis of the modular addition task, providing a comprehensive understanding of the neural network's behavior. The authors' use of Fourier features and phase alignment to explain the emergence of robust and generalizable features is particularly noteworthy. The work also contributes to the growing field of neural network interpretability, providing a deeper understanding of the underlying mechanisms that enable neural networks to learn and generalize. However, the analysis is limited to a specific task and network architecture, which may limit its generalizability. Additionally, the work assumes a high level of mathematical and computational expertise, which may make it inaccessible to researchers without a strong background in neural networks and optimization.
Recommendations
- ✓ Future research should focus on extending the analysis to other problems and network architectures to improve the generalizability of the findings.
- ✓ Developing more efficient and robust neural network architectures that incorporate mechanisms that promote diversification and phase symmetry could have significant practical implications for AI development.
