Marginals Before Conditionals
arXiv:2603.10074v1 Announce Type: new Abstract: We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6* across a 7* {\eta} range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [202
arXiv:2603.10074v1 Announce Type: new Abstract: We construct a minimal task that isolates conditional learning in neural networks: a surjective map with K-fold ambiguity, resolved by a selector token z, so H(A | B) = log K while H(A | B, z) = 0. The model learns the marginal P(A | B) first, producing a plateau at exactly log K, before acquiring the full conditional in a sharp, collective transition. The plateau has a clean decomposition: height = log K (set by ambiguity), duration = f(D) (set by dataset size D, not K). Gradient noise stabilizes the marginal solution: higher learning rates monotonically slow the transition (3.6 across a 7 {\eta} range at fixed throughput), and batch-size reduction delays escape, consistent with an entropic force opposing departure from the low-gradient marginal. Internally, a selector-routing head assembles during the plateau, leading the loss transition by ~50% of the waiting time. This is the Type 2 directional asymmetry of Papadopoulos et al. [2024], measured dynamically: we track the excess risk from log K to zero and characterize what stabilizes it, what triggers its collapse, and how long it takes.
Executive Summary
The article 'Marginals Before Conditionals' presents an intriguing investigation into the learning dynamics of neural networks. The authors devise a minimal task that isolates conditional learning, where a selector token resolves K-fold ambiguity and the model learns the marginal distribution first. The study reveals a plateau in the learning process, characterized by a height log K and duration f(D), which is stabilized by gradient noise. The findings suggest that the marginal solution is more stable and takes longer to transition to the full conditional, with implications for understanding neural network learning. The research has significant implications for the design of neural networks and the development of robust learning algorithms.
Key Points
- ▸ The authors develop a minimal task to isolate conditional learning in neural networks.
- ▸ The model learns the marginal distribution first, producing a plateau at log K.
- ▸ Gradient noise stabilizes the marginal solution and slows the transition to the full conditional.
Merits
Strength in Methodology
The authors employ a rigorously designed task to isolate conditional learning, allowing for a precise investigation of neural network learning dynamics.
Insights into Learning Dynamics
The study provides valuable insights into the learning process, revealing the role of gradient noise in stabilizing the marginal solution and slowing the transition to the full conditional.
Demerits
Limitation in Generalizability
The minimal task developed in this study may not be directly applicable to more complex tasks, limiting the generalizability of the findings.
Need for Further Investigation
The study's focus on a specific task and learning scenario necessitates further investigation to determine the broader implications for neural network learning.
Expert Commentary
The article 'Marginals Before Conditionals' makes a valuable contribution to the field of neural network learning, providing new insights into the learning dynamics of deep neural networks. The authors' rigorous methodology and analytical framework are exemplary, and their findings have significant implications for the design of neural networks and the development of robust learning algorithms. However, the study's focus on a specific task and learning scenario necessitates further investigation to determine the broader implications for neural network learning. Nonetheless, the research has significant potential for advancing our understanding of neural network learning and its applications in various fields.
Recommendations
- ✓ Future studies should investigate the broader implications of the marginal solution and its stabilization by gradient noise for neural network learning.
- ✓ The research methodology and analytical framework developed in this study should be applied to more complex tasks and learning scenarios to determine their generalizability and scalability.