Why Code, Why Now: Learnability, Computability, and the Real Limits of Machine Learning
arXiv:2602.13934v1 Announce Type: new Abstract: Code generation has progressed more reliably than reinforcement learning, largely because code has an information structure that makes it learnable. Code provides dense, local, verifiable feedback at every token, whereas most reinforcement learning problems do not. This difference in feedback quality is not binary but graded. We propose a five-level hierarchy of learnability based on information structure and argue that the ceiling on ML progress depends less on model size than on whether a task is learnable at all. The hierarchy rests on a formal distinction among three properties of computational problems (expressibility, computability, and learnability). We establish their pairwise relationships, including where implications hold and where they fail, and present a unified template that makes the structural differences explicit. The analysis suggests why supervised learning on code scales predictably while reinforcement learning does n
arXiv:2602.13934v1 Announce Type: new Abstract: Code generation has progressed more reliably than reinforcement learning, largely because code has an information structure that makes it learnable. Code provides dense, local, verifiable feedback at every token, whereas most reinforcement learning problems do not. This difference in feedback quality is not binary but graded. We propose a five-level hierarchy of learnability based on information structure and argue that the ceiling on ML progress depends less on model size than on whether a task is learnable at all. The hierarchy rests on a formal distinction among three properties of computational problems (expressibility, computability, and learnability). We establish their pairwise relationships, including where implications hold and where they fail, and present a unified template that makes the structural differences explicit. The analysis suggests why supervised learning on code scales predictably while reinforcement learning does not, and why the common assumption that scaling alone will solve remaining ML challenges warrants scrutiny.
Executive Summary
The article 'Why Code, Why Now: Learnability, Computability, and the Real Limits of Machine Learning' explores the rapid progress in code generation compared to reinforcement learning, attributing this to the inherent information structure of code. The authors propose a five-level hierarchy of learnability based on this structure, arguing that the ceiling of machine learning (ML) progress is more dependent on task learnability than model size. They distinguish between expressibility, computability, and learnability, highlighting their relationships and implications for ML. The analysis suggests that supervised learning on code scales predictably due to its dense, local, and verifiable feedback, unlike many reinforcement learning problems. The article cautions against the assumption that scaling alone will resolve remaining ML challenges.
Key Points
- ▸ Code generation has progressed more reliably than reinforcement learning due to its learnable information structure.
- ▸ A five-level hierarchy of learnability is proposed, based on information structure, to assess the ceiling of ML progress.
- ▸ The distinction among expressibility, computability, and learnability is crucial for understanding ML limitations.
- ▸ Supervised learning on code scales predictably because of its dense, local, and verifiable feedback.
- ▸ Scaling alone may not solve remaining ML challenges, warranting scrutiny of the assumption.
Merits
Comprehensive Framework
The article provides a rigorous framework for understanding the learnability of different tasks, which is crucial for advancing ML research.
Clear Distinctions
The formal distinction among expressibility, computability, and learnability helps clarify the structural differences in computational problems.
Practical Insights
The analysis offers practical insights into why certain ML approaches succeed where others fail, guiding future research and application.
Demerits
Limited Scope
The focus on code and reinforcement learning may limit the generalizability of the findings to other areas of ML.
Abstract Nature
The theoretical nature of the analysis may make it less accessible to practitioners seeking immediate, actionable insights.
Assumptions
The article assumes that the hierarchy of learnability is universally applicable, which may not hold in all contexts.
Expert Commentary
The article presents a compelling argument for why code generation has seen more reliable progress compared to reinforcement learning, rooted in the inherent structure of code. The proposed hierarchy of learnability offers a valuable framework for assessing the potential of different ML tasks. However, the focus on code and reinforcement learning, while insightful, may limit the broader applicability of the findings. The distinction among expressibility, computability, and learnability is particularly noteworthy, as it provides a clear lens through which to view the structural differences in computational problems. The caution against over-reliance on scaling as a solution to ML challenges is well-founded and warrants further exploration. Overall, the article contributes significantly to the ongoing discourse on the limits and potential of machine learning, offering both theoretical and practical insights that can guide future research and policy decisions.
Recommendations
- ✓ Future research should explore the applicability of the learnability hierarchy to other domains beyond code and reinforcement learning.
- ✓ Practitioners should focus on tasks with high learnability to achieve more reliable and scalable ML progress, leveraging the dense feedback available in such tasks.
