Recursive Concept Evolution for Compositional Reasoning in Large Language Models
arXiv:2602.15725v1 Announce Type: new Abstract: Large language models achieve strong performance on many complex reasoning tasks, yet their accuracy degrades sharply on benchmarks that require compositional reasoning, including ARC-AGI-2, GPQA, MATH, BBH, and HLE. Existing methods improve reasoning by expanding token-level search through chain-of-thought prompting, self-consistency, or reinforcement learning, but they leave the model's latent representation space fixed. When the required abstraction is not already encoded in this space, performance collapses. We propose Recursive Concept Evolution (RCE), a framework that enables pretrained language models to modify their internal representation geometry during inference. RCE introduces dynamically generated low-rank concept subspaces that are spawned when representational inadequacy is detected, selected through a minimum description length criterion, merged when synergistic, and consolidated via constrained optimization to preserve s
arXiv:2602.15725v1 Announce Type: new Abstract: Large language models achieve strong performance on many complex reasoning tasks, yet their accuracy degrades sharply on benchmarks that require compositional reasoning, including ARC-AGI-2, GPQA, MATH, BBH, and HLE. Existing methods improve reasoning by expanding token-level search through chain-of-thought prompting, self-consistency, or reinforcement learning, but they leave the model's latent representation space fixed. When the required abstraction is not already encoded in this space, performance collapses. We propose Recursive Concept Evolution (RCE), a framework that enables pretrained language models to modify their internal representation geometry during inference. RCE introduces dynamically generated low-rank concept subspaces that are spawned when representational inadequacy is detected, selected through a minimum description length criterion, merged when synergistic, and consolidated via constrained optimization to preserve stability. This process allows the model to construct new abstractions rather than recombining existing ones. We integrate RCE with Mistral-7B and evaluate it across compositional reasoning benchmarks. RCE yields 12-18 point gains on ARC-AGI-2, 8-14 point improvements on GPQA and BBH, and consistent reductions in depth-induced error on MATH and HLE.
Executive Summary
This article proposes Recursive Concept Evolution (RCE), a framework that enables large language models to modify their internal representation geometry during inference, addressing compositional reasoning challenges. RCE introduces dynamically generated concept subspaces, selected through a minimum description length criterion, and consolidated via constrained optimization. Evaluations on compositional reasoning benchmarks demonstrate significant performance gains. While the introduction of RCE presents a compelling solution to compositional reasoning, its effectiveness is contingent upon the model's ability to adapt and generalize. As large language models continue to be a cornerstone of natural language processing, RCE's implications for future research and applications are substantial. The framework holds promise for addressing the limitations of current methods, which rely on recombining existing representations, and may serve as a stepping stone for more sophisticated compositional reasoning techniques.
Key Points
- ▸ RCE enables large language models to modify their internal representation geometry during inference
- ▸ RCE introduces dynamically generated concept subspaces through a minimum description length criterion
- ▸ RCE demonstrates significant performance gains on compositional reasoning benchmarks
Merits
Strength in Adaptability
RCE's ability to dynamically generate and adapt concept subspaces addresses the limitations of existing methods, which rely on recombining existing representations.
Improves Performance on Compositional Reasoning
RCE yields substantial performance gains on compositional reasoning benchmarks, demonstrating its effectiveness in addressing the challenges of compositional reasoning.
Demerits
Dependence on Model Adaptability
RCE's effectiveness is contingent upon the model's ability to adapt and generalize, which may pose challenges for certain models or applications.
Potential for Overfitting
The dynamic generation of concept subspaces may increase the risk of overfitting, which could negatively impact the model's performance on unseen data.
Expert Commentary
The introduction of RCE is a significant development in the field of large language models, as it addresses the limitations of existing methods and enables compositional reasoning. While the article demonstrates the effectiveness of RCE on compositional reasoning benchmarks, further research is necessary to fully understand its potential and limitations. The framework's adaptability and ability to generate new representations make it an attractive solution for addressing the challenges of compositional reasoning. However, the dependence on model adaptability and potential for overfitting are concerns that must be addressed through careful experimentation and evaluation.
Recommendations
- ✓ Future research should focus on integrating RCE with existing methods, such as chain-of-thought prompting and self-consistency, to further enhance performance and adaptability.
- ✓ Careful experimentation and evaluation are necessary to understand the potential for overfitting and to develop strategies for mitigating this risk.
