Simple Baselines are Competitive with Code Evolution
arXiv:2602.16805v1 Announce Type: new Abstract: Code evolution is a family of techniques that rely on large language models to search through possible computer programs by evolving or mutating existing code. Many proposed code evolution pipelines show impressive performance but are often not compared to simpler baselines. We test how well two simple baselines do over three domains: finding better mathematical bounds, designing agentic scaffolds, and machine learning competitions. We find that simple baselines match or exceed much more sophisticated methods in all three. By analyzing these results we find various shortcomings in how code evolution is both developed and used. For the mathematical bounds, a problem's search space and domain knowledge in the prompt are chiefly what dictate a search's performance ceiling and efficiency, with the code evolution pipeline being secondary. Thus, the primary challenge in finding improved bounds is designing good search spaces, which is done by
arXiv:2602.16805v1 Announce Type: new Abstract: Code evolution is a family of techniques that rely on large language models to search through possible computer programs by evolving or mutating existing code. Many proposed code evolution pipelines show impressive performance but are often not compared to simpler baselines. We test how well two simple baselines do over three domains: finding better mathematical bounds, designing agentic scaffolds, and machine learning competitions. We find that simple baselines match or exceed much more sophisticated methods in all three. By analyzing these results we find various shortcomings in how code evolution is both developed and used. For the mathematical bounds, a problem's search space and domain knowledge in the prompt are chiefly what dictate a search's performance ceiling and efficiency, with the code evolution pipeline being secondary. Thus, the primary challenge in finding improved bounds is designing good search spaces, which is done by domain experts, and not the search itself. When designing agentic scaffolds we find that high variance in the scaffolds coupled with small datasets leads to suboptimal scaffolds being selected, resulting in hand-designed majority vote scaffolds performing best. We propose better evaluation methods that reduce evaluation stochasticity while keeping the code evolution economically feasible. We finish with a discussion of avenues and best practices to enable more rigorous code evolution in future work.
Executive Summary
This study challenges the effectiveness of code evolution techniques by comparing their performance with simple baselines in three domains: mathematical bounds, agentic scaffolds, and machine learning competitions. The results show that simple baselines match or exceed the performance of more sophisticated code evolution methods, highlighting various shortcomings in the development and application of code evolution. The study emphasizes the importance of designing good search spaces and reducing evaluation stochasticity to improve code evolution. The findings have significant implications for the field of code evolution and suggest a need for more rigorous evaluation methods and best practices.
Key Points
- ▸ Code evolution techniques are often compared to more complex methods without evaluating simpler baselines.
- ▸ Simple baselines match or exceed the performance of more sophisticated code evolution methods in three domains.
- ▸ Designing good search spaces and reducing evaluation stochasticity are crucial for improving code evolution.
Merits
Revealing Shortcomings in Code Evolution
The study highlights the limitations and shortcomings of code evolution techniques, providing a more realistic understanding of their capabilities and challenges.
Emphasizing the Importance of Search Spaces
The study underscores the significance of designing good search spaces in code evolution, which is often overlooked in favor of more complex methods.
Improving Evaluation Methods
The study proposes better evaluation methods to reduce evaluation stochasticity, making code evolution more economically feasible and effective.
Demerits
Limited Generalizability
The study's results may not be generalizable to all domains and applications, as the three domains used in the study may not be representative of the broader field of code evolution.
Oversimplification of Complex Issues
The study's focus on simple baselines may oversimplify the complexities of code evolution, potentially masking important nuances and challenges in the field.
Lack of Clear Solutions
The study does not provide clear solutions or recommendations for improving code evolution, but rather highlights the need for more rigorous evaluation methods and best practices.
Expert Commentary
The study's findings have significant implications for the field of code evolution, highlighting the need for more rigorous evaluation methods and best practices. While the study's focus on simple baselines is a welcome addition to the field, it may not fully capture the complexities and nuances of code evolution. Nevertheless, the study's emphasis on designing good search spaces and reducing evaluation stochasticity is a crucial step towards improving code evolution. As the field continues to evolve, it is essential to prioritize the development of more effective and efficient code evolution techniques that can be applied across a range of domains and applications.
Recommendations
- ✓ Develop more rigorous evaluation methods and best practices for code evolution, incorporating feedback from domain experts and practitioners.
- ✓ Prioritize the development of techniques that can design good search spaces and reduce evaluation stochasticity, such as using domain knowledge and meta-learning algorithms.
