Academic

EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair

arXiv:2603.05553v1 Announce Type: cross Abstract: Function-calling agents -- large language models that invoke tools and APIs -- require high-quality, domain-specific training data spanning executable environments, backing databases, and diverse multi-turn trajectories. We introduce EigenData, an integrated, self-evolving platform that automates the full data lifecycle through a multi-agent architecture. A top-level orchestrator, EigenCore, coordinates three specialized sub-systems: DatabaseAgent for realistic domain database construction, CodingAgent for verified executable environment generation with iterative test-debug loops, and DataAgent for multi-turn trajectory synthesis with self-evolving prompt optimization. Cross-component feedback ensures consistency across all artifacts. We apply EigenData to audit and repair the Berkeley Function-Calling Leaderboard (BFCL-V3), identifying systematic errors in function schemas, implementations, and reference trajectories, automatically co

J
Jiaao Chen, Jingyuan Qi, Mingye Gao, Wei-Chen Wang, Hanrui Wang, Di Jin · · 1 min read · 20 views
#cs.SE #cs.AI #cs.CL

arXiv:2603.05553v1 Announce Type: cross Abstract: Function-calling agents -- large language models that invoke tools and APIs -- require high-quality, domain-specific training data spanning executable environments, backing databases, and diverse multi-turn trajectories. We introduce EigenData, an integrated, self-evolving platform that automates the full data lifecycle through a multi-agent architecture. A top-level orchestrator, EigenCore, coordinates three specialized sub-systems: DatabaseAgent for realistic domain database construction, CodingAgent for verified executable environment generation with iterative test-debug loops, and DataAgent for multi-turn trajectory synthesis with self-evolving prompt optimization. Cross-component feedback ensures consistency across all artifacts. We apply EigenData to audit and repair the Berkeley Function-Calling Leaderboard (BFCL-V3), identifying systematic errors in function schemas, implementations, and reference trajectories, automatically correcting them through coordinated schema refinement, code-level bug fixes, and trajectory modification, and introducing an outcome-aware evaluation protocol that assesses task success via database-state correctness rather than turn-level trajectory matching. We demonstrate that the repaired benchmark, coupled with outcome-aware metrics, produces model rankings substantially better correlated with human judgments of functional correctness.

Executive Summary

EigenData presents a novel, self-evolving multi-agent framework designed to automate the end-to-end lifecycle of data synthesis, auditing, and repair for function-calling agents. By integrating specialized agents—DatabaseAgent, CodingAgent, and DataAgent—under a centralized orchestrator (EigenCore), the platform addresses a critical gap in the availability of high-quality domain-specific data for large language models that invoke external tools. The system’s self-evolving capabilities, particularly through iterative prompt optimization and cross-component feedback, represent a significant advancement in data lifecycle automation. Application to the BFCL-V3 benchmark demonstrates tangible improvements in audit accuracy and evaluation alignment with human-centric correctness metrics, suggesting broader applicability in benchmark curation and AI evaluation.

Key Points

  • ▸ Multi-agent architecture with specialized sub-systems for data lifecycle automation
  • ▸ Self-evolving capabilities via iterative prompt optimization and cross-component feedback
  • ▸ Application to BFCL-V3 audit and repair with improved outcome-aware evaluation protocol

Merits

Innovation

EigenData introduces a cohesive integration of autonomous agents for end-to-end data lifecycle management, a previously fragmented area.

Impact

The application to BFCL-V3 demonstrates measurable gains in audit fidelity and evaluation alignment with human judgments, validating the platform’s efficacy.

Demerits

Scalability Concern

While effective on a specific benchmark, the system’s complexity may pose challenges in scaling to heterogeneous or multi-domain environments without additional customization.

Generalizability Limitation

The focus on a single benchmark (BFCL-V3) limits immediate applicability to broader domains without empirical validation.

Expert Commentary

EigenData represents a sophisticated synthesis of agent-based automation and self-evolutionary learning, addressing a persistent bottleneck in the production of reliable training data for function-calling LMs. The architecture’s modularity—allowing independent specialization of database, code, and trajectory synthesis—is a masterstroke in design, enabling targeted problem-solving without systemic compromise. Moreover, the shift from trajectory-based evaluation to database-state correctness as the primary success criterion is not merely a technical tweak; it is a paradigm shift that aligns AI evaluation with real-world usability. This reflects a deeper philosophical commitment to functional integrity over syntactic correctness. While the limitations in scalability and generalizability are valid concerns, they are offset by the platform’s foundational impact on how data integrity is conceptualized and operationalized in AI research. This work may catalyze a new standard in the field, much like the shift from accuracy to robustness metrics did in machine learning a decade ago.

Recommendations

  • ✓ Researchers should integrate outcome-aware evaluation metrics into future benchmark design for function-calling agents.
  • ✓ Academic institutions and AI labs should explore adapting EigenData’s multi-agent framework as a modular component in their data curation pipelines, particularly for domains requiring high-fidelity training data.

Sources

Related Articles