Academic

Saarthi for AGI: Towards Domain-Specific General Intelligence for Formal Verification

arXiv:2603.03175v1 Announce Type: new Abstract: Saarthi is an agentic AI framework that uses multi-agent collaboration to perform end-to-end formal verification. Even though the framework provides a complete flow from specification to coverage closure, with around 40% efficacy, there are several challenges that need to be addressed to make it more robust and reliable. Artificial General Intelligence (AGI) is still a distant goal, and current Large Language Model (LLM)-based agents are prone to hallucinations and making mistakes, especially when dealing with complex tasks such as formal verification. However, with the right enhancements and improvements, we believe that Saarthi can be a significant step towards achieving domain-specific general intelligence for formal verification. Especially for problems that require Short Term, Short Context (STSC) capabilities, such as formal verification, Saarthi can be a powerful tool to assist verification engineers in their work. In this paper,

A
Aman Kumar, Deepak Narayan Gadde, Luu Danh Minh, Vaisakh Naduvodi Viswambharan, Keerthan Kopparam Radhakrishna, Sivaram Pothireddypalli · · 1 min read · 9 views
#cs.AI

arXiv:2603.03175v1 Announce Type: new Abstract: Saarthi is an agentic AI framework that uses multi-agent collaboration to perform end-to-end formal verification. Even though the framework provides a complete flow from specification to coverage closure, with around 40% efficacy, there are several challenges that need to be addressed to make it more robust and reliable. Artificial General Intelligence (AGI) is still a distant goal, and current Large Language Model (LLM)-based agents are prone to hallucinations and making mistakes, especially when dealing with complex tasks such as formal verification. However, with the right enhancements and improvements, we believe that Saarthi can be a significant step towards achieving domain-specific general intelligence for formal verification. Especially for problems that require Short Term, Short Context (STSC) capabilities, such as formal verification, Saarthi can be a powerful tool to assist verification engineers in their work. In this paper, we present two key enhancements to the Saarthi framework: (1) a structured rulebook and specification grammar to improve the accuracy and controllability of SystemVerilog Assertion (SVA) generation, and (2) integration of advanced Retrieval Augmented Generation (RAG) techniques, such as GraphRAG, to provide agents with access to technical knowledge and best practices for iterative refinement and improvement of outputs. We also benchmark these enhancements for the overall Saarthi framework using challenging test cases from NVIDIA's CVDP benchmark targeting formal verification. Our benchmark results stand out with a 70% improvement in the accuracy of generated assertions, and a 50% reduction in the number of iterations required to achieve coverage closure.

Executive Summary

The article introduces Saarthi, an agentic AI framework for end-to-end formal verification, and proposes enhancements to achieve domain-specific general intelligence. The framework is improved with a structured rulebook and specification grammar, as well as integration of Retrieval Augmented Generation techniques. Benchmark results show a 70% improvement in assertion accuracy and a 50% reduction in iterations required for coverage closure. The enhancements demonstrate Saarthi's potential as a tool for verification engineers, particularly for Short Term, Short Context tasks like formal verification.

Key Points

  • ▸ Saarthi is an AI framework for formal verification using multi-agent collaboration
  • ▸ The framework is enhanced with a structured rulebook and specification grammar
  • ▸ Retrieval Augmented Generation techniques are integrated for iterative refinement and improvement

Merits

Improved Accuracy

The enhancements result in a 70% improvement in the accuracy of generated assertions, making Saarthi a more reliable tool for formal verification.

Demerits

Limited Domain-Specificity

While Saarthi shows promise for domain-specific general intelligence in formal verification, its applicability to other domains is unclear and may require significant modifications.

Expert Commentary

The article presents a significant step forward in the development of domain-specific general intelligence for formal verification. The integration of Retrieval Augmented Generation techniques and the structured rulebook are notable enhancements that address some of the limitations of current Large Language Model-based agents. However, the article also highlights the ongoing challenges in achieving true Artificial General Intelligence, and the need for continued research and development in this area. The benchmark results demonstrate the potential of Saarthi as a tool for verification engineers, and its potential to improve the efficiency and accuracy of formal verification processes.

Recommendations

  • ✓ Further research is needed to explore the applicability of Saarthi to other domains and to continue improving its accuracy and reliability
  • ✓ The development of standards and regulations for the use of AI in formal verification and other critical applications should be a priority for policymakers and industry leaders

Sources

Related Articles