LJ-Bench: Ontology-Based Benchmark for U.S. Crime
arXiv:2603.20572v1 Announce Type: new Abstract: The potential of Large Language Models (LLMs) to provide harmful information remains a significant concern due to the vast breadth of illegal queries they may encounter. Unfortunately, existing benchmarks only focus on a handful types of illegal activities, and are not grounded in legal works. In this work, we introduce an ontology of crime-related concepts grounded in the legal frameworks of Model Panel Code, which serves as an influential reference for criminal law and has been adopted by many U.S. states, and instantiated using Californian Law. This structured knowledge forms the foundation for LJ-Bench, the first comprehensive benchmark designed to evaluate LLM robustness against a wide range of illegal activities. Spanning 76 distinct crime types organized taxonomically, LJ-Bench enables systematic assessment of diverse attacks, revealing valuable insights into LLM vulnerabilities across various crime categories: LLMs exhibit height
arXiv:2603.20572v1 Announce Type: new Abstract: The potential of Large Language Models (LLMs) to provide harmful information remains a significant concern due to the vast breadth of illegal queries they may encounter. Unfortunately, existing benchmarks only focus on a handful types of illegal activities, and are not grounded in legal works. In this work, we introduce an ontology of crime-related concepts grounded in the legal frameworks of Model Panel Code, which serves as an influential reference for criminal law and has been adopted by many U.S. states, and instantiated using Californian Law. This structured knowledge forms the foundation for LJ-Bench, the first comprehensive benchmark designed to evaluate LLM robustness against a wide range of illegal activities. Spanning 76 distinct crime types organized taxonomically, LJ-Bench enables systematic assessment of diverse attacks, revealing valuable insights into LLM vulnerabilities across various crime categories: LLMs exhibit heightened susceptibility to attacks targeting societal harm rather than those directly impacting individuals. Our benchmark aims to facilitate the development of more robust and trustworthy LLMs. The LJ-Bench benchmark and LJ-Ontology, along with experiments implementation for reproducibility are publicly available at https://github.com/AndreaTseng/LJ-Bench.
Executive Summary
This article introduces LJ-Bench, a comprehensive ontology-based benchmark for evaluating the robustness of Large Language Models (LLMs) against a wide range of illegal activities. Grounded in the legal frameworks of the Model Penal Code and Californian Law, LJ-Bench enables systematic assessment of diverse attacks on LLMs, revealing valuable insights into their vulnerabilities. The benchmark consists of 76 distinct crime types organized taxonomically and aims to facilitate the development of more robust and trustworthy LLMs. The LJ-Bench benchmark and LJ-Ontology, along with experiments implementation for reproducibility, are publicly available on GitHub. This research has significant implications for the development of AI systems that can provide accurate and trustworthy information without perpetuating harm.
Key Points
- ▸ LJ-Bench is the first comprehensive benchmark for evaluating LLM robustness against a wide range of illegal activities.
- ▸ The benchmark is grounded in the legal frameworks of the Model Penal Code and Californian Law.
- ▸ LJ-Bench reveals valuable insights into LLM vulnerabilities across various crime categories.
Merits
Strength
LJ-Bench is the first comprehensive benchmark for evaluating LLM robustness against a wide range of illegal activities, providing a systematic and structured approach to assessing LLM vulnerabilities.
Grounding in Legal Frameworks
The benchmark is grounded in the legal frameworks of the Model Penal Code and Californian Law, ensuring that it is informed by a robust understanding of the law and its applications.
Publicly Available
The LJ-Bench benchmark and LJ-Ontology, along with experiments implementation for reproducibility, are publicly available on GitHub, enabling other researchers to build upon and expand this work.
Demerits
Limitation
The benchmark may be limited by its reliance on a specific legal framework, which may not be representative of all legal jurisdictions or contexts.
Scope
The benchmark may not be comprehensive in its scope, potentially omitting certain types of crimes or legal nuances that are relevant to LLM development and deployment.
Expert Commentary
The LJ-Bench benchmark represents a significant contribution to the field of AI research, providing a comprehensive and structured approach to evaluating the robustness of LLMs against a wide range of illegal activities. The benchmark's grounding in legal frameworks and its focus on evaluating LLM vulnerabilities across various crime categories highlights the need for continued research into bias and fairness in AI systems. Furthermore, the benchmark's implications for the regulation of AI systems raises important questions about the role of law in shaping the development and deployment of AI systems. As such, the LJ-Bench benchmark has significant practical and policy implications for the development and deployment of AI systems.
Recommendations
- ✓ Developers and researchers should use the LJ-Bench benchmark to evaluate the robustness of LLMs in real-world applications.
- ✓ Policymakers and regulators should consider the implications of the LJ-Bench benchmark for the regulation of AI systems and their potential impact on society.
Sources
Original: arXiv - cs.LG
