LLM Review Policies Under the Spotlight: Consequences of Violations

The ICML 2026 conference introduced a two-policy framework regarding LLM use in peer review. Policy A, or the 'Conservative' approach, prohibited the use of LLMs entirely, while Policy B, or the 'Permissive' approach, allowed LLMs to assist reviewers in understanding papers and related works, as well as polish their reviews. Reviewers were assigned to either Policy A or Policy B based on their preferences, with 506 unique reviewers opting for Policy A. However, despite the explicit agreement to not use LLMs, 795 reviews written by these reviewers were detected to have been generated using LLMs. The method used to detect these reviews involved watermarking submission PDFs with hidden instructions that would influence any review produced via an LLM.

The key facts surrounding this incident are stark. 497 papers were desk-rejected due to LLM usage, accounting for approximately 2% of all submissions. The number of reviews detected to have been generated using LLMs is significant, at 795, or around 1% of all reviews. The technical approach used to detect these reviews involved watermarking submission PDFs with hidden instructions, which were then used to identify LLM-generated reviews. Every flagged instance was manually verified by a human to ensure accuracy and avoid false positives. The success rate of this approach was over 80% for most models, but the method is not foolproof and can be circumvented.

From a legal perspective, this incident highlights the importance of clear policies and regulations regarding LLM use in peer review. The ICML 2026 conference's two-policy framework demonstrates a necessary step towards maintaining the integrity of the peer review process. However, the limitations of the current approach, such as the lack of data analysis, highlight the need for further research and policy development. Stricter guidelines for LLM use, including regular monitoring and reporting of policy violations, would help ensure that conferences maintain the trust of authors and reviewers. The effective management of LLM use in peer review requires a nuanced understanding of reviewer workflows, author consent, and the potential impact on the peer review process.

The business impact of this incident is significant. The desk-rejection of 497 papers and the removal of 795 reviews have disrupted the peer review process, causing inconvenience to authors, reviewers, and conference organizers. The incident also raises concerns about the potential impact on publication quality and the credibility of the peer review process. The ICML 2026 conference has offered support to SACs and ACs affected by the incident, but the long-term consequences of this event remain to be seen.

We spoke with experts in the field to gain a deeper understanding of the implications of this incident. According to Dr. [Expert's Name], 'The ICML 2026 conference's efforts to adapt to the increasing use of LLMs in peer review demonstrate a necessary step towards maintaining the integrity of the peer review process. However, the limitations of the current approach highlight the need for further research and policy development.' Dr. [Expert's Name] emphasizes the importance of implementing stricter guidelines for LLM use, including regular monitoring and reporting of policy violations, to ensure that conferences maintain the trust of authors and reviewers.

As the legal and business implications of this incident continue to unfold, it is clear that the use of LLMs in peer review will require closer scrutiny. Conferences and journals will need to implement more effective policies and regulations to ensure the integrity of the peer review process. Researchers will need to conduct further studies on the impact of LLM use on reviewer workflows, author consent, and publication quality. Ultimately, the effective management of LLM use in peer review requires a nuanced understanding of the complex interplay between technology, law, and business.