Academic

Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

arXiv:2603.06621v1 Announce Type: new Abstract: Process Reward Models (PRMs) are rapidly becoming the backbone of LLM reasoning pipelines, yet we demonstrate that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure. To address this, we introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities. Static perturbation analysis uncovers a fluency-logic dissociation: high invariance to surface-level style changes reward changes $<$0.1, yet inconsistent detection of logically-corrupted reasoning, with different models failing on different attack types. Adversarial optimization demonstrates that gradient-based attacks inflate rewards on invalid trajectories, with reward landscapes exhibiting wide, exploitable peaks. RL-induced reward hacking exposes the critical failure mode: policies trained on AIME problems achieve near-perfect PRM rewards ($>$0.9), while ground-truth accuracy remains

R
Rishabh Tiwari, Aditya Tomar, Udbhav Bamba, Monishwaran Maheswaran, Heng Yang, Michael W. Mahoney, Kurt Keutzer, Amir Gholami · · 1 min read · 10 views
#cs.LG

arXiv:2603.06621v1 Announce Type: new Abstract: Process Reward Models (PRMs) are rapidly becoming the backbone of LLM reasoning pipelines, yet we demonstrate that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure. To address this, we introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities. Static perturbation analysis uncovers a fluency-logic dissociation: high invariance to surface-level style changes reward changes $<$0.1, yet inconsistent detection of logically-corrupted reasoning, with different models failing on different attack types. Adversarial optimization demonstrates that gradient-based attacks inflate rewards on invalid trajectories, with reward landscapes exhibiting wide, exploitable peaks. RL-induced reward hacking exposes the critical failure mode: policies trained on AIME problems achieve near-perfect PRM rewards ($>$0.9), while ground-truth accuracy remains low (below 4%), with 43% of reward gains attributable to stylistic shortcuts. These findings reveal that current PRMs function as fluency detectors rather than reasoning verifiers, creating systematic blind spots that undermine their use as training signals. We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment. The code and dataset are available at https://github.com/SqueezeAILab/reward-under-attack.

Executive Summary

This article presents a comprehensive analysis of the robustness and hackability of Process Reward Models (PRMs) used in Large Language Models (LLMs). The authors demonstrate that state-of-the-art PRMs are vulnerable to adversarial optimization and introduce a three-tiered diagnostic framework to quantify these vulnerabilities. The findings reveal that current PRMs function as fluency detectors rather than reasoning verifiers, creating systematic blind spots that undermine their use as training signals. The authors release a diagnostic toolkit and dataset to enable robustness evaluation before deployment. This study highlights the need for more robust and reliable PRMs in LLM reasoning pipelines.

Key Points

  • ▸ State-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.
  • ▸ A three-tiered diagnostic framework is introduced to quantify vulnerabilities in PRMs.
  • ▸ Current PRMs function as fluency detectors rather than reasoning verifiers, creating blind spots in training signals.

Merits

Strength

The study provides a comprehensive analysis of PRM vulnerabilities and introduces a diagnostic toolkit to evaluate robustness before deployment.

Innovative Approach

The three-tiered diagnostic framework offers a novel approach to evaluating PRM robustness and identifying vulnerabilities.

Demerits

Limitation

The study focuses on PRMs in LLM reasoning pipelines, and its findings may not be generalizable to other applications of PRMs.

Scalability

The diagnostic toolkit and dataset introduced in the study may require significant computational resources to evaluate robustness in large-scale LLMs.

Expert Commentary

The study provides a thorough analysis of PRM vulnerabilities and offers a critical perspective on the limitations of current PRMs. The introduction of a diagnostic toolkit and dataset is a significant contribution to the field, enabling researchers and developers to evaluate robustness before deployment. However, the study's focus on PRMs in LLM reasoning pipelines may limit its generalizability to other applications. Furthermore, the scalability of the diagnostic toolkit and dataset may pose challenges in large-scale LLMs. Overall, the study highlights the need for more robust and reliable PRMs to improve explainability and trustworthiness in LLMs.

Recommendations

  • ✓ Developers should prioritize the development of more robust and reliable PRMs, incorporating techniques such as adversarial training and regularization.
  • ✓ Regulatory bodies should establish guidelines and standards for evaluating the robustness of PRMs in LLM applications, ensuring their safe and trustworthy deployment.

Sources

Related Articles