Academic

Know When You're Wrong: Aligning Confidence with Correctness for LLM Error Detection

arXiv:2603.06604v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning met

X
Xie Xiaohu, Liu Xiaohu, Yao Benjamin · · 1 min read · 8 views
#cs.LG #cs.CL

arXiv:2603.06604v1 Announce Type: new Abstract: As large language models (LLMs) are increasingly deployed in critical decision-making systems, the lack of reliable methods to measure their uncertainty presents a fundamental trustworthiness risk. We introduce a normalized confidence score based on output anchor token probabilities: classification labels for structured tasks and self-evaluation responses (Yes/No) for open-ended generation. This enables direct detection of errors and hallucinations with minimal overhead and without external validation. We make three key contributions. First, we propose a normalized confidence score and self-evaluation framework that exposes reliable confidence estimates for error detection across seven diverse benchmark tasks and five LLMs of varying architectures and sizes. Second, our theoretical analysis reveals that supervised fine-tuning (SFT) yields well-calibrated confidence through maximum-likelihood estimation, whereas reinforcement learning methods (PPO, GRPO) and DPO induce overconfidence via reward exploitation. Third, we propose post-RL SFT with self-distillation to restore confidence reliability in RL-trained models. Empirical results demonstrated that SFT improved average confidence-correctness AUROC from 0.806 to 0.879 and reduced calibration error from 0.163 to 0.034 on Qwen3-4B, while GRPO and DPO degraded confidence reliability. We demonstrated practical value through adaptive retrieval-augmented generation (RAG) that selectively retrieves context when the model lacks confidence, using only 58\% of retrieval operations to recover 95\% of the maximum achievable accuracy gain on TriviaQA

Executive Summary

This article proposes a novel approach to detect errors in large language models (LLMs) by aligning confidence with correctness. The authors introduce a normalized confidence score based on output anchor token probabilities, enabling direct detection of errors and hallucinations. The study evaluates the effectiveness of this approach across seven benchmark tasks and five LLMs, demonstrating improved confidence-correctness alignment and reduced calibration error. The findings have significant implications for the development of reliable LLMs in critical decision-making systems.

Key Points

  • ▸ Introduction of a normalized confidence score for error detection in LLMs
  • ▸ Theoretical analysis of the impact of supervised fine-tuning and reinforcement learning on confidence reliability
  • ▸ Proposal of post-RL supervised fine-tuning with self-distillation to restore confidence reliability

Merits

Improved Confidence-Correctness Alignment

The proposed approach demonstrates improved confidence-correctness alignment, enabling more reliable error detection in LLMs

Reduced Calibration Error

The study shows reduced calibration error, indicating more accurate confidence estimates in LLMs

Demerits

Limited Evaluation Scope

The evaluation is limited to seven benchmark tasks and five LLMs, which may not be representative of all possible applications and models

Dependence on Supervised Fine-Tuning

The approach relies on supervised fine-tuning, which may not be feasible or effective in all scenarios

Expert Commentary

The article presents a significant contribution to the development of reliable LLMs by introducing a novel approach to detect errors and hallucinations. The study's findings on the impact of supervised fine-tuning and reinforcement learning on confidence reliability are particularly insightful. However, further research is needed to evaluate the approach's effectiveness in more diverse scenarios and to address potential limitations. The implications of this study are far-reaching, with potential applications in various fields, including healthcare, finance, and education.

Recommendations

  • ✓ Further evaluation of the proposed approach in more diverse scenarios and applications
  • ✓ Investigation of alternative methods to improve confidence reliability in LLMs, such as ensemble methods or uncertainty estimation techniques

Sources

Related Articles