Academic

CTRL-RAG: Contrastive Likelihood Reward Based Reinforcement Learning for Context-Faithful RAG Models

arXiv:2603.04406v1 Announce Type: new Abstract: With the growing use of Retrieval-Augmented Generation (RAG), training large language models (LLMs) for context-sensitive reasoning and faithfulness is increasingly important. Existing RAG-oriented reinforcement learning (RL) methods rely on external rewards that often fail to evaluate document faithfulness, and may misjudge similar answers in open-domain settings. In addition, there is no RAG-based selfreward mechanism. Moreover, although such a mechanism could in principle estimate answer confidence given documents, the absence of objective feedback in a self-judgment can cause hallucination accumulation and eventual model collapse. To tackle these issues, we propose a novel "internal-external" hybrid reward framework centered on a Contrastive Likelihood Reward (CLR). CLR directly optimizes the log-likelihood gap between responses conditioned on prompts with and without supporting evidence. This encourages the model to extract relevant

Z
Zhehao Tan, Yihan Jiao, Dan Yang, Junjie Wang, Duolin Sun, Jie Feng, Xidong Wang, Lei Liu, Yue Shen, Jian Wang, Jinjie Gu · · 1 min read · 3 views
#cs.CL #cs.AI

arXiv:2603.04406v1 Announce Type: new Abstract: With the growing use of Retrieval-Augmented Generation (RAG), training large language models (LLMs) for context-sensitive reasoning and faithfulness is increasingly important. Existing RAG-oriented reinforcement learning (RL) methods rely on external rewards that often fail to evaluate document faithfulness, and may misjudge similar answers in open-domain settings. In addition, there is no RAG-based selfreward mechanism. Moreover, although such a mechanism could in principle estimate answer confidence given documents, the absence of objective feedback in a self-judgment can cause hallucination accumulation and eventual model collapse. To tackle these issues, we propose a novel "internal-external" hybrid reward framework centered on a Contrastive Likelihood Reward (CLR). CLR directly optimizes the log-likelihood gap between responses conditioned on prompts with and without supporting evidence. This encourages the model to extract relevant evidence and increases its confidence when grounded in a specific context. Experiments show that our method (used alone or combined with external correctness rewards) achieves strong performance on singlehop, multi-hop, vertical-domain, and faithfulness benchmarks. Our training code and models are coming soon.

Executive Summary

This article proposes a novel reinforcement learning framework, Contrastive Likelihood Reward (CLR), to improve the performance of Retrieval-Augmented Generation (RAG) models in context-sensitive reasoning and faithfulness. The CLR framework combines internal and external rewards to optimize the log-likelihood gap between responses conditioned on prompts with and without supporting evidence. Experiments demonstrate the effectiveness of CLR in single-hop, multi-hop, vertical-domain, and faithfulness benchmarks. The proposed method can be used alone or in combination with external correctness rewards, providing a robust approach to training RAG models. The authors' code and models will be released soon, enabling further research and development.

Key Points

  • ▸ CLR combines internal and external rewards for RAG model training
  • ▸ CLR optimizes log-likelihood gap between responses with and without supporting evidence
  • ▸ Experiments demonstrate CLR's effectiveness in various benchmarks

Merits

Strength in Addressing Faithfulness Issues

CLR directly evaluates document faithfulness, addressing a significant limitation of existing RAG-oriented RL methods.

Robustness and Flexibility

The proposed method can be used alone or in combination with external correctness rewards, providing a robust and flexible approach to RAG model training.

Demerits

Potential Overreliance on Internal Rewards

The CLR framework relies heavily on internal rewards, which may lead to model overreliance and decreased performance in scenarios where external rewards are necessary.

Limited Exploration of Model Collapse

The article does not thoroughly explore the potential for model collapse due to the absence of objective feedback in self-judgment, which is a critical consideration for large-scale RAG model training.

Expert Commentary

The proposed CLR framework is a significant contribution to the field of RAG model training, addressing critical challenges in context-sensitive reasoning and faithfulness. The combination of internal and external rewards provides a robust and flexible approach to model training. However, further research is necessary to fully explore the potential for model collapse and to develop more robust and reliable AI model evaluation metrics. Nonetheless, the CLR framework has the potential to significantly impact the development of more accurate and reliable AI models, with important implications for both practical applications and policy considerations.

Recommendations

  • ✓ Further research should be conducted to explore the potential for model collapse and to develop more robust and reliable AI model evaluation metrics.
  • ✓ The CLR framework should be applied to a wide range of RAG-based applications to demonstrate its effectiveness and versatility.

Sources

Related Articles