Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind
arXiv:2602.13832v1 Announce Type: new Abstract: Large Language Models (LLMs) have developed rapidly and are widely applied to both general-purpose and professional tasks to assist human users. However, they still struggle to comprehend and respond to the true user needs when intentions and instructions are imprecisely conveyed, leading to a divergence between subjective user believes and true environment states. Resolving this epistemic divergence requires Theory of Mind (ToM), yet existing ToM evaluations for LLMs primarily focus on isolated belief inference, overlooking its functional utility in real-world interaction. To this end, we formalize ToM for LLMs as a mechanism for epistemic divergence detection and resolution, and propose a benchmark, \benchname, to assess how models reconcile user beliefs and profiles in practice. Results across 11 leading models reveal a significant limitation to identify underlying cognitive gaps that impede task success. To bridge this gap, we furthe
arXiv:2602.13832v1 Announce Type: new Abstract: Large Language Models (LLMs) have developed rapidly and are widely applied to both general-purpose and professional tasks to assist human users. However, they still struggle to comprehend and respond to the true user needs when intentions and instructions are imprecisely conveyed, leading to a divergence between subjective user believes and true environment states. Resolving this epistemic divergence requires Theory of Mind (ToM), yet existing ToM evaluations for LLMs primarily focus on isolated belief inference, overlooking its functional utility in real-world interaction. To this end, we formalize ToM for LLMs as a mechanism for epistemic divergence detection and resolution, and propose a benchmark, \benchname, to assess how models reconcile user beliefs and profiles in practice. Results across 11 leading models reveal a significant limitation to identify underlying cognitive gaps that impede task success. To bridge this gap, we further curate a trajectory-based ToM dataset linking belief tracking with task-related state inference. The model trained on this data via reinforcement learning shows consistent improvement in reasoning about user mental states, leading to enhanced downstream performance. Our work highlights the practical value of ToM as an essential interaction-level mechanism rather than as a standalone reasoning skill.
Executive Summary
The article 'Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind' explores the challenges Large Language Models (LLMs) face in comprehending and responding to user needs when instructions are imprecisely conveyed. The study introduces the concept of epistemic divergence, where there is a gap between subjective user beliefs and true environmental states. The authors propose a benchmark to assess how models reconcile user beliefs and profiles, revealing significant limitations in identifying cognitive gaps that impede task success. To address this, they curate a trajectory-based Theory of Mind (ToM) dataset and train models using reinforcement learning, showing improved reasoning about user mental states and enhanced downstream performance. The work underscores the practical value of ToM as an essential interaction-level mechanism.
Key Points
- ▸ LLMs struggle with epistemic divergence, leading to misalignment between user beliefs and true environmental states.
- ▸ Existing ToM evaluations focus on isolated belief inference, overlooking its functional utility in real-world interactions.
- ▸ The authors propose a benchmark to assess how models reconcile user beliefs and profiles, revealing significant limitations.
- ▸ A trajectory-based ToM dataset and reinforcement learning improve models' reasoning about user mental states and downstream performance.
- ▸ ToM is highlighted as an essential interaction-level mechanism rather than a standalone reasoning skill.
Merits
Innovative Approach
The article introduces a novel approach to evaluating and bridging epistemic divergence using Theory of Mind, which is a significant advancement in the field of human-AI interaction.
Comprehensive Benchmark
The proposed benchmark provides a robust framework for assessing how models reconcile user beliefs and profiles, offering valuable insights into current limitations.
Practical Applications
The study demonstrates the practical value of ToM in improving the performance of LLMs in real-world interactions, making it highly relevant for both academic and industry applications.
Demerits
Limited Scope
The study focuses primarily on epistemic divergence and ToM, which may not fully capture the broader range of challenges in human-AI interaction.
Data Limitations
The trajectory-based ToM dataset, while innovative, may have limitations in terms of generalizability and scalability, which could affect the broader applicability of the findings.
Model Training Complexity
The use of reinforcement learning for training models adds complexity and may require significant computational resources, potentially limiting its immediate adoption.
Expert Commentary
The article 'Beyond Words: Evaluating and Bridging Epistemic Divergence in User-Agent Interaction via Theory of Mind' presents a rigorous and well-reasoned exploration of the challenges faced by Large Language Models (LLMs) in comprehending user intentions. The introduction of the concept of epistemic divergence is particularly insightful, as it highlights a critical gap in current AI systems' ability to align with user beliefs and environmental states. The proposed benchmark for assessing how models reconcile user beliefs and profiles is a significant contribution to the field, providing a robust framework for evaluating and improving AI performance. The use of a trajectory-based ToM dataset and reinforcement learning to enhance models' reasoning about user mental states demonstrates a practical and innovative approach to addressing this challenge. The study's findings have broad implications for both academic research and industry applications, emphasizing the importance of ToM as an essential interaction-level mechanism. However, the study's limitations, such as the potential complexity of model training and the generalizability of the dataset, should be considered in future research. Overall, the article offers valuable insights and sets a strong foundation for further exploration in the field of human-AI interaction.
Recommendations
- ✓ Future research should explore the generalizability and scalability of the trajectory-based ToM dataset to ensure broader applicability.
- ✓ Developers should consider integrating the proposed benchmark into their evaluation frameworks to improve the performance of AI models in understanding user intentions.