InfoGatherer: Principled Information Seeking via Evidence Retrieval and Strategic Questioning
arXiv:2603.05909v1 Announce Type: new Abstract: LLMs are increasingly deployed in high-stakes domains such as medical triage and legal assistance, often as document-grounded QA systems in which a user provides a description, relevant sources are retrieved, and an LLM generates a prediction. In practice, initial user queries are often underspecified, and a single retrieval pass is insufficient for reliable decision-making, leading to incorrect and overly confident answers. While follow-up questioning can elicit missing information, existing methods typically depend on implicit, unstructured confidence signals from the LLM, making it difficult to determine what remains unknown, what information matters most, and when to stop asking questions. We propose InfoGatherer, a framework that gathers missing information from two complementary sources: retrieved domain documents and targeted follow-up questions to the user. InfoGatherer models uncertainty using Dempster-Shafer belief assignments
arXiv:2603.05909v1 Announce Type: new Abstract: LLMs are increasingly deployed in high-stakes domains such as medical triage and legal assistance, often as document-grounded QA systems in which a user provides a description, relevant sources are retrieved, and an LLM generates a prediction. In practice, initial user queries are often underspecified, and a single retrieval pass is insufficient for reliable decision-making, leading to incorrect and overly confident answers. While follow-up questioning can elicit missing information, existing methods typically depend on implicit, unstructured confidence signals from the LLM, making it difficult to determine what remains unknown, what information matters most, and when to stop asking questions. We propose InfoGatherer, a framework that gathers missing information from two complementary sources: retrieved domain documents and targeted follow-up questions to the user. InfoGatherer models uncertainty using Dempster-Shafer belief assignments over a structured evidential network, enabling principled fusion of incomplete and potentially contradictory evidence from both sources without prematurely collapsing to a definitive answer. Across legal and medical tasks, InfoGatherer outperforms strong baselines while requiring fewer turns. By grounding uncertainty in formal evidential theory rather than heuristic LLM signals, InfoGatherer moves towards trustworthy, interpretable decision support in domains where reliability is critical.
Executive Summary
The article introduces InfoGatherer, a novel framework designed to improve information gathering in high-stakes domains by addressing the shortcomings of current LLM-based QA systems. Recognizing that user queries are often underspecified and single retrieval passes are insufficient, InfoGatherer leverages both retrieved documents and targeted follow-up questions to gather missing information. The framework introduces a principled approach to uncertainty modeling via Dempster-Shafer belief assignments over a structured evidential network, enabling a more reliable fusion of incomplete and contradictory evidence without prematurely concluding. The authors demonstrate that InfoGatherer outperforms existing baselines in legal and medical tasks while minimizing the number of user interactions, thereby enhancing trustworthiness and interpretability in critical decision-making contexts.
Key Points
- ▸ InfoGatherer addresses underspecified queries and insufficient retrieval passes by integrating document retrieval and targeted questioning.
- ▸ The framework employs Dempster-Shafer belief assignments for principled uncertainty modeling over a structured evidential network.
- ▸ Empirical results show improved performance and fewer interaction turns in legal and medical applications compared to strong baselines.
Merits
Strength in Formal Uncertainty Modeling
InfoGatherer’s use of Dempster-Shafer belief assignments introduces a rigorous, interpretable framework for uncertainty quantification, moving beyond heuristic LLM signals to a formal evidential theory-based approach.
Empirical Validation Across Domains
Cross-domain validation in both legal and medical contexts substantiates the effectiveness of InfoGatherer in reducing incorrect answers and improving decision support reliability.
Demerits
Complexity of Implementation
Integrating structured evidential networks with Dempster-Shafer assignments may pose challenges in scalability and computational overhead, potentially limiting real-time applicability.
Expert Commentary
InfoGatherer represents a significant conceptual advancement in the domain of information retrieval for AI-assisted decision support. By shifting the focus from implicit LLM confidence indicators to a formalized evidential structure, the authors align their work with foundational principles of epistemic reasoning. This shift is particularly valuable in high-stakes environments where the credibility of conclusions is paramount. The integration of Dempster-Shafer belief assignments—while theoretically robust—introduces a layer of complexity that warrants further exploration in practical deployment. Nonetheless, the empirical validation across legal and medical tasks is compelling and positions InfoGatherer as a benchmark for future work in this area. The ability to minimize interaction turns without compromising accuracy is a pragmatic benefit that aligns with user expectations and operational constraints. Overall, this work bridges a critical gap between theoretical epistemology and applied AI systems, offering a scalable model for more transparent and reliable decision-making.
Recommendations
- ✓ 1. Conduct comparative studies on scalability of InfoGatherer’s evidential network architecture across larger datasets and real-time applications.
- ✓ 2. Explore integration of InfoGatherer with existing LLM platforms as a modular uncertainty-aware module to enhance existing QA workflows.