Consistency-Guided Decoding with Proof-Driven Disambiguation for Three-Way Logical Question Answering
arXiv:2604.06196v1 Announce Type: new Abstract: Three-way logical question answering (QA) assigns $True/False/Unknown$ to a hypothesis $H$ given a premise set $S$. While modern large language models (LLMs) can be accurate on isolated examples, we identify two recurring failure modes in 3-way logic QA: (i) negation inconsistency, where answers to $H$ and $\neg H$ violate the deterministic label mapping, and (ii) epistemic $Unknown$, where the model predicts $Unknown$ due to uncertainty or instability even when $S$ entails one side. We present CGD-PD, a lightweight test-time layer that (a) queries a single 3-way classifier on both $H$ and a mechanically negated form of $H$, (b) projects the pair onto a negation-consistent decision when possible, and (c) invokes a proof-driven disambiguation step that uses targeted binary entailment probes to selectively resolve $Unknown$ outcomes, requiring only an average of 4-5 model calls. On the FOLIO benchmark's first-order-logic fields, CGD-PD yie
arXiv:2604.06196v1 Announce Type: new Abstract: Three-way logical question answering (QA) assigns $True/False/Unknown$ to a hypothesis $H$ given a premise set $S$. While modern large language models (LLMs) can be accurate on isolated examples, we identify two recurring failure modes in 3-way logic QA: (i) negation inconsistency, where answers to $H$ and $\neg H$ violate the deterministic label mapping, and (ii) epistemic $Unknown$, where the model predicts $Unknown$ due to uncertainty or instability even when $S$ entails one side. We present CGD-PD, a lightweight test-time layer that (a) queries a single 3-way classifier on both $H$ and a mechanically negated form of $H$, (b) projects the pair onto a negation-consistent decision when possible, and (c) invokes a proof-driven disambiguation step that uses targeted binary entailment probes to selectively resolve $Unknown$ outcomes, requiring only an average of 4-5 model calls. On the FOLIO benchmark's first-order-logic fields, CGD-PD yields consistent gains across frontier LLMs, with relative improvements in accuracy of up to 16% over the base model, while also reducing $Unknown$ predictions.
Executive Summary
The paper introduces CGD-PD, a novel, lightweight test-time method designed to enhance the accuracy and consistency of Large Language Models (LLMs) in three-way logical question answering (QA). This task requires assigning True, False, or Unknown to a hypothesis given a premise set. CGD-PD addresses two critical failure modes: negation inconsistency and epistemic Unknown predictions. By querying both a hypothesis and its negation, projecting to a consistent decision, and employing proof-driven disambiguation via targeted binary entailment probes, CGD-PD significantly improves performance on the FOLIO benchmark, achieving up to 16% relative accuracy gains and reducing 'Unknown' predictions with minimal additional computational cost.
Key Points
- ▸ Identifies and addresses two critical failure modes in LLMs for 3-way logical QA: negation inconsistency and epistemic Unknowns.
- ▸ Introduces CGD-PD, a test-time layer that combines negation-consistency checks with proof-driven disambiguation.
- ▸ CGD-PD operates by querying a single 3-way classifier on H and ¬H, projecting to consistent decisions, and using binary entailment probes to resolve 'Unknown' outcomes.
- ▸ Achieves substantial accuracy gains (up to 16% relative improvement) and reduces 'Unknown' predictions on the FOLIO benchmark, requiring only 4-5 average model calls.
- ▸ The method is lightweight and applicable to frontier LLMs, suggesting broad utility without extensive retraining.
Merits
Addresses Core LLM Fragilities
Directly tackles known weaknesses of LLMs in logical reasoning, specifically negation handling and over-prediction of 'Unknown' due to uncertainty, which are critical for reliable AI systems.
High Efficacy with Low Overhead
Achieves significant performance improvements (up to 16% relative accuracy) with a remarkably low computational cost (4-5 additional model calls), making it practical for real-world deployment.
Generalizability Across LLMs
Demonstrates consistent gains across 'frontier LLMs', suggesting that the method is not model-specific but rather a robust technique applicable to a range of advanced language models.
Improved Interpretability (Implicit)
The 'proof-driven disambiguation' step, by using targeted binary entailment probes, implicitly moves towards a more structured and potentially interpretable reasoning process, rather than relying solely on black-box LLM outputs.
Demerits
Dependence on Negation Generation
The mechanical negation of H (¬H) is crucial. Errors or ambiguities in this mechanical negation process could undermine the effectiveness of the consistency-guided decoding.
Scope of 'Proof-Driven Disambiguation'
While effective, the nature and limits of the 'targeted binary entailment probes' are not fully detailed in the abstract, leaving open questions about their complexity and generalizability beyond first-order logic.
Benchmark Specificity
Results are primarily showcased on the FOLIO benchmark. While a strong indicator, it would be valuable to see performance across a wider variety of logical reasoning datasets to confirm robustness.
Potential for Error Propagation
If the initial 3-way classifier has systemic biases or errors that are not adequately addressed by the consistency checks, these might still propagate, albeit reduced.
Expert Commentary
This paper presents a highly pertinent and elegant solution to two persistent challenges in LLM-based logical reasoning: semantic consistency in negation and the problematic ambiguity of 'Unknown' predictions. The ingenuity of CGD-PD lies in its lightweight, test-time application, avoiding the prohibitive costs of model retraining while yielding substantial performance gains. The focus on 'proof-driven disambiguation' marks a crucial step towards instilling a more structured, almost 'symbolic', layer of reasoning atop the inherently probabilistic nature of LLMs. From a jurisprudential perspective, the ability to resolve 'Unknown' states through targeted probes is particularly compelling; legal reasoning frequently grapples with conditions where an initial assessment is ambiguous, requiring further, targeted inquiry to establish 'True' or 'False'. The method's generalizability across frontier LLMs underscores its foundational value, suggesting it could become a standard post-processing technique. Future work should delve deeper into the formal guarantees of the 'proof-driven' component and its applicability to higher-order logic, beyond the first-order domain of FOLIO.
Recommendations
- ✓ Conduct further research into the formal properties and theoretical limits of the 'proof-driven disambiguation' component, especially concerning its ability to generalize to more complex logical structures and domains.
- ✓ Evaluate CGD-PD on a broader range of logical reasoning benchmarks, including those with different logical formalisms and potentially adversarial examples, to thoroughly assess its robustness and generalizability.
- ✓ Investigate methods to improve the robustness and accuracy of the 'mechanical negation' step, as its reliability is foundational to CGD-PD's effectiveness.
- ✓ Explore how the 'proof-driven' aspect could be made more transparent or explainable, potentially by explicitly showing the binary entailment probes and their outcomes to the user, enhancing trust and understanding.
Sources
Original: arXiv - cs.CL