The Detection--Extraction Gap: Models Know the Answer Before They Can Say It
arXiv:2604.06613v1 Announce Type: new Abstract: Modern reasoning models continue generating long after the answer is already determined. Across five model configurations, two families, and three benchmarks, we find that \textbf{52--88\% of chain-of-thought tokens are produced after the answer is recoverable} from a partial prefix. This post-commitment generation reveals a structural phenomenon: the \textbf{detection--extraction gap}. Free continuations from early prefixes recover the correct answer even at 10\% of the trace, while forced extraction fails on 42\% of these cases. The answer is recoverable from the model state, yet prompt-conditioned decoding fails to extract it. We formalize this mismatch via a total-variation bound between free and forced continuation distributions, yielding quantitative estimates of suffix-induced shift. Exploiting this asymmetry, we propose Black-box Adaptive Early Exit (\BAEE{}), which uses free continuations for both detection and extraction, trunc
arXiv:2604.06613v1 Announce Type: new Abstract: Modern reasoning models continue generating long after the answer is already determined. Across five model configurations, two families, and three benchmarks, we find that \textbf{52--88\% of chain-of-thought tokens are produced after the answer is recoverable} from a partial prefix. This post-commitment generation reveals a structural phenomenon: the \textbf{detection--extraction gap}. Free continuations from early prefixes recover the correct answer even at 10\% of the trace, while forced extraction fails on 42\% of these cases. The answer is recoverable from the model state, yet prompt-conditioned decoding fails to extract it. We formalize this mismatch via a total-variation bound between free and forced continuation distributions, yielding quantitative estimates of suffix-induced shift. Exploiting this asymmetry, we propose Black-box Adaptive Early Exit (\BAEE{}), which uses free continuations for both detection and extraction, truncating \textbf{70--78\% of serial generation} while \textbf{improving accuracy by 1--5\,pp} across all models. For thinking-mode models, early exit prevents post-commitment overwriting, yielding gains of up to 5.8\,pp; a cost-optimized variant achieves 68--73\% reduction at a median of 9 API calls. Code is available at https://github.com/EdWangLoDaSc/know2say.
Executive Summary
This article introduces the 'detection--extraction gap' in large language models (LLMs), observing that models often determine the correct answer much earlier in their chain-of-thought (CoT) generation than when they finally articulate it. The authors find that 52-88% of CoT tokens are redundant, produced after the answer is logically recoverable. This gap highlights a fundamental inefficiency: the model's internal state 'knows' the answer, but standard prompt-conditioned decoding struggles to 'say' it early. The proposed Black-box Adaptive Early Exit (BAEE) mechanism leverages this by using free continuations for both early detection and extraction, significantly truncating generation (70-78% reduction) while simultaneously improving accuracy (1-5 pp), particularly by preventing post-commitment overwriting.
Key Points
- ▸ LLMs exhibit a 'detection--extraction gap,' where answers are recoverable from early CoT prefixes (often 10% of trace) but generation continues unnecessarily.
- ▸ 52-88% of CoT tokens are generated *after* the answer is already determined, indicating significant redundancy and inefficiency.
- ▸ Standard forced extraction methods fail in 42% of cases where the answer is recoverable from early prefixes, contrasting with successful free continuations.
- ▸ The proposed Black-box Adaptive Early Exit (BAEE) method truncates 70-78% of serial generation and improves accuracy by 1-5 percentage points.
- ▸ BAEE is particularly effective for 'thinking-mode' models, preventing post-commitment overwriting and achieving substantial cost reduction with minimal API calls.
Merits
Novel Conceptualization
The introduction of the 'detection--extraction gap' provides a powerful and intuitive framework for understanding a critical inefficiency in LLM reasoning, moving beyond mere observation of verbosity to a structural analysis.
Empirical Rigor
The findings are robust, demonstrated across multiple model configurations, two model families, and three distinct benchmarks, lending significant credibility to the generality of the phenomenon.
Practical Solution with Dual Benefits
BAEE offers a concrete, implementable solution that simultaneously improves efficiency (drastic token reduction) and accuracy, a rare and highly desirable combination in LLM optimization.
Formalization of Mismatch
The use of a total-variation bound to formalize the mismatch between free and forced continuation distributions provides a rigorous quantitative basis for the observed suffix-induced shift.
Demerits
Black-box Nature of BAEE
While BAEE is designed for black-box models, the underlying mechanism for 'detection' and 'extraction' via free continuations might still lack full transparency, making it challenging to debug or predict behavior in edge cases.
Generalizability to Complex Tasks
While tested on three benchmarks, the extent to which these findings and BAEE's effectiveness generalize to highly complex, multi-step, or open-ended reasoning tasks remains to be fully explored. The 'answer' in such contexts might be more fluid.
Computational Overhead of Multiple API Calls
The cost-optimized BAEE variant mentions a median of 9 API calls. While significantly reducing total tokens, the *number* of API calls could still be a consideration for latency-sensitive applications or specific API pricing models, despite overall cost reduction.
Expert Commentary
This article presents a profoundly insightful and empirically grounded contribution to our understanding of large language model (LLM) behavior. The 'detection--extraction gap' is not merely an observation of verbosity but a structural diagnosis of how LLMs process and articulate information. Its formalization via total-variation bounds elevates the discussion beyond anecdotal evidence, providing a robust analytical framework. The dual benefit of BAEE—simultaneously improving efficiency and accuracy—is particularly compelling, as these are often conflicting objectives in LLM optimization. This work challenges the implicit assumption that a longer chain-of-thought necessarily equates to better reasoning or more robust answers, suggesting instead that it can be a source of noise and error through 'post-commitment overwriting.' This paper will undoubtedly spur further research into dynamic decoding, internal model state analysis, and more sophisticated prompting strategies that capitalize on these early 'knowledge' states, moving us closer to truly intelligent and efficient AI systems. The implications for practical deployment and cost reduction are immediate and substantial.
Recommendations
- ✓ Further investigate the 'detection--extraction gap' across a wider range of complex reasoning tasks, including those requiring planning, multi-modal input, or long-context understanding, to assess generalizability.
- ✓ Explore methods to make the 'detection' and 'extraction' mechanisms within BAEE more transparent and interpretable, perhaps by correlating early 'knowledge' with specific internal model activations or representations.
- ✓ Develop standardized metrics and benchmarks that explicitly measure the 'detection--extraction gap' and the efficiency of early exit strategies, fostering comparative research.
- ✓ Research the integration of BAEE or similar dynamic truncation methods directly into LLM architectures or API designs, rather than as a post-hoc application, for even greater efficiency gains and seamless deployment.
Sources
Original: arXiv - cs.CL