Speak or Stay Silent: Context-Aware Turn-Taking in Multi-Party Dialogue
arXiv:2603.11409v1 Announce Type: new Abstract: Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capabi
arXiv:2603.11409v1 Announce Type: new Abstract: Existing voice AI assistants treat every detected pause as an invitation to speak. This works in dyadic dialogue, but in multi-party settings, where an AI assistant participates alongside multiple speakers, pauses are abundant and ambiguous. An assistant that speaks on every pause becomes disruptive rather than useful. In this work, we formulate context-aware turn-taking: at every detected pause, given the full conversation context, our method decides whether the assistant should speak or stay silent. We introduce a benchmark of over 120K labeled conversations spanning three multi-party corpora. Evaluating eight recent large language models, we find that they consistently fail at context-aware turn-taking under zero-shot prompting. We then propose a supervised fine-tuning approach with reasoning traces, improving balanced accuracy by up to 23 percentage points. Our findings suggest that context-aware turn-taking is not an emergent capability; it must be explicitly trained.
Executive Summary
This article presents a critical issue in voice AI assistants' ability to engage in multi-party dialogue. By treating every detected pause as an invitation to speak, these assistants can become disruptive rather than useful. The authors propose a context-aware turn-taking approach that decides whether the assistant should speak or stay silent based on the full conversation context. They introduce a benchmark of over 120K labeled conversations and evaluate eight recent large language models, finding that they fail at context-aware turn-taking under zero-shot prompting. A supervised fine-tuning approach with reasoning traces is proposed to improve balanced accuracy. The findings suggest that context-aware turn-taking is not an emergent capability and must be explicitly trained. This research has significant implications for the development of voice AI assistants and their integration into various applications.
Key Points
- ▸ Existing voice AI assistants treat every detected pause as an invitation to speak, which can be disruptive in multi-party settings.
- ▸ A context-aware turn-taking approach is proposed to decide whether the assistant should speak or stay silent based on the full conversation context.
- ▸ Recent large language models consistently fail at context-aware turn-taking under zero-shot prompting.
Merits
Strength of Methodology
The authors introduce a comprehensive benchmark of over 120K labeled conversations and evaluate eight recent large language models, providing a robust evaluation of the proposed approach.
Demerits
Limitation of Current State-of-the-Art Models
The findings suggest that current large language models lack the ability to perform context-aware turn-taking, which is a critical capability for voice AI assistants in multi-party dialogue settings.
Expert Commentary
The article presents a significant contribution to the field of natural language processing and speech recognition, highlighting the importance of context-aware turn-taking capabilities in voice AI assistants. The findings suggest that current large language models lack this critical capability, which must be explicitly trained. The proposed supervised fine-tuning approach with reasoning traces offers a promising solution to this problem. However, further research is needed to fully understand the implications of this work and to develop more effective context-aware turn-taking models. Additionally, the authors' emphasis on the need for explicit training raises questions about the role of zero-shot learning in natural language processing and speech recognition.
Recommendations
- ✓ Future research should focus on developing more effective context-aware turn-taking models that can learn from large datasets and adapt to various conversation settings.
- ✓ Policymakers and developers should prioritize the development of voice AI assistants with context-aware turn-taking capabilities to ensure their safe and effective use in various applications.