LK Losses: Direct Acceptance Rate Optimization for Speculative Decoding
arXiv:2602.23881v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to propose candidate tokens that are then verified in parallel by the target model. The speedup is significantly determined by the acceptance rate, yet standard training minimizes Kullback-Leibler (KL) divergence as a proxy objective. While KL divergence and acceptance rate share the same global optimum, small draft models, having limited capacity, typically converge to suboptimal solutions where minimizing KL does not guarantee maximizing acceptance rate. To address this issue, we propose LK losses, special training objectives that directly target acceptance rate. Comprehensive experiments across four draft architectures and six target models, ranging from 8B to 685B parameters, demonstrate consistent improvements in acceptance metrics across all configurations compared to the standard KL-based training. We evaluate
arXiv:2602.23881v1 Announce Type: cross Abstract: Speculative decoding accelerates autoregressive large language model (LLM) inference by using a lightweight draft model to propose candidate tokens that are then verified in parallel by the target model. The speedup is significantly determined by the acceptance rate, yet standard training minimizes Kullback-Leibler (KL) divergence as a proxy objective. While KL divergence and acceptance rate share the same global optimum, small draft models, having limited capacity, typically converge to suboptimal solutions where minimizing KL does not guarantee maximizing acceptance rate. To address this issue, we propose LK losses, special training objectives that directly target acceptance rate. Comprehensive experiments across four draft architectures and six target models, ranging from 8B to 685B parameters, demonstrate consistent improvements in acceptance metrics across all configurations compared to the standard KL-based training. We evaluate our approach on general, coding and math domains and report gains of up to 8-10% in average acceptance length. LK losses are easy to implement, introduce no computational overhead and can be directly integrated into any existing speculator training framework, making them a compelling alternative to the existing draft training objectives.
Executive Summary
The authors propose LK losses, a novel training objective that directly optimizes the acceptance rate of speculative decoding in autoregressive large language models. By targeting the acceptance rate, LK losses address the suboptimal solutions that standard KL divergence training can lead to, particularly in small draft models. The approach is demonstrated to yield consistent improvements in acceptance metrics across various configurations, including gains of up to 8-10% in average acceptance length. LK losses are easy to implement, computationally efficient, and can be integrated into existing training frameworks, making them a promising alternative to traditional draft training objectives.
Key Points
- ▸ LK losses directly optimize acceptance rate for speculative decoding
- ▸ Approach addresses suboptimal solutions in small draft models
- ▸ Consistent improvements in acceptance metrics across various configurations
Merits
Strength
LK losses address a critical limitation of standard KL divergence training, enabling more effective speculative decoding in autoregressive LLMs.
Practical Application
LK losses are easy to implement, computationally efficient, and can be integrated into existing training frameworks, making them a practical solution for industry and research applications.
Demerits
Limitation
The effectiveness of LK losses may be dependent on the specific configuration of the draft and target models, requiring further investigation into their robustness across different architectures.
Scalability
While LK losses are computationally efficient, their application to very large models may require additional computational resources or optimizations to ensure scalability.
Expert Commentary
The introduction of LK losses marks a significant advancement in the field of speculative decoding for autoregressive large language models. By directly targeting the acceptance rate, LK losses address a critical limitation of standard KL divergence training, enabling more effective speculative decoding and improved model performance. The ease of implementation, computational efficiency, and scalability of LK losses make them a compelling alternative to traditional draft training objectives. As the field continues to evolve, the adoption of LK losses will likely have far-reaching implications for the development and deployment of autoregressive LLMs.
Recommendations
- ✓ Further investigation into the robustness of LK losses across different draft and target model architectures is necessary to fully understand their limitations and potential.
- ✓ The integration of LK losses into existing training frameworks and the development of new models optimized for LK losses will be crucial for widespread adoption and practical application.