Academic

Free Lunch for Pass@$k$? Low Cost Diverse Sampling for Diffusion Language Models

arXiv:2603.04893v1 Announce Type: new Abstract: Diverse outputs in text generation are necessary for effective exploration in complex reasoning tasks, such as code generation and mathematical problem solving. Such Pass@$k$ problems benefit from distinct candidates covering the solution space. However, traditional sampling approaches often waste computational resources on repetitive failure modes. While Diffusion Language Models have emerged as a competitive alternative to the prevailing Autoregressive paradigm, they remain susceptible to this redundancy, with independent samples frequently collapsing into similar modes. To address this, we propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models. Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples, actively penalising redundancy. Unlike prior methods that require retraining or beam search, our str

S
Sean Lamont, Christian Walder, Paul Montague, Amir Dezfouli, Michael Norrish · · 1 min read · 2 views
#cs.CL #cs.AI

arXiv:2603.04893v1 Announce Type: new Abstract: Diverse outputs in text generation are necessary for effective exploration in complex reasoning tasks, such as code generation and mathematical problem solving. Such Pass@$k$ problems benefit from distinct candidates covering the solution space. However, traditional sampling approaches often waste computational resources on repetitive failure modes. While Diffusion Language Models have emerged as a competitive alternative to the prevailing Autoregressive paradigm, they remain susceptible to this redundancy, with independent samples frequently collapsing into similar modes. To address this, we propose a training free, low cost intervention to enhance generative diversity in Diffusion Language Models. Our approach modifies intermediate samples in a batch sequentially, where each sample is repelled from the feature space of previous samples, actively penalising redundancy. Unlike prior methods that require retraining or beam search, our strategy incurs negligible computational overhead, while ensuring that each sample contributes a unique perspective to the batch. We evaluate our method on the HumanEval and GSM8K benchmarks using the LLaDA-8B-Instruct model. Our results demonstrate significantly improved diversity and Pass@$k$ performance across various temperature settings. As a simple modification to the sampling process, our method offers an immediate, low-cost improvement for current and future Diffusion Language Models in tasks that benefit from diverse solution search. We make our code available at https://github.com/sean-lamont/odd.

Executive Summary

The article proposes a low-cost intervention to enhance generative diversity in Diffusion Language Models. The approach modifies intermediate samples in a batch to penalize redundancy, ensuring each sample contributes a unique perspective. This method is training-free, incurs negligible computational overhead, and demonstrates improved diversity and Pass@$k$ performance across various temperature settings. The approach is evaluated on the HumanEval and GSM8K benchmarks using the LLaDA-8B-Instruct model, with significant improvements observed. The code is made available, offering an immediate, low-cost improvement for current and future Diffusion Language Models in tasks that benefit from diverse solution search.

Key Points

  • ▸ Low-cost intervention to enhance generative diversity in Diffusion Language Models
  • ▸ Training-free approach with negligible computational overhead
  • ▸ Improved diversity and Pass@$k$ performance across various temperature settings

Merits

Efficient Sampling

The proposed method efficiently samples diverse outputs without wasting computational resources on repetitive failure modes.

Improved Performance

The approach demonstrates significant improvements in diversity and Pass@$k$ performance, making it a valuable contribution to the field.

Demerits

Limited Evaluation

The evaluation is limited to two benchmarks, and further testing on a broader range of tasks and models is necessary to fully assess the approach's effectiveness.

Expert Commentary

The article presents a significant contribution to the field of natural language processing, particularly in the context of Diffusion Language Models. The proposed approach addresses a critical limitation of traditional sampling methods, which often result in repetitive failure modes. By modifying intermediate samples to penalize redundancy, the method efficiently samples diverse outputs, demonstrating improved performance on benchmarks. While further evaluation is necessary, the approach has the potential to improve the effectiveness of language models in tasks that benefit from diverse solution search, such as code generation and mathematical problem solving.

Recommendations

  • ✓ Further evaluation of the approach on a broader range of tasks and models to fully assess its effectiveness
  • ✓ Exploration of potential applications of the method in other areas of AI research, such as computer vision and robotics

Sources

Related Articles