CoDAR: Continuous Diffusion Language Models are More Powerful Than You Think
arXiv:2603.02547v1 Announce Type: new Abstract: We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the flu
arXiv:2603.02547v1 Announce Type: new Abstract: We study why continuous diffusion language models (DLMs) have lagged behind discrete diffusion approaches despite their appealing continuous generative dynamics. Under a controlled token--recovery study, we identify token rounding, the final projection from denoised embeddings to tokens, as a primary bottleneck. Building on these insights, we propose CoDAR (Continuous Diffusion with Contextual AutoRegressive Decoder), a two--stage framework that keeps diffusion entirely continuous in an embedding space while learning a strong, context--conditional discretizer: an autoregressive Transformer decoder that cross--attends to the denoised embedding sequence and performs contextualized rounding to tokens. Experiments on LM1B and OpenWebText demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs, while exposing a simple decoder--temperature knob to navigate the fluency--diversity trade off.
Executive Summary
This article explores the limitations of continuous diffusion language models (DLMs) and proposes a novel two-stage framework, CoDAR, to address the primary bottleneck of token rounding. Through controlled token-recovery studies and experiments on LM1B and OpenWebText, the authors demonstrate that CoDAR substantially improves generation quality and becomes competitive with strong discrete DLMs. The proposed framework enables a trade-off between fluency and diversity through a decoder-temperature knob. This breakthrough has significant implications for the field of natural language processing and the development of more powerful and flexible DLMs.
Key Points
- ▸ Continuous diffusion language models (DLMs) have lagged behind discrete approaches due to token rounding
- ▸ CoDAR addresses the bottleneck of token rounding through a two-stage framework
- ▸ Experiments demonstrate CoDAR's improved generation quality and competitiveness with strong discrete DLMs
Merits
Strength in Addressing Token Rounding
CoDAR's two-stage framework effectively addresses the primary bottleneck of token rounding, enabling more powerful and flexible continuous DLMs.
Improved Generation Quality
Experiments demonstrate that CoDAR substantially improves generation quality over latent diffusion and becomes competitive with strong discrete DLMs.
Flexibility in Fluency-Diversity Trade-off
CoDAR's decoder-temperature knob allows for a simple navigation of the fluency-diversity trade-off.
Demerits
Limited Scope of Experiments
The article's experiments are limited to two datasets, LM1B and OpenWebText, which may not be representative of the broader scope of natural language processing applications.
Dependence on Autoregressive Transformer Decoder
CoDAR's performance relies on the autoregressive Transformer decoder, which may not be suitable for all applications or datasets.
Expert Commentary
The article's contributions to the field of natural language processing are significant, as CoDAR addresses a critical limitation of continuous DLMs and demonstrates improved generation quality and competitiveness with strong discrete DLMs. However, the article's dependence on autoregressive Transformer decoder and limited scope of experiments may be seen as limitations. As a potential area for future research, exploring the application of CoDAR to a broader range of datasets and tasks would be beneficial.
Recommendations
- ✓ Further experimentation with CoDAR on a wider range of datasets and tasks to confirm its improved generation quality and competitiveness with strong discrete DLMs.
- ✓ Investigation into the potential applications of CoDAR in areas such as language translation, text summarization, and question answering.