This paper establishes sequence-level optimization as the superior paradigm for fine-tuning diffusion LLMs. They introduce a new machine learning framework called ELBO-based Sequence-level Policy Optimization (ESPO), designed to address the fundamental mismatch when applying Reinforcement Learning (RL) to non-autoregressive diffusion Large Language Models (dLLMs). Traditional RL methods rely on token-level conditional probabilities, which dLLMs lack due to their holistic, non-autoregressive generation process. ESPO resolves this by treating the entire sequence generation as a single action and utilizing the Evidence Lower Bound (ELBO) as a tractable, sequence-level likelihood proxy for optimization. Through comprehensive experiments on tasks like mathematical reasoning and planning, the authors demonstrate that ESPO consistently and significantly outperforms prior token-level RL baselines by enabling stable and principled large-scale training. The results establish sequence-level optimization as the superior paradigm for fine-tuning dLLMs.

Ver todos los episodios