Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization Article Swipe
Jiahe Lin
,
Anderson Schneider
,
Molei Tao
,
Wei Deng
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2510.08554
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2510.08554
Diffusion language models (DLMs) enable parallel, order-agnostic generation with iterative refinement, offering a flexible alternative to autoregressive large language models (LLMs). However, adapting reinforcement learning (RL) fine-tuning to DLMs remains an open challenge because of the intractable likelihood. Pioneering work such as diffu-GRPO estimated token-level likelihoods via one-step unmasking. While computationally efficient, this approach is severely biased. A more principled foundation lies in sequence-level likelihoods, where the evidence lower bound (ELBO) serves as a surrogate. Yet, despite this clean mathematical connection, ELBO-based methods have seen limited adoption due to the prohibitive cost of likelihood evaluation. In this work, we revisit ELBO estimation and disentangle its sources of variance. This decomposition motivates reducing variance through fast, deterministic integral approximations along a few pivotal dimensions. Building on this insight, we introduce \textbf{Group Diffusion Policy Optimization (GDPO)}, a new RL algorithm tailored for DLMs. GDPO leverages simple yet effective Semi-deterministic Monte Carlo schemes to mitigate the variance explosion of ELBO estimators under vanilla double Monte Carlo sampling, yielding a provably lower-variance estimator under tight evaluation budgets. Empirically, GDPO achieves consistent gains over pretrained checkpoints and outperforms diffu-GRPO, one of the state-of-the-art baselines, on the majority of math, reasoning, and coding benchmarks.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2510.08554
- https://arxiv.org/pdf/2510.08554
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4416385828
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4416385828Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2510.08554Digital Object Identifier
- Title
-
Improving Reasoning for Diffusion Language Models via Group Diffusion Policy OptimizationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-10-09Full publication date if available
- Authors
-
Jiahe Lin, Anderson Schneider, Molei Tao, Wei DengList of authors in order
- Landing page
-
https://arxiv.org/abs/2510.08554Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2510.08554Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2510.08554Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4416385828 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2510.08554 |
| ids.doi | https://doi.org/10.48550/arxiv.2510.08554 |
| ids.openalex | https://openalex.org/W4416385828 |
| fwci | |
| type | preprint |
| title | Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2510.08554 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://arxiv.org/pdf/2510.08554 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2510.08554 |
| locations[1].id | doi:10.48550/arxiv.2510.08554 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2510.08554 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5073407264 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-0282-659X |
| authorships[0].author.display_name | Jiahe Lin |
| authorships[0].author_position | middle |
| authorships[0].raw_author_name | Lin, Jiahe |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5018463862 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Anderson Schneider |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Schneider, Anderson |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5073666656 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-3308-6176 |
| authorships[2].author.display_name | Molei Tao |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Tao, Molei |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5102291043 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Wei Deng |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Deng, Wei |
| authorships[3].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2510.08554 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-11T00:00:00 |
| display_name | Improving Reasoning for Diffusion Language Models via Group Diffusion Policy Optimization |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-28T12:42:11.191002 |
| primary_topic | |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2510.08554 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2510.08554 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2510.08554 |
| primary_location.id | pmh:oai:arXiv.org:2510.08554 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://arxiv.org/pdf/2510.08554 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2510.08554 |
| publication_date | 2025-10-09 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.A | 57 |
| abstract_inverted_index.a | 12, 73, 119, 134, 165 |
| abstract_inverted_index.In | 95 |
| abstract_inverted_index.RL | 136 |
| abstract_inverted_index.an | 30 |
| abstract_inverted_index.as | 41, 72 |
| abstract_inverted_index.in | 62 |
| abstract_inverted_index.is | 54 |
| abstract_inverted_index.of | 34, 92, 106, 155, 185, 192 |
| abstract_inverted_index.on | 124, 189 |
| abstract_inverted_index.to | 15, 27, 88, 150 |
| abstract_inverted_index.we | 98, 127 |
| abstract_inverted_index.and | 102, 181, 195 |
| abstract_inverted_index.due | 87 |
| abstract_inverted_index.few | 120 |
| abstract_inverted_index.for | 139 |
| abstract_inverted_index.its | 104 |
| abstract_inverted_index.new | 135 |
| abstract_inverted_index.one | 184 |
| abstract_inverted_index.the | 35, 66, 89, 152, 186, 190 |
| abstract_inverted_index.via | 46 |
| abstract_inverted_index.yet | 144 |
| abstract_inverted_index.(RL) | 25 |
| abstract_inverted_index.DLMs | 28 |
| abstract_inverted_index.ELBO | 100, 156 |
| abstract_inverted_index.GDPO | 141, 174 |
| abstract_inverted_index.This | 108 |
| abstract_inverted_index.Yet, | 75 |
| abstract_inverted_index.cost | 91 |
| abstract_inverted_index.have | 83 |
| abstract_inverted_index.lies | 61 |
| abstract_inverted_index.more | 58 |
| abstract_inverted_index.open | 31 |
| abstract_inverted_index.over | 178 |
| abstract_inverted_index.seen | 84 |
| abstract_inverted_index.such | 40 |
| abstract_inverted_index.this | 52, 77, 96, 125 |
| abstract_inverted_index.with | 8 |
| abstract_inverted_index.work | 39 |
| abstract_inverted_index.Carlo | 148, 162 |
| abstract_inverted_index.DLMs. | 140 |
| abstract_inverted_index.Monte | 147, 161 |
| abstract_inverted_index.While | 49 |
| abstract_inverted_index.along | 118 |
| abstract_inverted_index.bound | 69 |
| abstract_inverted_index.clean | 78 |
| abstract_inverted_index.fast, | 114 |
| abstract_inverted_index.gains | 177 |
| abstract_inverted_index.large | 17 |
| abstract_inverted_index.lower | 68 |
| abstract_inverted_index.math, | 193 |
| abstract_inverted_index.tight | 170 |
| abstract_inverted_index.under | 158, 169 |
| abstract_inverted_index.where | 65 |
| abstract_inverted_index.work, | 97 |
| abstract_inverted_index.(DLMs) | 3 |
| abstract_inverted_index.(ELBO) | 70 |
| abstract_inverted_index.Policy | 131 |
| abstract_inverted_index.coding | 196 |
| abstract_inverted_index.double | 160 |
| abstract_inverted_index.enable | 4 |
| abstract_inverted_index.models | 2, 19 |
| abstract_inverted_index.serves | 71 |
| abstract_inverted_index.simple | 143 |
| abstract_inverted_index.(LLMs). | 20 |
| abstract_inverted_index.because | 33 |
| abstract_inverted_index.biased. | 56 |
| abstract_inverted_index.despite | 76 |
| abstract_inverted_index.limited | 85 |
| abstract_inverted_index.methods | 82 |
| abstract_inverted_index.pivotal | 121 |
| abstract_inverted_index.remains | 29 |
| abstract_inverted_index.revisit | 99 |
| abstract_inverted_index.schemes | 149 |
| abstract_inverted_index.sources | 105 |
| abstract_inverted_index.through | 113 |
| abstract_inverted_index.vanilla | 159 |
| abstract_inverted_index.(GDPO)}, | 133 |
| abstract_inverted_index.Building | 123 |
| abstract_inverted_index.However, | 21 |
| abstract_inverted_index.achieves | 175 |
| abstract_inverted_index.adapting | 22 |
| abstract_inverted_index.adoption | 86 |
| abstract_inverted_index.approach | 53 |
| abstract_inverted_index.budgets. | 172 |
| abstract_inverted_index.evidence | 67 |
| abstract_inverted_index.flexible | 13 |
| abstract_inverted_index.insight, | 126 |
| abstract_inverted_index.integral | 116 |
| abstract_inverted_index.language | 1, 18 |
| abstract_inverted_index.learning | 24 |
| abstract_inverted_index.majority | 191 |
| abstract_inverted_index.mitigate | 151 |
| abstract_inverted_index.offering | 11 |
| abstract_inverted_index.one-step | 47 |
| abstract_inverted_index.provably | 166 |
| abstract_inverted_index.reducing | 111 |
| abstract_inverted_index.severely | 55 |
| abstract_inverted_index.tailored | 138 |
| abstract_inverted_index.variance | 112, 153 |
| abstract_inverted_index.yielding | 164 |
| abstract_inverted_index.Diffusion | 0, 130 |
| abstract_inverted_index.algorithm | 137 |
| abstract_inverted_index.challenge | 32 |
| abstract_inverted_index.effective | 145 |
| abstract_inverted_index.estimated | 43 |
| abstract_inverted_index.estimator | 168 |
| abstract_inverted_index.explosion | 154 |
| abstract_inverted_index.introduce | 128 |
| abstract_inverted_index.iterative | 9 |
| abstract_inverted_index.leverages | 142 |
| abstract_inverted_index.motivates | 110 |
| abstract_inverted_index.parallel, | 5 |
| abstract_inverted_index.sampling, | 163 |
| abstract_inverted_index.variance. | 107 |
| abstract_inverted_index.ELBO-based | 81 |
| abstract_inverted_index.Pioneering | 38 |
| abstract_inverted_index.baselines, | 188 |
| abstract_inverted_index.consistent | 176 |
| abstract_inverted_index.diffu-GRPO | 42 |
| abstract_inverted_index.efficient, | 51 |
| abstract_inverted_index.estimation | 101 |
| abstract_inverted_index.estimators | 157 |
| abstract_inverted_index.evaluation | 171 |
| abstract_inverted_index.foundation | 60 |
| abstract_inverted_index.generation | 7 |
| abstract_inverted_index.likelihood | 93 |
| abstract_inverted_index.pretrained | 179 |
| abstract_inverted_index.principled | 59 |
| abstract_inverted_index.reasoning, | 194 |
| abstract_inverted_index.surrogate. | 74 |
| abstract_inverted_index.unmasking. | 48 |
| abstract_inverted_index.alternative | 14 |
| abstract_inverted_index.benchmarks. | 197 |
| abstract_inverted_index.checkpoints | 180 |
| abstract_inverted_index.connection, | 80 |
| abstract_inverted_index.diffu-GRPO, | 183 |
| abstract_inverted_index.dimensions. | 122 |
| abstract_inverted_index.disentangle | 103 |
| abstract_inverted_index.evaluation. | 94 |
| abstract_inverted_index.fine-tuning | 26 |
| abstract_inverted_index.intractable | 36 |
| abstract_inverted_index.likelihood. | 37 |
| abstract_inverted_index.likelihoods | 45 |
| abstract_inverted_index.outperforms | 182 |
| abstract_inverted_index.prohibitive | 90 |
| abstract_inverted_index.refinement, | 10 |
| abstract_inverted_index.token-level | 44 |
| abstract_inverted_index.Empirically, | 173 |
| abstract_inverted_index.Optimization | 132 |
| abstract_inverted_index.likelihoods, | 64 |
| abstract_inverted_index.mathematical | 79 |
| abstract_inverted_index.\textbf{Group | 129 |
| abstract_inverted_index.decomposition | 109 |
| abstract_inverted_index.deterministic | 115 |
| abstract_inverted_index.reinforcement | 23 |
| abstract_inverted_index.approximations | 117 |
| abstract_inverted_index.autoregressive | 16 |
| abstract_inverted_index.lower-variance | 167 |
| abstract_inverted_index.order-agnostic | 6 |
| abstract_inverted_index.sequence-level | 63 |
| abstract_inverted_index.computationally | 50 |
| abstract_inverted_index.state-of-the-art | 187 |
| abstract_inverted_index.Semi-deterministic | 146 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |