URPO: A Unified Reward & Policy Optimization Framework for Large Language Models Article Swipe

MAKE IMPACT

Sara A. Lu , Hua Wang , Zhi Chen , Yaohua Tang · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2507.17515

Large-scale alignment pipelines typically pair a policy model with a separately trained reward model whose parameters remain frozen during reinforcement learning (RL). This separation creates a complex, resource-intensive pipeline and suffers from a performance ceiling due to a static reward signal. We propose a novel framework, Unified Reward & Policy Optimization (URPO), that unifies instruction-following ("player") and reward modeling ("referee") within a single model and a single training phase. Our method recasts all alignment data-including preference pairs, verifiable reasoning, and open-ended instructions-into a unified generative format optimized by a single Group-Relative Policy Optimization (GRPO) loop. This enables the model to learn from ground-truth preferences and verifiable logic while simultaneously generating its own rewards for open-ended tasks. Experiments on the Qwen2.5-7B model demonstrate URPO's superiority. Our unified model significantly outperforms a strong baseline using a separate generative reward model, boosting the instruction-following score on AlpacaEval from 42.24 to 44.84 and the composite reasoning average from 32.66 to 35.66. Furthermore, URPO cultivates a superior internal evaluator as a byproduct of training, achieving a RewardBench score of 85.15 and surpassing the dedicated reward model it replaces (83.55). By eliminating the need for a separate reward model and fostering a co-evolutionary dynamic between generation and evaluation, URPO presents a simpler, more efficient, and more effective path towards robustly aligned language models.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2507.17515

PDF

https://arxiv.org/pdf/2507.17515

OA Status

green

OpenAlex ID

https://openalex.org/W4414881550

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4414881550

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2507.17515

Digital Object Identifier

Title

URPO: A Unified Reward & Policy Optimization Framework for Large Language Models

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-07-23

Full publication date if available

Authors

Sara A. Lu, Hua Wang, Zhi Chen, Yaohua Tang

List of authors in order

Landing page

https://arxiv.org/abs/2507.17515

Publisher landing page

PDF URL

https://arxiv.org/pdf/2507.17515

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2507.17515

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload