Weighted-Reward Preference Optimization for Implicit Model Fusion Article Swipe
Ziyi Yang
,
Fanqi Wan
,
Longguang Zhong
,
Tianyuan Shi
,
Xiaojun Quan
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2412.03187
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2412.03187
While fusing heterogeneous open-source LLMs with varying architectures and sizes can potentially integrate the strengths of different models, existing fusion methods face significant challenges, such as vocabulary alignment and merging distribution matrices. These procedures are not only complex but also prone to introducing noise and errors. In this paper, we propose an implicit fusion method, Weighted-Reward Preference Optimization (WRPO), which leverages preference optimization between the source LLMs and the target LLM to transfer their capabilities effectively. WRPO eliminates the need for vocabulary alignment and matrix fusion and can be efficiently scaled to accommodate various LLMs. To address distributional deviations between the source and target LLMs, WRPO introduces a progressive adaptation strategy that gradually shifts reliance on preferred examples from the target LLM to the source LLMs. Extensive experiments on the MT-Bench, AlpacaEval-2, and Arena-Hard benchmarks demonstrate that WRPO consistently outperforms existing knowledge fusion methods and various fine-tuning baselines. When applied to LLaMA3-8B-Instruct as the target model, WRPO achieves a length-controlled win rate of 55.9% against GPT-4-Preview-1106 on AlpacaEval-2 and a win rate of 46.2% against GPT-4-0314 on Arena-Hard. Our code is available at https://github.com/SLIT-AI/WRPO.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2412.03187
- https://arxiv.org/pdf/2412.03187
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4405089562
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4405089562Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2412.03187Digital Object Identifier
- Title
-
Weighted-Reward Preference Optimization for Implicit Model FusionWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-12-04Full publication date if available
- Authors
-
Ziyi Yang, Fanqi Wan, Longguang Zhong, Tianyuan Shi, Xiaojun QuanList of authors in order
- Landing page
-
https://arxiv.org/abs/2412.03187Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2412.03187Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2412.03187Direct OA link when available
- Concepts
-
Preference, Fusion, Psychology, Artificial intelligence, Computer science, Cognitive psychology, Econometrics, Mathematics, Statistics, Philosophy, LinguisticsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4405089562 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2412.03187 |
| ids.doi | https://doi.org/10.48550/arxiv.2412.03187 |
| ids.openalex | https://openalex.org/W4405089562 |
| fwci | |
| type | preprint |
| title | Weighted-Reward Preference Optimization for Implicit Model Fusion |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T14474 |
| topics[0].field.id | https://openalex.org/fields/22 |
| topics[0].field.display_name | Engineering |
| topics[0].score | 0.5992000102996826 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/2207 |
| topics[0].subfield.display_name | Control and Systems Engineering |
| topics[0].display_name | Industrial Technology and Control Systems |
| topics[1].id | https://openalex.org/T12095 |
| topics[1].field.id | https://openalex.org/fields/22 |
| topics[1].field.display_name | Engineering |
| topics[1].score | 0.5810999870300293 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2203 |
| topics[1].subfield.display_name | Automotive Engineering |
| topics[1].display_name | Vehicle emissions and performance |
| topics[2].id | https://openalex.org/T12945 |
| topics[2].field.id | https://openalex.org/fields/14 |
| topics[2].field.display_name | Business, Management and Accounting |
| topics[2].score | 0.5648000240325928 |
| topics[2].domain.id | https://openalex.org/domains/2 |
| topics[2].domain.display_name | Social Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1405 |
| topics[2].subfield.display_name | Management of Technology and Innovation |
| topics[2].display_name | Quality Function Deployment in Product Design |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2781249084 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7356219291687012 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q908656 |
| concepts[0].display_name | Preference |
| concepts[1].id | https://openalex.org/C158525013 |
| concepts[1].level | 2 |
| concepts[1].score | 0.4779791235923767 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q2593739 |
| concepts[1].display_name | Fusion |
| concepts[2].id | https://openalex.org/C15744967 |
| concepts[2].level | 0 |
| concepts[2].score | 0.44341331720352173 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q9418 |
| concepts[2].display_name | Psychology |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.41997015476226807 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C41008148 |
| concepts[4].level | 0 |
| concepts[4].score | 0.4161757528781891 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[4].display_name | Computer science |
| concepts[5].id | https://openalex.org/C180747234 |
| concepts[5].level | 1 |
| concepts[5].score | 0.357541024684906 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q23373 |
| concepts[5].display_name | Cognitive psychology |
| concepts[6].id | https://openalex.org/C149782125 |
| concepts[6].level | 1 |
| concepts[6].score | 0.34096378087997437 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q160039 |
| concepts[6].display_name | Econometrics |
| concepts[7].id | https://openalex.org/C33923547 |
| concepts[7].level | 0 |
| concepts[7].score | 0.30650007724761963 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[7].display_name | Mathematics |
| concepts[8].id | https://openalex.org/C105795698 |
| concepts[8].level | 1 |
| concepts[8].score | 0.24840348958969116 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[8].display_name | Statistics |
| concepts[9].id | https://openalex.org/C138885662 |
| concepts[9].level | 0 |
| concepts[9].score | 0.07228490710258484 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[9].display_name | Philosophy |
| concepts[10].id | https://openalex.org/C41895202 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q8162 |
| concepts[10].display_name | Linguistics |
| keywords[0].id | https://openalex.org/keywords/preference |
| keywords[0].score | 0.7356219291687012 |
| keywords[0].display_name | Preference |
| keywords[1].id | https://openalex.org/keywords/fusion |
| keywords[1].score | 0.4779791235923767 |
| keywords[1].display_name | Fusion |
| keywords[2].id | https://openalex.org/keywords/psychology |
| keywords[2].score | 0.44341331720352173 |
| keywords[2].display_name | Psychology |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.41997015476226807 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/computer-science |
| keywords[4].score | 0.4161757528781891 |
| keywords[4].display_name | Computer science |
| keywords[5].id | https://openalex.org/keywords/cognitive-psychology |
| keywords[5].score | 0.357541024684906 |
| keywords[5].display_name | Cognitive psychology |
| keywords[6].id | https://openalex.org/keywords/econometrics |
| keywords[6].score | 0.34096378087997437 |
| keywords[6].display_name | Econometrics |
| keywords[7].id | https://openalex.org/keywords/mathematics |
| keywords[7].score | 0.30650007724761963 |
| keywords[7].display_name | Mathematics |
| keywords[8].id | https://openalex.org/keywords/statistics |
| keywords[8].score | 0.24840348958969116 |
| keywords[8].display_name | Statistics |
| keywords[9].id | https://openalex.org/keywords/philosophy |
| keywords[9].score | 0.07228490710258484 |
| keywords[9].display_name | Philosophy |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2412.03187 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2412.03187 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2412.03187 |
| locations[1].id | doi:10.48550/arxiv.2412.03187 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2412.03187 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5100665678 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-7461-377X |
| authorships[0].author.display_name | Ziyi Yang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Yang, Ziyi |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5101262140 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Fanqi Wan |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Wan, Fanqi |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5104251305 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Longguang Zhong |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Zhong, Longguang |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5101107477 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Tianyuan Shi |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Shi, Tianyuan |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5040062188 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-8385-1083 |
| authorships[4].author.display_name | Xiaojun Quan |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Quan, Xiaojun |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2412.03187 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Weighted-Reward Preference Optimization for Implicit Model Fusion |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T14474 |
| primary_topic.field.id | https://openalex.org/fields/22 |
| primary_topic.field.display_name | Engineering |
| primary_topic.score | 0.5992000102996826 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/2207 |
| primary_topic.subfield.display_name | Control and Systems Engineering |
| primary_topic.display_name | Industrial Technology and Control Systems |
| related_works | https://openalex.org/W2090624569, https://openalex.org/W4252225730, https://openalex.org/W4381329258, https://openalex.org/W2099421762, https://openalex.org/W2530546662, https://openalex.org/W2090638348, https://openalex.org/W2181148280, https://openalex.org/W2891635047, https://openalex.org/W2368374794, https://openalex.org/W2967030268 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2412.03187 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2412.03187 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2412.03187 |
| primary_location.id | pmh:oai:arXiv.org:2412.03187 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2412.03187 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2412.03187 |
| publication_date | 2024-12-04 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 107, 158, 169 |
| abstract_inverted_index.In | 46 |
| abstract_inverted_index.To | 95 |
| abstract_inverted_index.an | 51 |
| abstract_inverted_index.as | 25, 152 |
| abstract_inverted_index.at | 182 |
| abstract_inverted_index.be | 88 |
| abstract_inverted_index.is | 180 |
| abstract_inverted_index.of | 15, 162, 172 |
| abstract_inverted_index.on | 115, 128, 166, 176 |
| abstract_inverted_index.to | 41, 71, 91, 122, 150 |
| abstract_inverted_index.we | 49 |
| abstract_inverted_index.LLM | 70, 121 |
| abstract_inverted_index.Our | 178 |
| abstract_inverted_index.and | 8, 28, 44, 67, 83, 86, 102, 132, 144, 168 |
| abstract_inverted_index.are | 34 |
| abstract_inverted_index.but | 38 |
| abstract_inverted_index.can | 10, 87 |
| abstract_inverted_index.for | 80 |
| abstract_inverted_index.not | 35 |
| abstract_inverted_index.the | 13, 64, 68, 78, 100, 119, 123, 129, 153 |
| abstract_inverted_index.win | 160, 170 |
| abstract_inverted_index.LLMs | 4, 66 |
| abstract_inverted_index.WRPO | 76, 105, 137, 156 |
| abstract_inverted_index.When | 148 |
| abstract_inverted_index.also | 39 |
| abstract_inverted_index.code | 179 |
| abstract_inverted_index.face | 21 |
| abstract_inverted_index.from | 118 |
| abstract_inverted_index.need | 79 |
| abstract_inverted_index.only | 36 |
| abstract_inverted_index.rate | 161, 171 |
| abstract_inverted_index.such | 24 |
| abstract_inverted_index.that | 111, 136 |
| abstract_inverted_index.this | 47 |
| abstract_inverted_index.with | 5 |
| abstract_inverted_index.46.2% | 173 |
| abstract_inverted_index.55.9% | 163 |
| abstract_inverted_index.LLMs, | 104 |
| abstract_inverted_index.LLMs. | 94, 125 |
| abstract_inverted_index.These | 32 |
| abstract_inverted_index.While | 0 |
| abstract_inverted_index.noise | 43 |
| abstract_inverted_index.prone | 40 |
| abstract_inverted_index.sizes | 9 |
| abstract_inverted_index.their | 73 |
| abstract_inverted_index.which | 59 |
| abstract_inverted_index.fusing | 1 |
| abstract_inverted_index.fusion | 19, 53, 85, 142 |
| abstract_inverted_index.matrix | 84 |
| abstract_inverted_index.model, | 155 |
| abstract_inverted_index.paper, | 48 |
| abstract_inverted_index.scaled | 90 |
| abstract_inverted_index.shifts | 113 |
| abstract_inverted_index.source | 65, 101, 124 |
| abstract_inverted_index.target | 69, 103, 120, 154 |
| abstract_inverted_index.(WRPO), | 58 |
| abstract_inverted_index.address | 96 |
| abstract_inverted_index.against | 164, 174 |
| abstract_inverted_index.applied | 149 |
| abstract_inverted_index.between | 63, 99 |
| abstract_inverted_index.complex | 37 |
| abstract_inverted_index.errors. | 45 |
| abstract_inverted_index.merging | 29 |
| abstract_inverted_index.method, | 54 |
| abstract_inverted_index.methods | 20, 143 |
| abstract_inverted_index.models, | 17 |
| abstract_inverted_index.propose | 50 |
| abstract_inverted_index.various | 93, 145 |
| abstract_inverted_index.varying | 6 |
| abstract_inverted_index.achieves | 157 |
| abstract_inverted_index.examples | 117 |
| abstract_inverted_index.existing | 18, 140 |
| abstract_inverted_index.implicit | 52 |
| abstract_inverted_index.reliance | 114 |
| abstract_inverted_index.strategy | 110 |
| abstract_inverted_index.transfer | 72 |
| abstract_inverted_index.Extensive | 126 |
| abstract_inverted_index.MT-Bench, | 130 |
| abstract_inverted_index.alignment | 27, 82 |
| abstract_inverted_index.available | 181 |
| abstract_inverted_index.different | 16 |
| abstract_inverted_index.gradually | 112 |
| abstract_inverted_index.integrate | 12 |
| abstract_inverted_index.knowledge | 141 |
| abstract_inverted_index.leverages | 60 |
| abstract_inverted_index.matrices. | 31 |
| abstract_inverted_index.preferred | 116 |
| abstract_inverted_index.strengths | 14 |
| abstract_inverted_index.Arena-Hard | 133 |
| abstract_inverted_index.GPT-4-0314 | 175 |
| abstract_inverted_index.Preference | 56 |
| abstract_inverted_index.adaptation | 109 |
| abstract_inverted_index.baselines. | 147 |
| abstract_inverted_index.benchmarks | 134 |
| abstract_inverted_index.deviations | 98 |
| abstract_inverted_index.eliminates | 77 |
| abstract_inverted_index.introduces | 106 |
| abstract_inverted_index.preference | 61 |
| abstract_inverted_index.procedures | 33 |
| abstract_inverted_index.vocabulary | 26, 81 |
| abstract_inverted_index.Arena-Hard. | 177 |
| abstract_inverted_index.accommodate | 92 |
| abstract_inverted_index.challenges, | 23 |
| abstract_inverted_index.demonstrate | 135 |
| abstract_inverted_index.efficiently | 89 |
| abstract_inverted_index.experiments | 127 |
| abstract_inverted_index.fine-tuning | 146 |
| abstract_inverted_index.introducing | 42 |
| abstract_inverted_index.open-source | 3 |
| abstract_inverted_index.outperforms | 139 |
| abstract_inverted_index.potentially | 11 |
| abstract_inverted_index.progressive | 108 |
| abstract_inverted_index.significant | 22 |
| abstract_inverted_index.AlpacaEval-2 | 167 |
| abstract_inverted_index.Optimization | 57 |
| abstract_inverted_index.capabilities | 74 |
| abstract_inverted_index.consistently | 138 |
| abstract_inverted_index.distribution | 30 |
| abstract_inverted_index.effectively. | 75 |
| abstract_inverted_index.optimization | 62 |
| abstract_inverted_index.AlpacaEval-2, | 131 |
| abstract_inverted_index.architectures | 7 |
| abstract_inverted_index.heterogeneous | 2 |
| abstract_inverted_index.distributional | 97 |
| abstract_inverted_index.Weighted-Reward | 55 |
| abstract_inverted_index.length-controlled | 159 |
| abstract_inverted_index.GPT-4-Preview-1106 | 165 |
| abstract_inverted_index.LLaMA3-8B-Instruct | 151 |
| abstract_inverted_index.https://github.com/SLIT-AI/WRPO. | 183 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |