Xwin-LM: Strong and Scalable Alignment Practice for LLMs Article Swipe
Bolin Ni
,
JingCheng Hu
,
Yixuan Wei
,
Houwen Peng
,
Zheng Zhang
,
Gaofeng Meng
,
Han Hu
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2405.20335
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2405.20335
In this work, we present Xwin-LM, a comprehensive suite of alignment methodologies for large language models (LLMs). This suite encompasses several key techniques, including supervised finetuning (SFT), reward modeling (RM), rejection sampling finetuning (RS), and direct preference optimization (DPO). The key components are as follows: (1) Xwin-LM-SFT, models initially finetuned with high-quality instruction data; (2) Xwin-Pair, a large-scale, multi-turn preference dataset meticulously annotated using GPT-4; (3) Xwin-RM, reward models trained on Xwin-Pair, developed at scales of 7B, 13B, and 70B parameters; (4) Xwin-Set, a multiwise preference dataset in which each prompt is linked to 64 unique responses generated by Xwin-LM-SFT and scored by Xwin-RM; (5) Xwin-LM-RS, models finetuned with the highest-scoring responses from Xwin-Set; (6) Xwin-LM-DPO, models further optimized on Xwin-Set using the DPO algorithm. Our evaluations on AlpacaEval and MT-bench demonstrate consistent and significant improvements across the pipeline, demonstrating the strength and scalability of Xwin-LM. The repository https://github.com/Xwin-LM/Xwin-LM will be continually updated to foster community research.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2405.20335
- https://arxiv.org/pdf/2405.20335
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4399270632
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4399270632Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2405.20335Digital Object Identifier
- Title
-
Xwin-LM: Strong and Scalable Alignment Practice for LLMsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-05-30Full publication date if available
- Authors
-
Bolin Ni, JingCheng Hu, Yixuan Wei, Houwen Peng, Zheng Zhang, Gaofeng Meng, Han HuList of authors in order
- Landing page
-
https://arxiv.org/abs/2405.20335Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2405.20335Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2405.20335Direct OA link when available
- Concepts
-
Scalability, Computer science, Business, DatabaseTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4399270632 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2405.20335 |
| ids.doi | https://doi.org/10.48550/arxiv.2405.20335 |
| ids.openalex | https://openalex.org/W4399270632 |
| fwci | |
| type | preprint |
| title | Xwin-LM: Strong and Scalable Alignment Practice for LLMs |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T13999 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9279000163078308 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1710 |
| topics[0].subfield.display_name | Information Systems |
| topics[0].display_name | Digital Rights Management and Security |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C48044578 |
| concepts[0].level | 2 |
| concepts[0].score | 0.5997198224067688 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q727490 |
| concepts[0].display_name | Scalability |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.35642850399017334 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C144133560 |
| concepts[2].level | 0 |
| concepts[2].score | 0.32891812920570374 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q4830453 |
| concepts[2].display_name | Business |
| concepts[3].id | https://openalex.org/C77088390 |
| concepts[3].level | 1 |
| concepts[3].score | 0.0 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q8513 |
| concepts[3].display_name | Database |
| keywords[0].id | https://openalex.org/keywords/scalability |
| keywords[0].score | 0.5997198224067688 |
| keywords[0].display_name | Scalability |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.35642850399017334 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/business |
| keywords[2].score | 0.32891812920570374 |
| keywords[2].display_name | Business |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2405.20335 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2405.20335 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2405.20335 |
| locations[1].id | doi:10.48550/arxiv.2405.20335 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2405.20335 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5012017422 |
| authorships[0].author.orcid | https://orcid.org/0009-0000-7160-5523 |
| authorships[0].author.display_name | Bolin Ni |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Ni, Bolin |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5111218285 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | JingCheng Hu |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Hu, JingCheng |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5009248953 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-1775-7301 |
| authorships[2].author.display_name | Yixuan Wei |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Wei, Yixuan |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5080495327 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-8544-8952 |
| authorships[3].author.display_name | Houwen Peng |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Peng, Houwen |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5100459174 |
| authorships[4].author.orcid | https://orcid.org/0000-0003-2076-6372 |
| authorships[4].author.display_name | Zheng Zhang |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Zhang, Zheng |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5100675867 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-7103-6321 |
| authorships[5].author.display_name | Gaofeng Meng |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Meng, Gaofeng |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5101976576 |
| authorships[6].author.orcid | https://orcid.org/0000-0001-5104-6146 |
| authorships[6].author.display_name | Han Hu |
| authorships[6].author_position | last |
| authorships[6].raw_author_name | Hu, Han |
| authorships[6].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2405.20335 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Xwin-LM: Strong and Scalable Alignment Practice for LLMs |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T13999 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9279000163078308 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1710 |
| primary_topic.subfield.display_name | Information Systems |
| primary_topic.display_name | Digital Rights Management and Security |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2389214306, https://openalex.org/W4396696052, https://openalex.org/W2382290278 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2405.20335 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2405.20335 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2405.20335 |
| primary_location.id | pmh:oai:arXiv.org:2405.20335 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2405.20335 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2405.20335 |
| publication_date | 2024-05-30 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 6, 56, 83 |
| abstract_inverted_index.64 | 94 |
| abstract_inverted_index.In | 0 |
| abstract_inverted_index.as | 43 |
| abstract_inverted_index.at | 73 |
| abstract_inverted_index.be | 150 |
| abstract_inverted_index.by | 98, 102 |
| abstract_inverted_index.in | 87 |
| abstract_inverted_index.is | 91 |
| abstract_inverted_index.of | 9, 75, 144 |
| abstract_inverted_index.on | 70, 119, 127 |
| abstract_inverted_index.to | 93, 153 |
| abstract_inverted_index.we | 3 |
| abstract_inverted_index.(1) | 45 |
| abstract_inverted_index.(2) | 54 |
| abstract_inverted_index.(3) | 65 |
| abstract_inverted_index.(4) | 81 |
| abstract_inverted_index.(5) | 104 |
| abstract_inverted_index.(6) | 114 |
| abstract_inverted_index.70B | 79 |
| abstract_inverted_index.7B, | 76 |
| abstract_inverted_index.DPO | 123 |
| abstract_inverted_index.Our | 125 |
| abstract_inverted_index.The | 39, 146 |
| abstract_inverted_index.and | 34, 78, 100, 129, 133, 142 |
| abstract_inverted_index.are | 42 |
| abstract_inverted_index.for | 12 |
| abstract_inverted_index.key | 21, 40 |
| abstract_inverted_index.the | 109, 122, 137, 140 |
| abstract_inverted_index.13B, | 77 |
| abstract_inverted_index.This | 17 |
| abstract_inverted_index.each | 89 |
| abstract_inverted_index.from | 112 |
| abstract_inverted_index.this | 1 |
| abstract_inverted_index.will | 149 |
| abstract_inverted_index.with | 50, 108 |
| abstract_inverted_index.(RM), | 29 |
| abstract_inverted_index.(RS), | 33 |
| abstract_inverted_index.data; | 53 |
| abstract_inverted_index.large | 13 |
| abstract_inverted_index.suite | 8, 18 |
| abstract_inverted_index.using | 63, 121 |
| abstract_inverted_index.which | 88 |
| abstract_inverted_index.work, | 2 |
| abstract_inverted_index.(DPO). | 38 |
| abstract_inverted_index.(SFT), | 26 |
| abstract_inverted_index.GPT-4; | 64 |
| abstract_inverted_index.across | 136 |
| abstract_inverted_index.direct | 35 |
| abstract_inverted_index.foster | 154 |
| abstract_inverted_index.linked | 92 |
| abstract_inverted_index.models | 15, 47, 68, 106, 116 |
| abstract_inverted_index.prompt | 90 |
| abstract_inverted_index.reward | 27, 67 |
| abstract_inverted_index.scales | 74 |
| abstract_inverted_index.scored | 101 |
| abstract_inverted_index.unique | 95 |
| abstract_inverted_index.(LLMs). | 16 |
| abstract_inverted_index.dataset | 60, 86 |
| abstract_inverted_index.further | 117 |
| abstract_inverted_index.present | 4 |
| abstract_inverted_index.several | 20 |
| abstract_inverted_index.trained | 69 |
| abstract_inverted_index.updated | 152 |
| abstract_inverted_index.MT-bench | 130 |
| abstract_inverted_index.Xwin-LM, | 5 |
| abstract_inverted_index.Xwin-LM. | 145 |
| abstract_inverted_index.Xwin-RM, | 66 |
| abstract_inverted_index.Xwin-RM; | 103 |
| abstract_inverted_index.Xwin-Set | 120 |
| abstract_inverted_index.follows: | 44 |
| abstract_inverted_index.language | 14 |
| abstract_inverted_index.modeling | 28 |
| abstract_inverted_index.sampling | 31 |
| abstract_inverted_index.strength | 141 |
| abstract_inverted_index.Xwin-Set, | 82 |
| abstract_inverted_index.Xwin-Set; | 113 |
| abstract_inverted_index.alignment | 10 |
| abstract_inverted_index.annotated | 62 |
| abstract_inverted_index.community | 155 |
| abstract_inverted_index.developed | 72 |
| abstract_inverted_index.finetuned | 49, 107 |
| abstract_inverted_index.generated | 97 |
| abstract_inverted_index.including | 23 |
| abstract_inverted_index.initially | 48 |
| abstract_inverted_index.multiwise | 84 |
| abstract_inverted_index.optimized | 118 |
| abstract_inverted_index.pipeline, | 138 |
| abstract_inverted_index.rejection | 30 |
| abstract_inverted_index.research. | 156 |
| abstract_inverted_index.responses | 96, 111 |
| abstract_inverted_index.AlpacaEval | 128 |
| abstract_inverted_index.Xwin-Pair, | 55, 71 |
| abstract_inverted_index.algorithm. | 124 |
| abstract_inverted_index.components | 41 |
| abstract_inverted_index.consistent | 132 |
| abstract_inverted_index.finetuning | 25, 32 |
| abstract_inverted_index.multi-turn | 58 |
| abstract_inverted_index.preference | 36, 59, 85 |
| abstract_inverted_index.repository | 147 |
| abstract_inverted_index.supervised | 24 |
| abstract_inverted_index.Xwin-LM-RS, | 105 |
| abstract_inverted_index.Xwin-LM-SFT | 99 |
| abstract_inverted_index.continually | 151 |
| abstract_inverted_index.demonstrate | 131 |
| abstract_inverted_index.encompasses | 19 |
| abstract_inverted_index.evaluations | 126 |
| abstract_inverted_index.instruction | 52 |
| abstract_inverted_index.parameters; | 80 |
| abstract_inverted_index.scalability | 143 |
| abstract_inverted_index.significant | 134 |
| abstract_inverted_index.techniques, | 22 |
| abstract_inverted_index.Xwin-LM-DPO, | 115 |
| abstract_inverted_index.Xwin-LM-SFT, | 46 |
| abstract_inverted_index.high-quality | 51 |
| abstract_inverted_index.improvements | 135 |
| abstract_inverted_index.large-scale, | 57 |
| abstract_inverted_index.meticulously | 61 |
| abstract_inverted_index.optimization | 37 |
| abstract_inverted_index.comprehensive | 7 |
| abstract_inverted_index.demonstrating | 139 |
| abstract_inverted_index.methodologies | 11 |
| abstract_inverted_index.highest-scoring | 110 |
| abstract_inverted_index.https://github.com/Xwin-LM/Xwin-LM | 148 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 7 |
| citation_normalized_percentile |