XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts Article Swipe

PDF

Yifeng Ding , Jiawei Liu , Yuxiang Wei , Terry Yue Zhuo , Lingming Zhang ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2404.15247

We introduce XFT, a simple yet powerful training scheme, by simply merging upcycled Mixture-of-Experts (MoE) to unleash the performance limit of instruction-tuned code Large Language Models (LLMs). While vanilla sparse upcycling fails to improve instruction tuning, XFT introduces a shared expert mechanism with a novel routing weight normalization strategy into sparse upcycling, which significantly boosts instruction tuning. After fine-tuning the upcycled MoE model, XFT introduces a learnable model merging mechanism to compile the upcycled MoE model back to a dense model, achieving upcycled MoE-level performance with only dense-model compute. By applying XFT to a 1.3B model, we create a new state-of-the-art tiny code LLM (<3B) with 67.1 and 64.6 pass@1 on HumanEval and HumanEval+ respectively. With the same data and model architecture, XFT improves supervised fine-tuning (SFT) by 13% on HumanEval+, along with consistent improvements from 2% to 13% on MBPP+, MultiPL-E, and DS-1000, demonstrating its generalizability. XFT is fully orthogonal to existing techniques such as Evol-Instruct and OSS-Instruct, opening a new dimension for improving code instruction tuning. Codes are available at https://github.com/ise-uiuc/xft.

Related Topics

Concepts

Code (set theory) Power (physics) Computer science Programming language Physics Quantum mechanics Set (abstract data type)

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2404.15247
PDF: https://arxiv.org/pdf/2404.15247
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4395443836

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4395443836

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2404.15247

Digital Object Identifier
Title: XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-04-23

Full publication date if available
Authors: Yifeng Ding, Jiawei Liu, Yuxiang Wei, Terry Yue Zhuo, Lingming Zhang

List of authors in order
Landing page: https://arxiv.org/abs/2404.15247

Publisher landing page
PDF URL: https://arxiv.org/pdf/2404.15247

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2404.15247

Direct OA link when available
Concepts: Code (set theory), Power (physics), Computer science, Programming language, Physics, Quantum mechanics, Set (abstract data type)

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4395443836
doi	https://doi.org/10.48550/arxiv.2404.15247
ids.doi	https://doi.org/10.48550/arxiv.2404.15247
ids.openalex	https://openalex.org/W4395443836
fwci
type	preprint
title	XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T11309
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9624999761581421
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1711
topics[0].subfield.display_name	Signal Processing
topics[0].display_name	Music and Audio Processing
topics[1].id	https://openalex.org/T12535
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9621999859809875
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Machine Learning and Data Classification
topics[2].id	https://openalex.org/T12072
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9585000276565552
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1702
topics[2].subfield.display_name	Artificial Intelligence
topics[2].display_name	Machine Learning and Algorithms
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C2776760102
concepts[0].level	3
concepts[0].score	0.7089268565177917
concepts[0].wikidata	https://www.wikidata.org/wiki/Q5139990
concepts[0].display_name	Code (set theory)
concepts[1].id	https://openalex.org/C163258240
concepts[1].level	2
concepts[1].score	0.5835591554641724
concepts[1].wikidata	https://www.wikidata.org/wiki/Q25342
concepts[1].display_name	Power (physics)
concepts[2].id	https://openalex.org/C41008148
concepts[2].level	0
concepts[2].score	0.5453017950057983
concepts[2].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[2].display_name	Computer science
concepts[3].id	https://openalex.org/C199360897
concepts[3].level	1
concepts[3].score	0.3329571783542633
concepts[3].wikidata	https://www.wikidata.org/wiki/Q9143
concepts[3].display_name	Programming language
concepts[4].id	https://openalex.org/C121332964
concepts[4].level	0
concepts[4].score	0.14584851264953613
concepts[4].wikidata	https://www.wikidata.org/wiki/Q413
concepts[4].display_name	Physics
concepts[5].id	https://openalex.org/C62520636
concepts[5].level	1
concepts[5].score	0.0
concepts[5].wikidata	https://www.wikidata.org/wiki/Q944
concepts[5].display_name	Quantum mechanics
concepts[6].id	https://openalex.org/C177264268
concepts[6].level	2
concepts[6].score	0.0
concepts[6].wikidata	https://www.wikidata.org/wiki/Q1514741
concepts[6].display_name	Set (abstract data type)
keywords[0].id	https://openalex.org/keywords/code
keywords[0].score	0.7089268565177917
keywords[0].display_name	Code (set theory)
keywords[1].id	https://openalex.org/keywords/power
keywords[1].score	0.5835591554641724
keywords[1].display_name	Power (physics)
keywords[2].id	https://openalex.org/keywords/computer-science
keywords[2].score	0.5453017950057983
keywords[2].display_name	Computer science
keywords[3].id	https://openalex.org/keywords/programming-language
keywords[3].score	0.3329571783542633
keywords[3].display_name	Programming language
keywords[4].id	https://openalex.org/keywords/physics
keywords[4].score	0.14584851264953613
keywords[4].display_name	Physics
language	en
locations[0].id	pmh:oai:arXiv.org:2404.15247
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2404.15247
locations[0].version	submittedVersion
locations[0].raw_type
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2404.15247
locations[1].id	doi:10.48550/arxiv.2404.15247
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2404.15247
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5100954379
authorships[0].author.orcid	https://orcid.org/0009-0008-2856-2271
authorships[0].author.display_name	Yifeng Ding
authorships[0].author_position	first
authorships[0].raw_author_name	Ding, Yifeng
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5100320734
authorships[1].author.orcid	https://orcid.org/0000-0001-7122-8625
authorships[1].author.display_name	Jiawei Liu
authorships[1].author_position	middle
authorships[1].raw_author_name	Liu, Jiawei
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5058312928
authorships[2].author.orcid	https://orcid.org/0000-0002-4391-3753
authorships[2].author.display_name	Yuxiang Wei
authorships[2].author_position	middle
authorships[2].raw_author_name	Wei, Yuxiang
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5082624117
authorships[3].author.orcid	https://orcid.org/0000-0002-5760-5188
authorships[3].author.display_name	Terry Yue Zhuo
authorships[3].author_position	middle
authorships[3].raw_author_name	Zhuo, Terry Yue
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5043546718
authorships[4].author.orcid	https://orcid.org/0000-0001-5175-2702
authorships[4].author.display_name	Lingming Zhang
authorships[4].author_position	last
authorships[4].raw_author_name	Zhang, Lingming
authorships[4].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2404.15247
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	XFT: Unlocking the Power of Code Instruction Tuning by Simply Merging Upcycled Mixture-of-Experts
has_fulltext	True
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T11309
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9624999761581421
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1711
primary_topic.subfield.display_name	Signal Processing
primary_topic.display_name	Music and Audio Processing
related_works	https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2382290278, https://openalex.org/W4395014643, https://openalex.org/W4391913857, https://openalex.org/W2350741829
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2404.15247
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2404.15247
best_oa_location.version	submittedVersion
best_oa_location.raw_type
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2404.15247
primary_location.id	pmh:oai:arXiv.org:2404.15247
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2404.15247
primary_location.version	submittedVersion
primary_location.raw_type
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2404.15247
publication_date	2024-04-23
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	3, 38, 43, 65, 78, 93, 98, 160
abstract_inverted_index.2%	136
abstract_inverted_index.By	89
abstract_inverted_index.We	0
abstract_inverted_index.as	155
abstract_inverted_index.at	171
abstract_inverted_index.by	9, 127
abstract_inverted_index.is	148
abstract_inverted_index.of	20
abstract_inverted_index.on	110, 129, 139
abstract_inverted_index.to	15, 32, 70, 77, 92, 137, 151
abstract_inverted_index.we	96
abstract_inverted_index.13%	128, 138
abstract_inverted_index.LLM	103
abstract_inverted_index.MoE	61, 74
abstract_inverted_index.XFT	36, 63, 91, 122, 147
abstract_inverted_index.and	107, 112, 119, 142, 157
abstract_inverted_index.are	169
abstract_inverted_index.for	163
abstract_inverted_index.its	145
abstract_inverted_index.new	99, 161
abstract_inverted_index.the	17, 59, 72, 116
abstract_inverted_index.yet	5
abstract_inverted_index.1.3B	94
abstract_inverted_index.64.6	108
abstract_inverted_index.67.1	106
abstract_inverted_index.With	115
abstract_inverted_index.XFT,	2
abstract_inverted_index.back	76
abstract_inverted_index.code	22, 102, 165
abstract_inverted_index.data	118
abstract_inverted_index.from	135
abstract_inverted_index.into	49
abstract_inverted_index.only	86
abstract_inverted_index.same	117
abstract_inverted_index.such	154
abstract_inverted_index.tiny	101
abstract_inverted_index.with	42, 85, 105, 132
abstract_inverted_index.(MoE)	14
abstract_inverted_index.(SFT)	126
abstract_inverted_index.After	57
abstract_inverted_index.Codes	168
abstract_inverted_index.Large	23
abstract_inverted_index.While	27
abstract_inverted_index.along	131
abstract_inverted_index.dense	79
abstract_inverted_index.fails	31
abstract_inverted_index.fully	149
abstract_inverted_index.limit	19
abstract_inverted_index.model	67, 75, 120
abstract_inverted_index.novel	44
abstract_inverted_index.which	52
abstract_inverted_index.MBPP+,	140
abstract_inverted_index.Models	25
abstract_inverted_index.boosts	54
abstract_inverted_index.create	97
abstract_inverted_index.expert	40
abstract_inverted_index.model,	62, 80, 95
abstract_inverted_index.pass@1	109
abstract_inverted_index.shared	39
abstract_inverted_index.simple	4
abstract_inverted_index.simply	10
abstract_inverted_index.sparse	29, 50
abstract_inverted_index.weight	46
abstract_inverted_index.(LLMs).	26
abstract_inverted_index.compile	71
abstract_inverted_index.improve	33
abstract_inverted_index.merging	11, 68
abstract_inverted_index.opening	159
abstract_inverted_index.routing	45
abstract_inverted_index.scheme,	8
abstract_inverted_index.tuning,	35
abstract_inverted_index.tuning.	56, 167
abstract_inverted_index.unleash	16
abstract_inverted_index.vanilla	28
abstract_inverted_index.(<3B)	104
abstract_inverted_index.DS-1000,	143
abstract_inverted_index.Language	24
abstract_inverted_index.applying	90
abstract_inverted_index.compute.	88
abstract_inverted_index.existing	152
abstract_inverted_index.improves	123
abstract_inverted_index.powerful	6
abstract_inverted_index.strategy	48
abstract_inverted_index.training	7
abstract_inverted_index.upcycled	12, 60, 73, 82
abstract_inverted_index.HumanEval	111
abstract_inverted_index.MoE-level	83
abstract_inverted_index.achieving	81
abstract_inverted_index.available	170
abstract_inverted_index.dimension	162
abstract_inverted_index.improving	164
abstract_inverted_index.introduce	1
abstract_inverted_index.learnable	66
abstract_inverted_index.mechanism	41, 69
abstract_inverted_index.upcycling	30
abstract_inverted_index.HumanEval+	113
abstract_inverted_index.MultiPL-E,	141
abstract_inverted_index.consistent	133
abstract_inverted_index.introduces	37, 64
abstract_inverted_index.orthogonal	150
abstract_inverted_index.supervised	124
abstract_inverted_index.techniques	153
abstract_inverted_index.upcycling,	51
abstract_inverted_index.HumanEval+,	130
abstract_inverted_index.dense-model	87
abstract_inverted_index.fine-tuning	58, 125
abstract_inverted_index.instruction	34, 55, 166
abstract_inverted_index.performance	18, 84
abstract_inverted_index.improvements	134
abstract_inverted_index.Evol-Instruct	156
abstract_inverted_index.OSS-Instruct,	158
abstract_inverted_index.architecture,	121
abstract_inverted_index.demonstrating	144
abstract_inverted_index.normalization	47
abstract_inverted_index.respectively.	114
abstract_inverted_index.significantly	53
abstract_inverted_index.state-of-the-art	100
abstract_inverted_index.generalizability.	146
abstract_inverted_index.instruction-tuned	21
abstract_inverted_index.Mixture-of-Experts	13
abstract_inverted_index.https://github.com/ise-uiuc/xft.	172
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	5
citation_normalized_percentile