Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch Article Swipe

PDF

Arthur Douillard , Yanislav Donchev , Keith Rush , Satyen Kale , Zachary Charles , Zachary Garrett , Gabriel Teston , Dave Lacey , Ross McIlroy , Jiajun Shen , Alexandre Ramé , Arthur Szlam , Marc’Aurelio Ranzato , Paul Barham ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2501.18512

Training of large language models (LLMs) is typically distributed across a large number of accelerators to reduce training time. Since internal states and parameter gradients need to be exchanged at each and every single gradient step, all devices need to be co-located using low-latency high-bandwidth communication links to support the required high volume of exchanged bits. Recently, distributed algorithms like DiLoCo have relaxed such co-location constraint: accelerators can be grouped into ``workers'', where synchronizations between workers only occur infrequently. This in turn means that workers can afford being connected by lower bandwidth communication links without affecting learning quality. However, in these methods, communication across workers still requires the same peak bandwidth as before, as the synchronizations require all parameters to be exchanged across all workers. In this paper, we improve DiLoCo in three ways. First, we synchronize only subsets of parameters in sequence, rather than all at once, which greatly reduces peak bandwidth. Second, we allow workers to continue training while synchronizing, which decreases wall clock time. Third, we quantize the data exchanged by workers, which further reduces bandwidth across workers. By properly combining these modifications, we show experimentally that we can distribute training of billion-scale parameters and reach similar quality as before, but reducing required bandwidth by two orders of magnitude.

Related Topics

Computer Science

Concepts

Computer science

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2501.18512
PDF: https://arxiv.org/pdf/2501.18512
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4407012558

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4407012558

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2501.18512

Digital Object Identifier
Title: Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-01-30

Full publication date if available
Authors: Arthur Douillard, Yanislav Donchev, Keith Rush, Satyen Kale, Zachary Charles, Zachary Garrett, Gabriel Teston, Dave Lacey, Ross McIlroy, Jiajun Shen, Alexandre Ramé, Arthur Szlam, Marc’Aurelio Ranzato, Paul Barham

List of authors in order
Landing page: https://arxiv.org/abs/2501.18512

Publisher landing page
PDF URL: https://arxiv.org/pdf/2501.18512

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2501.18512

Direct OA link when available
Concepts: Computer science

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4407012558
doi	https://doi.org/10.48550/arxiv.2501.18512
ids.doi	https://doi.org/10.48550/arxiv.2501.18512
ids.openalex	https://openalex.org/W4407012558
fwci
type	preprint
title	Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T12720
topics[0].field.id	https://openalex.org/fields/33
topics[0].field.display_name	Social Sciences
topics[0].score	0.4020000100135803
topics[0].domain.id	https://openalex.org/domains/2
topics[0].domain.display_name	Social Sciences
topics[0].subfield.id	https://openalex.org/subfields/3312
topics[0].subfield.display_name	Sociology and Political Science
topics[0].display_name	Multimedia Communication and Technology
topics[1].id	https://openalex.org/T10215
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.3765000104904175
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1702
topics[1].subfield.display_name	Artificial Intelligence
topics[1].display_name	Semantic Web and Ontologies
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.5745975971221924
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.5745975971221924
keywords[0].display_name	Computer science
language	en
locations[0].id	pmh:oai:arXiv.org:2501.18512
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2501.18512
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2501.18512
locations[1].id	doi:10.48550/arxiv.2501.18512
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2501.18512
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5052337951
authorships[0].author.orcid
authorships[0].author.display_name	Arthur Douillard
authorships[0].author_position	first
authorships[0].raw_author_name	Douillard, Arthur
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5116094346
authorships[1].author.orcid
authorships[1].author.display_name	Yanislav Donchev
authorships[1].author_position	middle
authorships[1].raw_author_name	Donchev, Yanislav
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5058003224
authorships[2].author.orcid	https://orcid.org/0000-0003-3190-0557
authorships[2].author.display_name	Keith Rush
authorships[2].author_position	middle
authorships[2].raw_author_name	Rush, Keith
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5114021051
authorships[3].author.orcid
authorships[3].author.display_name	Satyen Kale
authorships[3].author_position	middle
authorships[3].raw_author_name	Kale, Satyen
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5034877995
authorships[4].author.orcid	https://orcid.org/0000-0001-8997-874X
authorships[4].author.display_name	Zachary Charles
authorships[4].author_position	middle
authorships[4].raw_author_name	Charles, Zachary
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5017433837
authorships[5].author.orcid	https://orcid.org/0000-0001-8158-3997
authorships[5].author.display_name	Zachary Garrett
authorships[5].author_position	middle
authorships[5].raw_author_name	Garrett, Zachary
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5116094347
authorships[6].author.orcid
authorships[6].author.display_name	Gabriel Teston
authorships[6].author_position	middle
authorships[6].raw_author_name	Teston, Gabriel
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5116094348
authorships[7].author.orcid
authorships[7].author.display_name	Dave Lacey
authorships[7].author_position	middle
authorships[7].raw_author_name	Lacey, Dave
authorships[7].is_corresponding	False
authorships[8].author.id	https://openalex.org/A5116094349
authorships[8].author.orcid
authorships[8].author.display_name	Ross McIlroy
authorships[8].author_position	middle
authorships[8].raw_author_name	McIlroy, Ross
authorships[8].is_corresponding	False
authorships[9].author.id	https://openalex.org/A5033886406
authorships[9].author.orcid	https://orcid.org/0000-0002-5042-0201
authorships[9].author.display_name	Jiajun Shen
authorships[9].author_position	middle
authorships[9].raw_author_name	Shen, Jiajun
authorships[9].is_corresponding	False
authorships[10].author.id	https://openalex.org/A5057217987
authorships[10].author.orcid
authorships[10].author.display_name	Alexandre Ramé
authorships[10].author_position	middle
authorships[10].raw_author_name	Ramé, Alexandre
authorships[10].is_corresponding	False
authorships[11].author.id	https://openalex.org/A5053145694
authorships[11].author.orcid
authorships[11].author.display_name	Arthur Szlam
authorships[11].author_position	middle
authorships[11].raw_author_name	Szlam, Arthur
authorships[11].is_corresponding	False
authorships[12].author.id	https://openalex.org/A5111465122
authorships[12].author.orcid
authorships[12].author.display_name	Marc’Aurelio Ranzato
authorships[12].author_position	middle
authorships[12].raw_author_name	Ranzato, Marc'Aurelio
authorships[12].is_corresponding	False
authorships[13].author.id	https://openalex.org/A5116094345
authorships[13].author.orcid
authorships[13].author.display_name	Paul Barham
authorships[13].author_position	last
authorships[13].raw_author_name	Barham, Paul
authorships[13].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2501.18512
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Streaming DiLoCo with overlapping communication: Towards a Distributed Free Lunch
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T12720
primary_topic.field.id	https://openalex.org/fields/33
primary_topic.field.display_name	Social Sciences
primary_topic.score	0.4020000100135803
primary_topic.domain.id	https://openalex.org/domains/2
primary_topic.domain.display_name	Social Sciences
primary_topic.subfield.id	https://openalex.org/subfields/3312
primary_topic.subfield.display_name	Sociology and Political Science
primary_topic.display_name	Multimedia Communication and Technology
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2501.18512
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2501.18512
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2501.18512
primary_location.id	pmh:oai:arXiv.org:2501.18512
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2501.18512
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2501.18512
publication_date	2025-01-30
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	10
abstract_inverted_index.By	181
abstract_inverted_index.In	125
abstract_inverted_index.as	111, 113, 201
abstract_inverted_index.at	29, 146
abstract_inverted_index.be	27, 40, 68, 120
abstract_inverted_index.by	89, 173, 207
abstract_inverted_index.in	80, 99, 131, 141
abstract_inverted_index.is	6
abstract_inverted_index.of	1, 13, 53, 139, 194, 210
abstract_inverted_index.to	15, 26, 39, 47, 119, 157
abstract_inverted_index.we	128, 135, 154, 168, 186, 190
abstract_inverted_index.all	36, 117, 123, 145
abstract_inverted_index.and	22, 31, 197
abstract_inverted_index.but	203
abstract_inverted_index.can	67, 85, 191
abstract_inverted_index.the	49, 107, 114, 170
abstract_inverted_index.two	208
abstract_inverted_index.This	79
abstract_inverted_index.data	171
abstract_inverted_index.each	30
abstract_inverted_index.have	61
abstract_inverted_index.high	51
abstract_inverted_index.into	70
abstract_inverted_index.like	59
abstract_inverted_index.need	25, 38
abstract_inverted_index.only	76, 137
abstract_inverted_index.peak	109, 151
abstract_inverted_index.same	108
abstract_inverted_index.show	187
abstract_inverted_index.such	63
abstract_inverted_index.than	144
abstract_inverted_index.that	83, 189
abstract_inverted_index.this	126
abstract_inverted_index.turn	81
abstract_inverted_index.wall	164
abstract_inverted_index.Since	19
abstract_inverted_index.allow	155
abstract_inverted_index.being	87
abstract_inverted_index.bits.	55
abstract_inverted_index.clock	165
abstract_inverted_index.every	32
abstract_inverted_index.large	2, 11
abstract_inverted_index.links	46, 93
abstract_inverted_index.lower	90
abstract_inverted_index.means	82
abstract_inverted_index.occur	77
abstract_inverted_index.once,	147
abstract_inverted_index.reach	198
abstract_inverted_index.step,	35
abstract_inverted_index.still	105
abstract_inverted_index.these	100, 184
abstract_inverted_index.three	132
abstract_inverted_index.time.	18, 166
abstract_inverted_index.using	42
abstract_inverted_index.ways.	133
abstract_inverted_index.where	72
abstract_inverted_index.which	148, 162, 175
abstract_inverted_index.while	160
abstract_inverted_index.(LLMs)	5
abstract_inverted_index.DiLoCo	60, 130
abstract_inverted_index.First,	134
abstract_inverted_index.Third,	167
abstract_inverted_index.across	9, 103, 122, 179
abstract_inverted_index.afford	86
abstract_inverted_index.models	4
abstract_inverted_index.number	12
abstract_inverted_index.orders	209
abstract_inverted_index.paper,	127
abstract_inverted_index.rather	143
abstract_inverted_index.reduce	16
abstract_inverted_index.single	33
abstract_inverted_index.states	21
abstract_inverted_index.volume	52
abstract_inverted_index.Second,	153
abstract_inverted_index.before,	112, 202
abstract_inverted_index.between	74
abstract_inverted_index.devices	37
abstract_inverted_index.further	176
abstract_inverted_index.greatly	149
abstract_inverted_index.grouped	69
abstract_inverted_index.improve	129
abstract_inverted_index.quality	200
abstract_inverted_index.reduces	150, 177
abstract_inverted_index.relaxed	62
abstract_inverted_index.require	116
abstract_inverted_index.similar	199
abstract_inverted_index.subsets	138
abstract_inverted_index.support	48
abstract_inverted_index.without	94
abstract_inverted_index.workers	75, 84, 104, 156
abstract_inverted_index.However,	98
abstract_inverted_index.Training	0
abstract_inverted_index.continue	158
abstract_inverted_index.gradient	34
abstract_inverted_index.internal	20
abstract_inverted_index.language	3
abstract_inverted_index.learning	96
abstract_inverted_index.methods,	101
abstract_inverted_index.properly	182
abstract_inverted_index.quality.	97
abstract_inverted_index.quantize	169
abstract_inverted_index.reducing	204
abstract_inverted_index.required	50, 205
abstract_inverted_index.requires	106
abstract_inverted_index.training	17, 159, 193
abstract_inverted_index.workers,	174
abstract_inverted_index.workers.	124, 180
abstract_inverted_index.Recently,	56
abstract_inverted_index.affecting	95
abstract_inverted_index.bandwidth	91, 110, 178, 206
abstract_inverted_index.combining	183
abstract_inverted_index.connected	88
abstract_inverted_index.decreases	163
abstract_inverted_index.exchanged	28, 54, 121, 172
abstract_inverted_index.gradients	24
abstract_inverted_index.parameter	23
abstract_inverted_index.sequence,	142
abstract_inverted_index.typically	7
abstract_inverted_index.algorithms	58
abstract_inverted_index.bandwidth.	152
abstract_inverted_index.co-located	41
abstract_inverted_index.distribute	192
abstract_inverted_index.magnitude.	211
abstract_inverted_index.parameters	118, 140, 196
abstract_inverted_index.co-location	64
abstract_inverted_index.constraint:	65
abstract_inverted_index.distributed	8, 57
abstract_inverted_index.low-latency	43
abstract_inverted_index.synchronize	136
abstract_inverted_index.``workers'',	71
abstract_inverted_index.accelerators	14, 66
abstract_inverted_index.billion-scale	195
abstract_inverted_index.communication	45, 92, 102
abstract_inverted_index.infrequently.	78
abstract_inverted_index.experimentally	188
abstract_inverted_index.high-bandwidth	44
abstract_inverted_index.modifications,	185
abstract_inverted_index.synchronizing,	161
abstract_inverted_index.synchronizations	73, 115
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	14
citation_normalized_percentile