Checkmate: Zero-Overhead Model Checkpointing via Network Gradient Replication Article Swipe

PDF

Ankit Bhardwaj , Weiyang Wang , Manya Ghobadi ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2507.13522

This paper presents Checkmate, a system that enables per-iteration checkpointing in DNN training without any training slowdown. The traditional approach to checkpointing requires a pause in training to copy model states to a separate location, allowing the state to be restored in the event of failure. This approach fundamentally has a tradeoff between the frequency of checkpoints and the cost of a failure. We avoid this tradeoff; our key insight is that in data-parallel training, all information necessary to create a checkpoint already exists in the network as gradients. Our core contribution is a new multicast abstraction that simultaneously delivers gradients to a separate CPU-based shadow cluster. The shadow maintains a checkpoint by applying those gradients to a copy of the model. Our evaluation shows that Checkmate performs per-iteration checkpointing with training throughput comparable to an ideal no-checkpoint baseline. Checkmate achieves 5 to 34.5x more frequent checkpointing compared to state-of-the-art checkpointing systems, resulting in 80% to 97.1% reduction in repeated work per failure. At the same checkpointing frequency, Checkmate delivers 1.3x to 6.5x throughput compared to other systems.

Related Topics

Role Model (Singer)

Tesla Model Y

Dominique Jackson (Model)

Smith & Wesson Model 29

Business Process Model And Notation

Kara Young (Model)

Technology Acceptance Model

Colt Model 1903 Pocket Hammerless

Boeing-Stearman Model 75

Debye Model

J. Alexander (Model)

Smith & Wesson Model 686

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2507.13522
PDF: https://arxiv.org/pdf/2507.13522
OA Status: green
OpenAlex ID: https://openalex.org/W4416833434

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4416833434

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2507.13522

Digital Object Identifier
Title: Checkmate: Zero-Overhead Model Checkpointing via Network Gradient Replication

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-07-17

Full publication date if available
Authors: Ankit Bhardwaj, Weiyang Wang, Manya Ghobadi

List of authors in order
Landing page: https://arxiv.org/abs/2507.13522

Publisher landing page
PDF URL: https://arxiv.org/pdf/2507.13522

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2507.13522

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4416833434
doi	https://doi.org/10.48550/arxiv.2507.13522
ids.doi	https://doi.org/10.48550/arxiv.2507.13522
ids.openalex	https://openalex.org/W4416833434
fwci
type	preprint
title	Checkmate: Zero-Overhead Model Checkpointing via Network Gradient Replication
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2507.13522
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2507.13522
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2507.13522
locations[1].id	doi:10.48550/arxiv.2507.13522
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2507.13522
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5056797683
authorships[0].author.orcid	https://orcid.org/0000-0002-5094-9711
authorships[0].author.display_name	Ankit Bhardwaj
authorships[0].author_position	first
authorships[0].raw_author_name	Bhardwaj, Ankit
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5042429449
authorships[1].author.orcid	https://orcid.org/0000-0001-9036-8543
authorships[1].author.display_name	Weiyang Wang
authorships[1].author_position	middle
authorships[1].raw_author_name	Wang, Weiyang
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5058729992
authorships[2].author.orcid	https://orcid.org/0000-0002-4095-1519
authorships[2].author.display_name	Manya Ghobadi
authorships[2].author_position	middle
authorships[2].raw_author_name	Ghobadi, Manya
authorships[2].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2507.13522
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Checkmate: Zero-Overhead Model Checkpointing via Network Gradient Replication
has_fulltext	False
is_retracted	False
updated_date	2025-12-01T01:23:19.965064
primary_topic
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2507.13522
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2507.13522
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2507.13522
primary_location.id	pmh:oai:arXiv.org:2507.13522
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2507.13522
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2507.13522
publication_date	2025-07-17
publication_year	2025
referenced_works_count	0
abstract_inverted_index.5	141
abstract_inverted_index.a	4, 23, 32, 50, 61, 80, 93, 102, 110, 117
abstract_inverted_index.At	163
abstract_inverted_index.We	63
abstract_inverted_index.an	135
abstract_inverted_index.as	87
abstract_inverted_index.be	39
abstract_inverted_index.by	112
abstract_inverted_index.in	10, 25, 41, 72, 84, 153, 158
abstract_inverted_index.is	70, 92
abstract_inverted_index.of	44, 55, 60, 119
abstract_inverted_index.to	20, 27, 31, 38, 78, 101, 116, 134, 142, 148, 155, 171, 175
abstract_inverted_index.80%	154
abstract_inverted_index.DNN	11
abstract_inverted_index.Our	89, 122
abstract_inverted_index.The	17, 107
abstract_inverted_index.all	75
abstract_inverted_index.and	57
abstract_inverted_index.any	14
abstract_inverted_index.has	49
abstract_inverted_index.key	68
abstract_inverted_index.new	94
abstract_inverted_index.our	67
abstract_inverted_index.per	161
abstract_inverted_index.the	36, 42, 53, 58, 85, 120, 164
abstract_inverted_index.1.3x	170
abstract_inverted_index.6.5x	172
abstract_inverted_index.This	0, 46
abstract_inverted_index.copy	28, 118
abstract_inverted_index.core	90
abstract_inverted_index.cost	59
abstract_inverted_index.more	144
abstract_inverted_index.same	165
abstract_inverted_index.that	6, 71, 97, 125
abstract_inverted_index.this	65
abstract_inverted_index.with	130
abstract_inverted_index.work	160
abstract_inverted_index.34.5x	143
abstract_inverted_index.97.1%	156
abstract_inverted_index.avoid	64
abstract_inverted_index.event	43
abstract_inverted_index.ideal	136
abstract_inverted_index.model	29
abstract_inverted_index.other	176
abstract_inverted_index.paper	1
abstract_inverted_index.pause	24
abstract_inverted_index.shows	124
abstract_inverted_index.state	37
abstract_inverted_index.those	114
abstract_inverted_index.create	79
abstract_inverted_index.exists	83
abstract_inverted_index.model.	121
abstract_inverted_index.shadow	105, 108
abstract_inverted_index.states	30
abstract_inverted_index.system	5
abstract_inverted_index.already	82
abstract_inverted_index.between	52
abstract_inverted_index.enables	7
abstract_inverted_index.insight	69
abstract_inverted_index.network	86
abstract_inverted_index.without	13
abstract_inverted_index.achieves	140
abstract_inverted_index.allowing	35
abstract_inverted_index.applying	113
abstract_inverted_index.approach	19, 47
abstract_inverted_index.cluster.	106
abstract_inverted_index.compared	147, 174
abstract_inverted_index.delivers	99, 169
abstract_inverted_index.failure.	45, 62, 162
abstract_inverted_index.frequent	145
abstract_inverted_index.performs	127
abstract_inverted_index.presents	2
abstract_inverted_index.repeated	159
abstract_inverted_index.requires	22
abstract_inverted_index.restored	40
abstract_inverted_index.separate	33, 103
abstract_inverted_index.systems,	151
abstract_inverted_index.systems.	177
abstract_inverted_index.tradeoff	51
abstract_inverted_index.training	12, 15, 26, 131
abstract_inverted_index.CPU-based	104
abstract_inverted_index.Checkmate	126, 139, 168
abstract_inverted_index.baseline.	138
abstract_inverted_index.frequency	54
abstract_inverted_index.gradients	100, 115
abstract_inverted_index.location,	34
abstract_inverted_index.maintains	109
abstract_inverted_index.multicast	95
abstract_inverted_index.necessary	77
abstract_inverted_index.reduction	157
abstract_inverted_index.resulting	152
abstract_inverted_index.slowdown.	16
abstract_inverted_index.tradeoff;	66
abstract_inverted_index.training,	74
abstract_inverted_index.Checkmate,	3
abstract_inverted_index.checkpoint	81, 111
abstract_inverted_index.comparable	133
abstract_inverted_index.evaluation	123
abstract_inverted_index.frequency,	167
abstract_inverted_index.gradients.	88
abstract_inverted_index.throughput	132, 173
abstract_inverted_index.abstraction	96
abstract_inverted_index.checkpoints	56
abstract_inverted_index.information	76
abstract_inverted_index.traditional	18
abstract_inverted_index.contribution	91
abstract_inverted_index.checkpointing	9, 21, 129, 146, 150, 166
abstract_inverted_index.data-parallel	73
abstract_inverted_index.fundamentally	48
abstract_inverted_index.no-checkpoint	137
abstract_inverted_index.per-iteration	8, 128
abstract_inverted_index.simultaneously	98
abstract_inverted_index.state-of-the-art	149
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	3
citation_normalized_percentile