Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification,\n and Local Computations Article Swipe

MAKE IMPACT

Debraj Basu , Deepesh Data , Can Karakus , Suhas Diggavi · YOU? · · 2019 · Open Access · · DOI: https://doi.org/10.48550/arxiv.1906.02367

Communication bottleneck has been identified as a significant issue in\ndistributed optimization of large-scale learning models. Recently, several\napproaches to mitigate this problem have been proposed, including different\nforms of gradient compression or computing local models and mixing them\niteratively. In this paper, we propose \\emph{Qsparse-local-SGD} algorithm,\nwhich combines aggressive sparsification with quantization and local\ncomputation along with error compensation, by keeping track of the difference\nbetween the true and compressed gradients. We propose both synchronous and\nasynchronous implementations of \\emph{Qsparse-local-SGD}. We analyze\nconvergence for \\emph{Qsparse-local-SGD} in the \\emph{distributed} setting for\nsmooth non-convex and convex objective functions. We demonstrate that\n\\emph{Qsparse-local-SGD} converges at the same rate as vanilla distributed SGD\nfor many important classes of sparsifiers and quantizers. We use\n\\emph{Qsparse-local-SGD} to train ResNet-50 on ImageNet and show that it\nresults in significant savings over the state-of-the-art, in the number of bits\ntransmitted to reach target accuracy.\n

Type

preprint

Landing Page

http://arxiv.org/abs/1906.02367

PDF

https://arxiv.org/pdf/1906.02367

OA Status

green

Cited By

1

Related Works

10

OpenAlex ID

https://openalex.org/W4288335809

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4288335809

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.1906.02367

Digital Object Identifier

Title

Qsparse-local-SGD: Distributed SGD with Quantization, Sparsification,\n and Local Computations

Work title

Type

preprint

OpenAlex work type

Publication year

2019

Year of publication

Publication date

2019-06-05

Full publication date if available

Authors

Debraj Basu, Deepesh Data, Can Karakus, Suhas Diggavi

List of authors in order

Landing page

https://arxiv.org/abs/1906.02367

Publisher landing page

PDF URL

https://arxiv.org/pdf/1906.02367

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/1906.02367

Direct OA link when available

Concepts

Bottleneck, Computation, Quantization (signal processing), Computer science, Asynchronous communication, Rate of convergence, Regular polygon, Algorithm, Theoretical computer science, Mathematical optimization, Mathematics, Key (lock), Geometry, Embedded system, Computer network, Computer security

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2023: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload