Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees Article Swipe

MAKE IMPACT

Jue Wang , Binhang Yuan , Luka Rimanić , Yongjun He , Tri Dao , Beidi Chen , Christopher Ré , Ce Zhang · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2206.01299

Communication compression is a crucial technique for modern distributed learning systems to alleviate their communication bottlenecks over slower networks. Despite recent intensive studies of gradient compression for data parallel-style training, compressing the activations for models trained with pipeline parallelism is still an open problem. In this paper, we propose AC-SGD, a novel activation compression algorithm for communication-efficient pipeline parallelism training over slow networks. Different from previous efforts in activation compression, instead of compressing activation values directly, AC-SGD compresses the changes of the activations. This allows us to show, to the best of our knowledge for the first time, that one can still achieve $O(1/\sqrt{T})$ convergence rate for non-convex objectives under activation compression, without making assumptions on gradient unbiasedness that do not hold for deep learning models with non-linear activation functions.We then show that AC-SGD can be optimized and implemented efficiently, without additional end-to-end runtime overhead.We evaluated AC-SGD to fine-tune language models with up to 1.5 billion parameters, compressing activations to 2-4 bits.AC-SGD provides up to 4.3X end-to-end speed-up in slower networks, without sacrificing model quality. Moreover, we also show that AC-SGD can be combined with state-of-the-art gradient compression algorithms to enable "end-to-end communication compression: All communications between machines, including model gradients, forward activations, and backward gradients are compressed into lower precision.This provides up to 4.9X end-to-end speed-up, without sacrificing model quality.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2206.01299

PDF

https://arxiv.org/pdf/2206.01299

OA Status

green

Cited By

2

Related Works

10

OpenAlex ID

https://openalex.org/W4281945007

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4281945007

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2206.01299

Digital Object Identifier

Title

Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-06-02

Full publication date if available

Authors

Jue Wang, Binhang Yuan, Luka Rimanić, Yongjun He, Tri Dao, Beidi Chen, Christopher Ré, Ce Zhang

List of authors in order

Landing page

https://arxiv.org/abs/2206.01299

Publisher landing page

PDF URL

https://arxiv.org/pdf/2206.01299

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2206.01299

Direct OA link when available

Concepts

Computer science, Pipeline (software), Overhead (engineering), Compression (physics), Data compression, Speedup, Convergence (economics), End-to-end principle, Data compression ratio, Parallel computing, Algorithm, Computer engineering, Image compression, Artificial intelligence, Operating system, Economic growth, Economics, Programming language, Image (mathematics), Image processing, Composite material, Materials science

Top concepts (fields/topics) attached by OpenAlex

Cited by

2

Total citation count in OpenAlex

Citations by year (recent)

2024: 1, 2023: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload