Efficient Long-context Language Model Training by Core Attention Disaggregation Article Swipe

MAKE IMPACT

Yonghao Zhuang , Jiajia Chen , Bo Pang , Yi Gu , Yibo Zhu , Yimin Jiang , Ion Stoica , Eric P. Xing , Hao Zhang · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2510.18121

We present core attention disaggregation (CAD), a technique that improves long-context large language model training by decoupling the core attention computation, softmax(QK^T)V, from the rest of the model and executing it on a separate pool of devices. In existing systems, core attention is colocated with other layers; at long context lengths, its quadratic compute growth compared to the near-linear growth of other components causes load imbalance and stragglers across data and pipeline parallel groups. CAD is enabled by two observations. First, core attention is stateless: it has no trainable parameters and only minimal transient data, so balancing reduces to scheduling compute-bound tasks. Second, it is composable: modern attention kernels retain high efficiency when processing fused batches of token-level shards with arbitrary lengths. CAD partitions core attention into token-level tasks and dispatches them to dedicated attention servers, which dynamically rebatch tasks to equalize compute without sacrificing kernel efficiency. We implement CAD in a system called DistCA, which uses a ping-pong execution scheme to fully overlap communication with computation and in-place execution on attention servers to reduce memory use. On 512 H200 GPUs and context lengths up to 512k tokens, DistCA improves end-to-end training throughput by up to 1.35x, eliminates data and pipeline parallel stragglers, and achieves near-perfect compute and memory balance.

Type

preprint

Landing Page

http://arxiv.org/abs/2510.18121

PDF

https://arxiv.org/pdf/2510.18121

OA Status

green

OpenAlex ID

https://openalex.org/W4416055065

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4416055065

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2510.18121

Digital Object Identifier

Title

Efficient Long-context Language Model Training by Core Attention Disaggregation

Work title

Type

preprint

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-10-20

Full publication date if available

Authors

Yonghao Zhuang, Jiajia Chen, Bo Pang, Yi Gu, Yibo Zhu, Yimin Jiang, Ion Stoica, Eric P. Xing, Hao Zhang

List of authors in order

Landing page

https://arxiv.org/abs/2510.18121

Publisher landing page

PDF URL

https://arxiv.org/pdf/2510.18121

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2510.18121

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload