InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference Article Swipe

MAKE IMPACT

Xiurui Pan , Erzhong Li , Qiao Li , Shengwen Liang , Yizhou Shan , Ke Zhou , Yingwei Luo , Xiaolin Wang , Jie Zhang · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2409.04992

The widespread of Large Language Models (LLMs) marks a significant milestone in generative AI. Nevertheless, the increasing context length and batch size in offline LLM inference escalate the memory requirement of the key-value (KV) cache, which imposes a huge burden on the GPU VRAM, especially for resource-constraint scenarios (e.g., edge computing and personal devices). Several cost-effective solutions leverage host memory or SSDs to reduce storage costs for offline inference scenarios and improve the throughput. Nevertheless, they suffer from significant performance penalties imposed by intensive KV cache accesses due to limited PCIe bandwidth. To address these issues, we propose InstInfer, a novel LLM inference system that offloads the most performance-critical computation (i.e., attention in decoding phase) and data (i.e., KV cache) parts to Computational Storage Drives (CSDs), which minimize the enormous KV transfer overheads. InstInfer designs a dedicated flash-aware in-storage attention engine with KV cache management mechanisms to exploit the high internal bandwidths of CSDs instead of being limited by the PCIe bandwidth. The optimized P2P transmission between GPU and CSDs further reduces data migration overheads. Experimental results demonstrate that for a 13B model using an NVIDIA A6000 GPU, InstInfer improves throughput for long-sequence inference by up to 11.1$\times$, compared to existing SSD-based solutions such as FlexGen.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2409.04992

PDF

https://arxiv.org/pdf/2409.04992

OA Status

green

Cited By

1

Related Works

10

OpenAlex ID

https://openalex.org/W4403617301

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4403617301

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2409.04992

Digital Object Identifier

Title

InstInfer: In-Storage Attention Offloading for Cost-Effective Long-Context LLM Inference

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-09-08

Full publication date if available

Authors

Xiurui Pan, Erzhong Li, Qiao Li, Shengwen Liang, Yizhou Shan, Ke Zhou, Yingwei Luo, Xiaolin Wang, Jie Zhang

List of authors in order

Landing page

https://arxiv.org/abs/2409.04992

Publisher landing page

PDF URL

https://arxiv.org/pdf/2409.04992

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2409.04992

Direct OA link when available

Concepts

Inference, Context (archaeology), Computer science, Human–computer interaction, Artificial intelligence, Biology, Paleontology

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2025: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload