KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference Article Swipe

MAKE IMPACT

Ye Tian , Zihan Wang , Yajun Peng , Ao Yuan , Zhu Wang , Bairen Yi , Xin Liu , Yong Cui , Tong Yang · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2504.09936

Efficient inference of large language models (LLMs) is hindered by an ever-growing key-value (KV) cache, making KV cache compression a critical research direction. Traditional methods selectively evict less important KV cache entries, which leads to information loss and hallucinations. Recently, merging-based strategies have been explored to retain more information by merging KV pairs that would be discarded; however, these existing approaches inevitably introduce inconsistencies in attention distributions before and after merging, causing degraded generation quality. To overcome this challenge, we propose KeepKV, a novel adaptive KV cache merging method designed to preserve performance under strict memory constraints, achieving single-step lossless compression and providing error bounds for multi-step compression. KeepKV introduces the Electoral Votes mechanism that records merging history and adaptively adjusts attention scores. Moreover, it further leverages a novel Zero Inference-Perturbation Merging method, compensating for attention loss resulting from cache merging. Extensive experiments on various benchmarks and LLM architectures demonstrate that KeepKV substantially reduces memory usage while successfully retaining essential context information, achieving over 2x inference throughput improvement and maintaining superior generation quality even with only 10% KV cache budgets.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2504.09936

PDF

https://arxiv.org/pdf/2504.09936

OA Status

green

OpenAlex ID

https://openalex.org/W4415158961

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4415158961

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2504.09936

Digital Object Identifier

Title

KeepKV: Achieving Periodic Lossless KV Cache Compression for Efficient LLM Inference

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-04-14

Full publication date if available

Authors

Ye Tian, Zihan Wang, Yajun Peng, Ao Yuan, Zhu Wang, Bairen Yi, Xin Liu, Yong Cui, Tong Yang

List of authors in order

Landing page

https://arxiv.org/abs/2504.09936

Publisher landing page

PDF URL

https://arxiv.org/pdf/2504.09936

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2504.09936

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload