One Size Does Not Fit All: A Distribution-Aware Sparsification for More Precise Model Merging Article Swipe

MAKE IMPACT

Yingfeng Luo , D. X. Lin , Junxin Wang , Ziqiang Xu , Kangkang Chang , Tong Zheng , Bei Li , A. Ma , Tong Xiao , Zhengtao Yu , Jingbo Zhu · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2508.06163

Model merging has emerged as a compelling data-free paradigm for multi-task learning, enabling the fusion of multiple fine-tuned models into a single, powerful entity. A key technique in merging methods is sparsification, which prunes redundant parameters from task vectors to mitigate interference. However, prevailing approaches employ a ``one-size-fits-all'' strategy, applying a uniform sparsity ratio that overlooks the inherent structural and statistical heterogeneity of model parameters. This often leads to a suboptimal trade-off, where critical parameters are inadvertently pruned while less useful ones are retained. To address this limitation, we introduce \textbf{TADrop} (\textbf{T}ensor-wise \textbf{A}daptive \textbf{Drop}), an adaptive sparsification strategy that respects this heterogeneity. Instead of a global ratio, TADrop assigns a tailored sparsity level to each parameter tensor based on its distributional properties. The core intuition is that tensors with denser, more redundant distributions can be pruned aggressively, while sparser, more critical ones are preserved. As a simple and plug-and-play module, we validate TADrop by integrating it with foundational, classic, and SOTA merging methods. Extensive experiments across diverse tasks (vision, language, and multimodal) and models (ViT, BEiT) demonstrate that TADrop consistently and significantly boosts their performance. For instance, when enhancing a leading merging method, it achieves an average performance gain of 2.0\% across 8 ViT-B/32 tasks. TADrop provides a more effective way to mitigate parameter interference by tailoring sparsification to the model's structure, offering a new baseline for high-performance model merging.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2508.06163

PDF

https://arxiv.org/pdf/2508.06163

OA Status

green

OpenAlex ID

https://openalex.org/W4415193068

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4415193068

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2508.06163

Digital Object Identifier

Title

One Size Does Not Fit All: A Distribution-Aware Sparsification for More Precise Model Merging

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-08-08

Full publication date if available

Authors

Yingfeng Luo, D. X. Lin, Junxin Wang, Ziqiang Xu, Kangkang Chang, Tong Zheng, Bei Li, A. Ma, Tong Xiao, Zhengtao Yu, Jingbo Zhu

List of authors in order

Landing page

https://arxiv.org/abs/2508.06163

Publisher landing page

PDF URL

https://arxiv.org/pdf/2508.06163

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2508.06163

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload