Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning Article Swipe

MAKE IMPACT

Hongli Xu , Shilong Wang , Liusheng Huang · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2506.05977

Federated fine-tuning (FedFT) of large language models (LLMs) has emerged as a promising solution for adapting models to distributed data environments while ensuring data privacy. Existing FedFT methods predominantly utilize parameter-efficient fine-tuning (PEFT) techniques to reduce communication and computation overhead. However, they often fail to adequately address the catastrophic forgetting, a critical challenge arising from continual adaptation in distributed environments. The traditional centralized fine-tuning methods, which are not designed for the heterogeneous and privacy-constrained nature of federated environments, struggle to mitigate this issue effectively. Moreover, the challenge is further exacerbated by significant variation in data distributions and device capabilities across clients, which leads to intensified forgetting and degraded model generalization. To tackle these issues, we propose FedBE, a novel FedFT framework that integrates an adaptive transformer block expansion mechanism with a dynamic trainable-block allocation strategy. Specifically, FedBE expands trainable blocks within the model architecture, structurally separating newly learned task-specific knowledge from the original pre-trained representations. Additionally, FedBE dynamically assigns these trainable blocks to clients based on their data distributions and computational capabilities. This enables the framework to better accommodate heterogeneous federated environments and enhances the generalization ability of the model.Extensive experiments show that compared with existing federated fine-tuning methods, FedBE achieves 12-74% higher accuracy retention on general tasks after fine-tuning and a model convergence acceleration ratio of 1.9-3.1x without degrading the accuracy of downstream tasks.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2506.05977

PDF

https://arxiv.org/pdf/2506.05977

OA Status

green

OpenAlex ID

https://openalex.org/W4417097558

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4417097558

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2506.05977

Digital Object Identifier

Title

Mitigating Catastrophic Forgetting with Adaptive Transformer Block Expansion in Federated Fine-Tuning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-06-06

Full publication date if available

Authors

Hongli Xu, Shilong Wang, Liusheng Huang

List of authors in order

Landing page

https://arxiv.org/abs/2506.05977

Publisher landing page

PDF URL

https://arxiv.org/pdf/2506.05977

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2506.05977

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload