Rethinking the Residual Distribution of Locate-then-Editing Methods in Model Editing Article Swipe

MAKE IMPACT

Xiaopeng Li , Shanwen Wang , Shasha Li , Shuqin Song , Bin Ji , Jun Ma , Jie Yu · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2502.03748

Model editing enables targeted updates to the knowledge of large language models (LLMs) with minimal retraining. Among existing approaches, locate-then-edit methods constitute a prominent paradigm: they first identify critical layers, then compute residuals at the final critical layer based on the target edit, and finally apply least-squares-based multi-layer updates via $\textbf{residual distribution}$. While empirically effective, we identify a counterintuitive failure mode: residual distribution, a core mechanism in these methods, introduces weight shift errors that undermine editing precision. Through theoretical and empirical analysis, we show that such errors increase with the distribution distance, batch size, and edit sequence length, ultimately leading to inaccurate or suboptimal edits. To address this, we propose the $\textbf{B}$oundary $\textbf{L}$ayer $\textbf{U}$pdat$\textbf{E (BLUE)}$ strategy to enhance locate-then-edit methods. Sequential batch editing experiments on three LLMs and two datasets demonstrate that BLUE not only delivers an average performance improvement of 35.59\%, significantly advancing the state of the art in model editing, but also enhances the preservation of LLMs' general capabilities. Our code is available at https://github.com/xpq-tech/BLUE.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2502.03748

PDF

https://arxiv.org/pdf/2502.03748

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4407244830

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4407244830

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2502.03748

Digital Object Identifier

Title

Rethinking the Residual Distribution of Locate-then-Editing Methods in Model Editing

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-02-06

Full publication date if available

Authors

Xiaopeng Li, Shanwen Wang, Shasha Li, Shuqin Song, Bin Ji, Jun Ma, Jie Yu

List of authors in order

Landing page

https://arxiv.org/abs/2502.03748

Publisher landing page

PDF URL

https://arxiv.org/pdf/2502.03748

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2502.03748

Direct OA link when available

Concepts

Residual, Image editing, Genome editing, Computer science, Distribution (mathematics), Artificial intelligence, Algorithm, Mathematics, CRISPR, Biology, Mathematical analysis, Gene, Biochemistry, Image (mathematics)

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload