Selecting Auxiliary Data via Neural Tangent Kernels for Low-Resource Domains Article Swipe

MAKE IMPACT

Ziqing Fan , Yaxin Du , Shuo Tang , Yu Wang · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2511.07380

Large language models (LLMs) have achieved remarkable success across widespread tasks, yet their application in low-resource domains remains a significant challenge due to data scarcity and the high risk of overfitting. While in-domain data is limited, there exist vast amounts of similar general-domain data, and our initial findings reveal that they could potentially serve as auxiliary supervision for domain enhancement. This observation leads us to our central research question: \textbf{\textit{how to effectively select the most valuable auxiliary data to maximize domain-specific performance}}, particularly when traditional methods are inapplicable due to a lack of large in-domain data pools or validation sets. To address this, we propose \textbf{NTK-Selector}, a principled and efficient framework for selecting general-domain auxiliary data to enhance domain-specific performance via neural tangent kernels (NTK). Our method tackles two challenges of directly applying NTK to LLMs, theoretical assumptions and prohibitive computational cost, by empirically demonstrating a stable NTK-like behavior in LLMs during LoRA fine-tuning and proposing a Jacobian-free approximation method. Extensive experiments across four low-resource domains (medical, financial, legal, and psychological) demonstrate that NTK-Selector consistently improves downstream performance. Specifically, fine-tuning on 1,000 in-domain samples alone only yielded +0.8 points for Llama3-8B-Instruct and +0.9 points for Qwen3-8B. In contrast, enriching with 9,000 auxiliary samples selected by NTK-Selector led to substantial \textbf{gains of +8.7 and +5.1 points}, which corresponds to a \textbf{10.9x and 5.7x improvement} over the domain-only setting.

Type

preprint

Landing Page

http://arxiv.org/abs/2511.07380

PDF

https://arxiv.org/pdf/2511.07380

OA Status

green

OpenAlex ID

https://openalex.org/W4416159260

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4416159260

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2511.07380

Digital Object Identifier

Title

Selecting Auxiliary Data via Neural Tangent Kernels for Low-Resource Domains

Work title

Type

preprint

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-11-10

Full publication date if available

Authors

Ziqing Fan, Yaxin Du, Shuo Tang, Yu Wang

List of authors in order

Landing page

https://arxiv.org/abs/2511.07380

Publisher landing page

PDF URL

https://arxiv.org/pdf/2511.07380

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2511.07380

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload