Utilize the Flow before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuning Article Swipe

MAKE IMPACT

Rupeng Zhu , Zhiyuan Ma , Jiang Wu , Junyuan Gao , Jiaqi Wang , Dahua Lin , Changming He · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2410.06913

Refusal-Aware Instruction Tuning (RAIT) enables Large Language Models (LLMs) to refuse to answer unknown questions. By modifying responses of unknown questions in the training data to refusal responses such as "I don't know", RAIT enhances the reliability of LLMs and reduces their hallucination. Generally, RAIT modifies training samples based on the correctness of the initial LLM's response. However, this crude approach can cause LLMs to excessively refuse answering questions they could have correctly answered, the problem we call over-refusal. In this paper, we explore two primary causes of over-refusal: Static conflict occurs when similar samples within the LLM's feature space receive differing supervision signals (original vs. modified "I don't know"). Dynamic conflict arises as the LLM's evolving knowledge during SFT enables it to answer previously unanswerable questions, but the now-answerable training samples still retain the original "I don't know" supervision signals from the initial LLM state, leading to inconsistencies. These conflicts cause the trained LLM to misclassify known questions as unknown, resulting in over-refusal. To address this issue, we introduce Certainty Represented Knowledge Flow for Refusal-Aware Instructions Tuning (CRaFT). CRaFT centers on two main contributions: First, we additionally incorporate response certainty to selectively filter and modify data, reducing static conflicts. Second, we implement preliminary rehearsal training to characterize changes in the LLM's knowledge state, which helps mitigate dynamic conflicts during the fine-tuning process. We conducted extensive experiments on open-ended question answering and multiple-choice question task. Experiment results show that CRaFT can improve LLM's overall performance during the RAIT process. Code and data will be released at https://github.com/opendatalab/CRaFT .

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2410.06913

PDF

https://arxiv.org/pdf/2410.06913

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4403345587

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4403345587

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2410.06913

Digital Object Identifier

Title

Utilize the Flow before Stepping into the Same River Twice: Certainty Represented Knowledge Flow for Refusal-Aware Instruction Tuning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-10-09

Full publication date if available

Authors

Rupeng Zhu, Zhiyuan Ma, Jiang Wu, Junyuan Gao, Jiaqi Wang, Dahua Lin, Changming He

List of authors in order

Landing page

https://arxiv.org/abs/2410.06913

Publisher landing page

PDF URL

https://arxiv.org/pdf/2410.06913

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2410.06913

Direct OA link when available

Concepts

Certainty, Flow (mathematics), Knowledge flow, Computer science, Psychology, Mathematics, Epistemology, Knowledge management, Philosophy, Geometry

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload