Preference Poisoning Attacks on Reward Model Learning Article Swipe

MAKE IMPACT

Junlin Wu , Jiongxiao Wang , Chaowei Xiao , Chenguang Wang , Ning Zhang , Yevgeniy Vorobeychik · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2402.01920

Learning reward models from pairwise comparisons is a fundamental component in a number of domains, including autonomous control, conversational agents, and recommendation systems, as part of a broad goal of aligning automated decisions with user preferences. These approaches entail collecting preference information from people, with feedback often provided anonymously. Since preferences are subjective, there is no gold standard to compare against; yet, reliance of high-impact systems on preference learning creates a strong motivation for malicious actors to skew data collected in this fashion to their ends. We investigate the nature and extent of this vulnerability by considering an attacker who can flip a small subset of preference comparisons to either promote or demote a target outcome. We propose two classes of algorithmic approaches for these attacks: a gradient-based framework, and several variants of rank-by-distance methods. Next, we evaluate the efficacy of best attacks in both these classes in successfully achieving malicious goals on datasets from three domains: autonomous control, recommendation system, and textual prompt-response preference learning. We find that the best attacks are often highly successful, achieving in the most extreme case 100\% success rate with only 0.3\% of the data poisoned. However, \emph{which} attack is best can vary significantly across domains. In addition, we observe that the simpler and more scalable rank-by-distance approaches are often competitive with, and on occasion significantly outperform, gradient-based methods. Finally, we show that state-of-the-art defenses against other classes of poisoning attacks exhibit limited efficacy in our setting.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2402.01920

PDF

https://arxiv.org/pdf/2402.01920

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4391620947

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4391620947

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2402.01920

Digital Object Identifier

Title

Preference Poisoning Attacks on Reward Model Learning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-02-02

Full publication date if available

Authors

Junlin Wu, Jiongxiao Wang, Chaowei Xiao, Chenguang Wang, Ning Zhang, Yevgeniy Vorobeychik

List of authors in order

Landing page

https://arxiv.org/abs/2402.01920

Publisher landing page

PDF URL

https://arxiv.org/pdf/2402.01920

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2402.01920

Direct OA link when available

Concepts

Preference, Psychology, Computer science, Economics, Microeconomics

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload