Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning Article Swipe

MAKE IMPACT

Calarina Muslimani , Matthew E. Taylor · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2405.00746

To create useful reinforcement learning (RL) agents, step zero is to design a suitable reward function that captures the nuances of the task. However, reward engineering can be a difficult and time-consuming process. Instead, human-in-the-loop RL methods hold the promise of learning reward functions from human feedback. Despite recent successes, many of the human-in-the-loop RL methods still require numerous human interactions to learn successful reward functions. To improve the feedback efficiency of human-in-the-loop RL methods (i.e., require less human interaction), this paper introduces Sub-optimal Data Pre-training, SDP, an approach that leverages reward-free, sub-optimal data to improve scalar- and preference-based RL algorithms. In SDP, we start by pseudo-labeling all low-quality data with the minimum environment reward. Through this process, we obtain reward labels to pre-train our reward model without requiring human labeling or preferences. This pre-training phase provides the reward model a head start in learning, enabling it to recognize that low-quality transitions should be assigned low rewards. Through extensive experiments with both simulated and human teachers, we find that SDP can at least meet, but often significantly improve, state of the art human-in-the-loop RL performance across a variety of simulated robotic tasks.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2405.00746

PDF

https://arxiv.org/pdf/2405.00746

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4396821559

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4396821559

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2405.00746

Digital Object Identifier

Title

Leveraging Sub-Optimal Data for Human-in-the-Loop Reinforcement Learning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-04-30

Full publication date if available

Authors

Calarina Muslimani, Matthew E. Taylor

List of authors in order

Landing page

https://arxiv.org/abs/2405.00746

Publisher landing page

PDF URL

https://arxiv.org/pdf/2405.00746

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2405.00746

Direct OA link when available

Concepts

Reinforcement learning, Loop (graph theory), Human-in-the-loop, Computer science, Reinforcement, Artificial intelligence, Psychology, Mathematics, Social psychology, Combinatorics

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload