Highway Reinforcement Learning Article Swipe

MAKE IMPACT

Yuhui Wang , Miroslav Štrupl , Francesco Faccio , Qingyuan Wu , Haozhe Liu , Michał Grudzień , Xiaoyang Tan , Jürgen Schmidhuber · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2405.18289

Learning from multi-step off-policy data collected by a set of policies is a core problem of reinforcement learning (RL). Approaches based on importance sampling (IS) often suffer from large variances due to products of IS ratios. Typical IS-free methods, such as $n$-step Q-learning, look ahead for $n$ time steps along the trajectory of actions (where $n$ is called the lookahead depth) and utilize off-policy data directly without any additional adjustment. They work well for proper choices of $n$. We show, however, that such IS-free methods underestimate the optimal value function (VF), especially for large $n$, restricting their capacity to efficiently utilize information from distant future time steps. To overcome this problem, we introduce a novel, IS-free, multi-step off-policy method that avoids the underestimation issue and converges to the optimal VF. At its core lies a simple but non-trivial \emph{highway gate}, which controls the information flow from the distant future by comparing it to a threshold. The highway gate guarantees convergence to the optimal VF for arbitrary $n$ and arbitrary behavioral policies. It gives rise to a novel family of off-policy RL algorithms that safely learn even when $n$ is very large, facilitating rapid credit assignment from the far future to the past. On tasks with greatly delayed rewards, including video games where the reward is given only at the end of the game, our new methods outperform many existing multi-step off-policy algorithms.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2405.18289

PDF

https://arxiv.org/pdf/2405.18289

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4399152062

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4399152062

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2405.18289

Digital Object Identifier

Title

Highway Reinforcement Learning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-05-28

Full publication date if available

Authors

Yuhui Wang, Miroslav Štrupl, Francesco Faccio, Qingyuan Wu, Haozhe Liu, Michał Grudzień, Xiaoyang Tan, Jürgen Schmidhuber

List of authors in order

Landing page

https://arxiv.org/abs/2405.18289

Publisher landing page

PDF URL

https://arxiv.org/pdf/2405.18289

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2405.18289

Direct OA link when available

Concepts

Reinforcement, Reinforcement learning, Psychology, Computer science, Artificial intelligence, Social psychology

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload