A Dual Perspective of Reinforcement Learning for Imposing Policy Constraints Article Swipe

MAKE IMPACT

Bram De Cooman , Johan A. K. Suykens · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2404.16468

Model-free reinforcement learning methods lack an inherent mechanism to impose behavioural constraints on the trained policies. Although certain extensions exist, they remain limited to specific types of constraints, such as value constraints with additional reward signals or visitation density constraints. In this work we unify these existing techniques and bridge the gap with classical optimization and control theory, using a generic primal-dual framework for value-based and actor-critic reinforcement learning methods. The obtained dual formulations turn out to be especially useful for imposing additional constraints on the learned policy, as an intrinsic relationship between such dual constraints (or regularization terms) and reward modifications in the primal is revealed. Furthermore, using this framework, we are able to introduce some novel types of constraints, allowing to impose bounds on the policy's action density or on costs associated with transitions between consecutive states and actions. From the adjusted primal-dual optimization problems, a practical algorithm is derived that supports various combinations of policy constraints that are automatically handled throughout training using trainable reward modifications. The proposed $\texttt{DualCRL}$ method is examined in more detail and evaluated under different (combinations of) constraints on two interpretable environments. The results highlight the efficacy of the method, which ultimately provides the designer of such systems with a versatile toolbox of possible policy constraints.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2404.16468

PDF

https://arxiv.org/pdf/2404.16468

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4395686977

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4395686977

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2404.16468

Digital Object Identifier

Title

A Dual Perspective of Reinforcement Learning for Imposing Policy Constraints

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-04-25

Full publication date if available

Authors

Bram De Cooman, Johan A. K. Suykens

List of authors in order

Landing page

https://arxiv.org/abs/2404.16468

Publisher landing page

PDF URL

https://arxiv.org/pdf/2404.16468

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2404.16468

Direct OA link when available

Concepts

Reinforcement learning, Perspective (graphical), Dual (grammatical number), Computer science, Reinforcement, Artificial intelligence, Psychology, Social psychology, Philosophy, Linguistics

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload