Risk-Sensitive Reinforcement Learning via Policy Gradient Search Article Swipe

MAKE IMPACT

L. A. Prashanth , Michael C. Fu · YOU? · · 2018 · Open Access · · DOI: https://doi.org/10.48550/arxiv.1810.09126

The objective in a traditional reinforcement learning (RL) problem is to find a policy that optimizes the expected value of a performance metric such as the infinite-horizon cumulative discounted or long-run average cost/reward. In practice, optimizing the expected value alone may not be satisfactory, in that it may be desirable to incorporate the notion of risk into the optimization problem formulation, either in the objective or as a constraint. Various risk measures have been proposed in the literature, e.g., exponential utility, variance, percentile performance, chance constraints, value at risk (quantile), conditional value-at-risk, prospect theory and its later enhancement, cumulative prospect theory. In this book, we consider risk-sensitive RL in two settings: one where the goal is to find a policy that optimizes the usual expected value objective while ensuring that a risk constraint is satisfied, and the other where the risk measure is the objective. We survey some of the recent work in this area specifically where policy gradient search is the solution approach. In the first risk-sensitive RL setting, we cover popular risk measures based on variance, conditional value-at-risk, and chance constraints, and present a template for policy gradient-based risk-sensitive RL algorithms using a Lagrangian formulation. For the setting where risk is incorporated directly into the objective function, we consider an exponential utility formulation, cumulative prospect theory, and coherent risk measures. This non-exhaustive survey aims to give a flavor of the challenges involved in solving risk-sensitive RL problems using policy gradient methods, as well as outlining some potential future research directions.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/1810.09126

PDF

https://arxiv.org/pdf/1810.09126

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4308351611

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4308351611

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.1810.09126

Digital Object Identifier

Title

Risk-Sensitive Reinforcement Learning via Policy Gradient Search

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2018

Year of publication

Publication date

2018-10-22

Full publication date if available

Authors

L. A. Prashanth, Michael C. Fu

List of authors in order

Landing page

https://arxiv.org/abs/1810.09126

Publisher landing page

PDF URL

https://arxiv.org/pdf/1810.09126

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/1810.09126

Direct OA link when available

Concepts

Cumulative prospect theory, Reinforcement learning, Risk measure, Expected shortfall, Value at risk, Metric (unit), Variance (accounting), Mathematical optimization, Exponential utility, Computer science, Exponential function, Risk management, Constraint (computer-aided design), Econometrics, Mathematics, Expected utility hypothesis, Statistics, Economics, Artificial intelligence, Finance, Accounting, Geometry, Mathematical analysis, Portfolio, Operations management

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload