A Cubic-regularized Policy Newton Algorithm for Reinforcement Learning Article Swipe

MAKE IMPACT

Mizhaan Prajit Maniyar , Akash Mondal , L. A. Prashanth , Shalabh Bhatnagar · YOU? · · 2023 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2304.10951

We consider the problem of control in the setting of reinforcement learning (RL), where model information is not available. Policy gradient algorithms are a popular solution approach for this problem and are usually shown to converge to a stationary point of the value function. In this paper, we propose two policy Newton algorithms that incorporate cubic regularization. Both algorithms employ the likelihood ratio method to form estimates of the gradient and Hessian of the value function using sample trajectories. The first algorithm requires an exact solution of the cubic regularized problem in each iteration, while the second algorithm employs an efficient gradient descent-based approximation to the cubic regularized problem. We establish convergence of our proposed algorithms to a second-order stationary point (SOSP) of the value function, which results in the avoidance of traps in the form of saddle points. In particular, the sample complexity of our algorithms to find an $ε$-SOSP is $O(ε^{-3.5})$, which is an improvement over the state-of-the-art sample complexity of $O(ε^{-4.5})$.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2304.10951

PDF

https://arxiv.org/pdf/2304.10951

OA Status

green

Related Works

10

OpenAlex ID

https://openalex.org/W4366835603

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4366835603

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2304.10951

Digital Object Identifier

Title

A Cubic-regularized Policy Newton Algorithm for Reinforcement Learning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2023

Year of publication

Publication date

2023-04-21

Full publication date if available

Authors

Mizhaan Prajit Maniyar, Akash Mondal, L. A. Prashanth, Shalabh Bhatnagar

List of authors in order

Landing page

https://arxiv.org/abs/2304.10951

Publisher landing page

PDF URL

https://arxiv.org/pdf/2304.10951

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2304.10951

Direct OA link when available

Concepts

Reinforcement learning, Hessian matrix, Stationary point, Algorithm, Saddle point, Regularization (linguistics), Descent direction, Function (biology), Gradient descent, Convergence (economics), Bellman equation, Mathematics, Newton's method, Computer science, Applied mathematics, Mathematical optimization, Artificial intelligence, Artificial neural network, Mathematical analysis, Physics, Biology, Economics, Economic growth, Geometry, Nonlinear system, Quantum mechanics, Evolutionary biology

Top concepts (fields/topics) attached by OpenAlex

Cited by

0

Total citation count in OpenAlex

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload