LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models Article Swipe

MAKE IMPACT

Qianyue Hao , Yong Sang Song , Qingmin Liao , Jian Yuan , Yijia Cao · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2505.15293

Policy exploration is critical in reinforcement learning (RL), where existing approaches include greedy, Gaussian process, etc. However, these approaches utilize preset stochastic processes and are indiscriminately applied in all kinds of RL tasks without considering task-specific features that influence policy exploration. Moreover, during RL training, the evolution of such stochastic processes is rigid, which typically only incorporates a decay in the variance, failing to adjust flexibly according to the agent's real-time learning status. Inspired by the analyzing and reasoning capability of large language models (LLMs), we design LLM-Explorer to adaptively generate task-specific exploration strategies with LLMs, enhancing the policy exploration in RL. In our design, we sample the learning trajectory of the agent during the RL training in a given task and prompt the LLM to analyze the agent's current policy learning status and then generate a probability distribution for future policy exploration. Updating the probability distribution periodically, we derive a stochastic process specialized for the particular task and dynamically adjusted to adapt to the learning process. Our design is a plug-in module compatible with various widely applied RL algorithms, including the DQN series, DDPG, TD3, and any possible variants developed based on them. Through extensive experiments on the Atari and MuJoCo benchmarks, we demonstrate LLM-Explorer's capability to enhance RL policy exploration, achieving an average performance improvement up to 37.27%. Our code is open-source at https://github.com/tsinghua-fib-lab/LLM-Explorer for reproducibility.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2505.15293

PDF

https://arxiv.org/pdf/2505.15293

OA Status

green

OpenAlex ID

https://openalex.org/W4415025005

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4415025005

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2505.15293

Digital Object Identifier

Title

LLM-Explorer: A Plug-in Reinforcement Learning Policy Exploration Enhancement Driven by Large Language Models

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-05-21

Full publication date if available

Authors

Qianyue Hao, Yong Sang Song, Qingmin Liao, Jian Yuan, Yijia Cao

List of authors in order

Landing page

https://arxiv.org/abs/2505.15293

Publisher landing page

PDF URL

https://arxiv.org/pdf/2505.15293

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2505.15293

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload