Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment? Article Swipe

MAKE IMPACT

Hannah Markgraf , Shrutika S. Sawant , Hanna Krasowski , Lukas Schäfer , Sébastien Gros , Matthias Althoff · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2509.12833

Projection-based safety filters, which modify unsafe actions by mapping them to the closest safe alternative, are widely used to enforce safety constraints in reinforcement learning (RL). Two integration strategies are commonly considered: Safe environment RL (SE-RL), where the safeguard is treated as part of the environment, and safe policy RL (SP-RL), where it is embedded within the policy through differentiable optimization layers. Despite their practical relevance in safety-critical settings, a formal understanding of their differences is lacking. In this work, we present a theoretical comparison of SE-RL and SP-RL. We identify a key distinction in how each approach is affected by action aliasing, a phenomenon in which multiple unsafe actions are projected to the same safe action, causing information loss in the policy gradients. In SE-RL, this effect is implicitly approximated by the critic, while in SP-RL, it manifests directly as rank-deficient Jacobians during backpropagation through the safeguard. Our contributions are threefold: (i) a unified formalization of SE-RL and SP-RL in the context of actor-critic algorithms, (ii) a theoretical analysis of their respective policy gradient estimates, highlighting the role of action aliasing, and (iii) a comparative study of mitigation strategies, including a novel penalty-based improvement for SP-RL that aligns with established SE-RL practices. Empirical results support our theoretical predictions, showing that action aliasing is more detrimental for SP-RL than for SE-RL. However, with appropriate improvement strategies, SP-RL can match or outperform improved SE-RL across a range of environments. These findings provide actionable insights for choosing and refining projection-based safe RL methods based on task characteristics.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2509.12833

PDF

https://arxiv.org/pdf/2509.12833

OA Status

green

OpenAlex ID

https://openalex.org/W4415316914

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4415316914

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2509.12833

Digital Object Identifier

Title

Safe Reinforcement Learning using Action Projection: Safeguard the Policy or the Environment?

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-09-16

Full publication date if available

Authors

Hannah Markgraf, Shrutika S. Sawant, Hanna Krasowski, Lukas Schäfer, Sébastien Gros, Matthias Althoff

List of authors in order

Landing page

https://arxiv.org/abs/2509.12833

Publisher landing page

PDF URL

https://arxiv.org/pdf/2509.12833

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2509.12833

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload