Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learning Article Swipe

MAKE IMPACT

Sajad Khodadadian , Pranay Sharma , Gauri Joshi , Siva Theja Maguluri · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2206.10185

Since reinforcement learning algorithms are notoriously data-intensive, the task of sampling observations from the environment is usually split across multiple agents. However, transferring these observations from the agents to a central location can be prohibitively expensive in terms of communication cost, and it can also compromise the privacy of each agent's local behavior policy. Federated reinforcement learning is a framework in which $N$ agents collaboratively learn a global model, without sharing their individual data and policies. This global model is the unique fixed point of the average of $N$ local operators, corresponding to the $N$ agents. Each agent maintains a local copy of the global model and updates it using locally sampled data. In this paper, we show that by careful collaboration of the agents in solving this joint fixed point problem, we can find the global model $N$ times faster, also known as linear speedup. We first propose a general framework for federated stochastic approximation with Markovian noise and heterogeneity, showing linear speedup in convergence. We then apply this framework to federated reinforcement learning algorithms, examining the convergence of federated on-policy TD, off-policy TD, and $Q$-learning.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2206.10185

PDF

https://arxiv.org/pdf/2206.10185

OA Status

green

Cited By

13

Related Works

10

OpenAlex ID

https://openalex.org/W4283313823

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4283313823

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2206.10185

Digital Object Identifier

Title

Federated Stochastic Approximation under Markov Noise and Heterogeneity: Applications in Reinforcement Learning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-06-21

Full publication date if available

Authors

Sajad Khodadadian, Pranay Sharma, Gauri Joshi, Siva Theja Maguluri

List of authors in order

Landing page

https://arxiv.org/abs/2206.10185

Publisher landing page

PDF URL

https://arxiv.org/pdf/2206.10185

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2206.10185

Direct OA link when available

Concepts

Speedup, Reinforcement learning, Computer science, Leverage (statistics), Convergence (economics), Markov decision process, Markov process, Sampling (signal processing), Task (project management), Artificial intelligence, Theoretical computer science, Machine learning, Parallel computing, Mathematics, Computer vision, Statistics, Economics, Filter (signal processing), Management, Economic growth

Top concepts (fields/topics) attached by OpenAlex

Cited by

13

Total citation count in OpenAlex

Citations by year (recent)

2025: 3, 2024: 4, 2023: 5, 2021: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload