Investigating the Properties of Neural Network Representations in Reinforcement Learning Article Swipe

MAKE IMPACT

Han Wang , Erfan Miahi , Martha White , Marlos C. Machado , Zaheer Abbas , Raksha Kumaraswamy , Vincent Liu , Adam White · YOU? · · 2022 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2203.15955

In this paper we investigate the properties of representations learned by deep reinforcement learning systems. Much of the early work on representations for reinforcement learning focused on designing fixed-basis architectures to achieve properties thought to be desirable, such as orthogonality and sparsity. In contrast, the idea behind deep reinforcement learning methods is that the agent designer should not encode representational properties, but rather that the data stream should determine the properties of the representation -- good representations emerge under appropriate training schemes. In this paper we bring these two perspectives together, empirically investigating the properties of representations that support transfer in reinforcement learning. We introduce and measure six representational properties over more than 25 thousand agent-task settings. We consider Deep Q-learning agents with different auxiliary losses in a pixel-based navigation environment, with source and transfer tasks corresponding to different goal locations. We develop a method to better understand why some representations work better for transfer, through a systematic approach varying task similarity and measuring and correlating representation properties with transfer performance. We demonstrate the generality of the methodology by investigating representations learned by a Rainbow agent that successfully transfer across games modes in Atari 2600.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2203.15955

PDF

https://arxiv.org/pdf/2203.15955

OA Status

green

Cited By

1

Related Works

10

OpenAlex ID

https://openalex.org/W4224862964

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4224862964

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2203.15955

Digital Object Identifier

Title

Investigating the Properties of Neural Network Representations in Reinforcement Learning

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2022

Year of publication

Publication date

2022-03-30

Full publication date if available

Authors

Han Wang, Erfan Miahi, Martha White, Marlos C. Machado, Zaheer Abbas, Raksha Kumaraswamy, Vincent Liu, Adam White

List of authors in order

Landing page

https://arxiv.org/abs/2203.15955

Publisher landing page

PDF URL

https://arxiv.org/pdf/2203.15955

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2203.15955

Direct OA link when available

Concepts

Reinforcement learning, Generality, Computer science, Orthogonality, Transfer of learning, Representation (politics), Task (project management), Artificial intelligence, ENCODE, Similarity (geometry), Contrast (vision), Reinforcement, Machine learning, Mathematics, Image (mathematics), Gene, Management, Chemistry, Law, Psychology, Biochemistry, Social psychology, Economics, Psychotherapist, Politics, Geometry, Political science

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2023: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload