Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption Article Swipe

PDF

Yiding Chen , Xuezhou Zhang , Qiaomin Xie , Xiaojin Zhu ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.1609/aaai.v38i10.29022

We study offline reinforcement learning (RL) with heavy-tailed reward distribution and data corruption: (i) Moving beyond subGaussian reward distribution, we allow the rewards to have infinite variances; (ii) We allow corruptions where an attacker can arbitrarily modify a small fraction of the rewards and transitions in the dataset. We first derive a sufficient optimality condition for generalized Pessimistic Value Iteration (PEVI), which allows various estimators with proper confidence bounds and can be applied to multiple learning settings. In order to handle the data corruption and heavy-tailed reward setting, we prove that the trimmed-mean estimation achieves the minimax optimal error rate for robust mean estimation under heavy-tailed distributions. In the PEVI algorithm, we plug in the trimmed mean estimation and the confidence bound to solve the robust offline RL problem. Standard analysis reveals that data corruption induces a bias term in the suboptimality gap, which gives the false impression that any data corruption prevents optimal policy learning. By using the optimality condition for the generalized PEVI, we show that as long as the bias term is less than the ``action gap'', the policy returned by PEVI achieves the optimal value given sufficient data.

Related Topics

Concepts

Language change Computer science Computer security Political science Art Literature

Metadata

Type: article
Language: en
Landing Page: https://doi.org/10.1609/aaai.v38i10.29022
PDF: https://ojs.aaai.org/index.php/AAAI/article/download/29022/29939
OA Status: diamond
References: 69
Related Works: 10
OpenAlex ID: https://openalex.org/W4393159790

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4393159790

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.1609/aaai.v38i10.29022

Digital Object Identifier
Title: Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption

Work title
Type: article

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-03-24

Full publication date if available
Authors: Yiding Chen, Xuezhou Zhang, Qiaomin Xie, Xiaojin Zhu

List of authors in order
Landing page: https://doi.org/10.1609/aaai.v38i10.29022

Publisher landing page
PDF URL: https://ojs.aaai.org/index.php/AAAI/article/download/29022/29939

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: diamond

Open access status per OpenAlex
OA URL: https://ojs.aaai.org/index.php/AAAI/article/download/29022/29939

Direct OA link when available
Concepts: Language change, Computer science, Computer security, Political science, Art, Literature

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
References (count): 69

Number of works referenced by this work
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4393159790
doi	https://doi.org/10.1609/aaai.v38i10.29022
ids.doi	https://doi.org/10.1609/aaai.v38i10.29022
ids.openalex	https://openalex.org/W4393159790
fwci	0.0
type	article
title	Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption
biblio.issue	10
biblio.volume	38
biblio.last_page	11424
biblio.first_page	11416
topics[0].id	https://openalex.org/T10558
topics[0].field.id	https://openalex.org/fields/22
topics[0].field.display_name	Engineering
topics[0].score	0.9642999768257141
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/2208
topics[0].subfield.display_name	Electrical and Electronic Engineering
topics[0].display_name	Advancements in Semiconductor Devices and Circuit Design
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C2780027415
concepts[0].level	2
concepts[0].score	0.6551860570907593
concepts[0].wikidata	https://www.wikidata.org/wiki/Q524648
concepts[0].display_name	Language change
concepts[1].id	https://openalex.org/C41008148
concepts[1].level	0
concepts[1].score	0.419445663690567
concepts[1].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[1].display_name	Computer science
concepts[2].id	https://openalex.org/C38652104
concepts[2].level	1
concepts[2].score	0.38792288303375244
concepts[2].wikidata	https://www.wikidata.org/wiki/Q3510521
concepts[2].display_name	Computer security
concepts[3].id	https://openalex.org/C17744445
concepts[3].level	0
concepts[3].score	0.34901946783065796
concepts[3].wikidata	https://www.wikidata.org/wiki/Q36442
concepts[3].display_name	Political science
concepts[4].id	https://openalex.org/C142362112
concepts[4].level	0
concepts[4].score	0.1325281262397766
concepts[4].wikidata	https://www.wikidata.org/wiki/Q735
concepts[4].display_name	Art
concepts[5].id	https://openalex.org/C124952713
concepts[5].level	1
concepts[5].score	0.049704283475875854
concepts[5].wikidata	https://www.wikidata.org/wiki/Q8242
concepts[5].display_name	Literature
keywords[0].id	https://openalex.org/keywords/language-change
keywords[0].score	0.6551860570907593
keywords[0].display_name	Language change
keywords[1].id	https://openalex.org/keywords/computer-science
keywords[1].score	0.419445663690567
keywords[1].display_name	Computer science
keywords[2].id	https://openalex.org/keywords/computer-security
keywords[2].score	0.38792288303375244
keywords[2].display_name	Computer security
keywords[3].id	https://openalex.org/keywords/political-science
keywords[3].score	0.34901946783065796
keywords[3].display_name	Political science
keywords[4].id	https://openalex.org/keywords/art
keywords[4].score	0.1325281262397766
keywords[4].display_name	Art
keywords[5].id	https://openalex.org/keywords/literature
keywords[5].score	0.049704283475875854
keywords[5].display_name	Literature
language	en
locations[0].id	doi:10.1609/aaai.v38i10.29022
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4210191458
locations[0].source.issn	2159-5399, 2374-3468
locations[0].source.type	conference
locations[0].source.is_oa	True
locations[0].source.issn_l	2159-5399
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	Proceedings of the AAAI Conference on Artificial Intelligence
locations[0].source.host_organization	https://openalex.org/P4310320058
locations[0].source.host_organization_name	Association for the Advancement of Artificial Intelligence
locations[0].source.host_organization_lineage	https://openalex.org/P4310320058
locations[0].source.host_organization_lineage_names	Association for the Advancement of Artificial Intelligence
locations[0].license
locations[0].pdf_url	https://ojs.aaai.org/index.php/AAAI/article/download/29022/29939
locations[0].version	publishedVersion
locations[0].raw_type	journal-article
locations[0].license_id
locations[0].is_accepted	True
locations[0].is_published	True
locations[0].raw_source_name	Proceedings of the AAAI Conference on Artificial Intelligence
locations[0].landing_page_url	https://doi.org/10.1609/aaai.v38i10.29022
indexed_in	crossref
authorships[0].author.id	https://openalex.org/A5006044524
authorships[0].author.orcid	https://orcid.org/0000-0003-1759-9112
authorships[0].author.display_name	Yiding Chen
authorships[0].countries	US
authorships[0].affiliations[0].institution_ids	https://openalex.org/I135310074
authorships[0].affiliations[0].raw_affiliation_string	University of Wisconsin-Madison
authorships[0].institutions[0].id	https://openalex.org/I135310074
authorships[0].institutions[0].ror	https://ror.org/01y2jtd41
authorships[0].institutions[0].type	education
authorships[0].institutions[0].lineage	https://openalex.org/I135310074
authorships[0].institutions[0].country_code	US
authorships[0].institutions[0].display_name	University of Wisconsin–Madison
authorships[0].author_position	first
authorships[0].raw_author_name	Yiding Chen
authorships[0].is_corresponding	False
authorships[0].raw_affiliation_strings	University of Wisconsin-Madison
authorships[1].author.id	https://openalex.org/A5022094334
authorships[1].author.orcid	https://orcid.org/0009-0006-7966-1013
authorships[1].author.display_name	Xuezhou Zhang
authorships[1].affiliations[0].raw_affiliation_string	Boston University
authorships[1].author_position	middle
authorships[1].raw_author_name	Xuezhou Zhang
authorships[1].is_corresponding	False
authorships[1].raw_affiliation_strings	Boston University
authorships[2].author.id	https://openalex.org/A5008882694
authorships[2].author.orcid	https://orcid.org/0000-0003-2834-6866
authorships[2].author.display_name	Qiaomin Xie
authorships[2].countries	US
authorships[2].affiliations[0].institution_ids	https://openalex.org/I135310074
authorships[2].affiliations[0].raw_affiliation_string	University of Wisconsin-Madison
authorships[2].institutions[0].id	https://openalex.org/I135310074
authorships[2].institutions[0].ror	https://ror.org/01y2jtd41
authorships[2].institutions[0].type	education
authorships[2].institutions[0].lineage	https://openalex.org/I135310074
authorships[2].institutions[0].country_code	US
authorships[2].institutions[0].display_name	University of Wisconsin–Madison
authorships[2].author_position	middle
authorships[2].raw_author_name	Qiaomin Xie
authorships[2].is_corresponding	False
authorships[2].raw_affiliation_strings	University of Wisconsin-Madison
authorships[3].author.id	https://openalex.org/A5103428074
authorships[3].author.orcid
authorships[3].author.display_name	Xiaojin Zhu
authorships[3].countries	US
authorships[3].affiliations[0].institution_ids	https://openalex.org/I135310074
authorships[3].affiliations[0].raw_affiliation_string	University of Wisconsin-Madison
authorships[3].institutions[0].id	https://openalex.org/I135310074
authorships[3].institutions[0].ror	https://ror.org/01y2jtd41
authorships[3].institutions[0].type	education
authorships[3].institutions[0].lineage	https://openalex.org/I135310074
authorships[3].institutions[0].country_code	US
authorships[3].institutions[0].display_name	University of Wisconsin–Madison
authorships[3].author_position	last
authorships[3].raw_author_name	Xiaojin Zhu
authorships[3].is_corresponding	False
authorships[3].raw_affiliation_strings	University of Wisconsin-Madison
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://ojs.aaai.org/index.php/AAAI/article/download/29022/29939
open_access.oa_status	diamond
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Exact Policy Recovery in Offline RL with Both Heavy-Tailed Rewards and Data Corruption
has_fulltext	True
is_retracted	False
updated_date	2025-11-06T03:46:38.306776
primary_topic.id	https://openalex.org/T10558
primary_topic.field.id	https://openalex.org/fields/22
primary_topic.field.display_name	Engineering
primary_topic.score	0.9642999768257141
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/2208
primary_topic.subfield.display_name	Electrical and Electronic Engineering
primary_topic.display_name	Advancements in Semiconductor Devices and Circuit Design
related_works	https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2382290278, https://openalex.org/W2478288626, https://openalex.org/W4391913857, https://openalex.org/W2350741829, https://openalex.org/W2530322880
cited_by_count	0
locations_count	1
best_oa_location.id	doi:10.1609/aaai.v38i10.29022
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4210191458
best_oa_location.source.issn	2159-5399, 2374-3468
best_oa_location.source.type	conference
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l	2159-5399
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	Proceedings of the AAAI Conference on Artificial Intelligence
best_oa_location.source.host_organization	https://openalex.org/P4310320058
best_oa_location.source.host_organization_name	Association for the Advancement of Artificial Intelligence
best_oa_location.source.host_organization_lineage	https://openalex.org/P4310320058
best_oa_location.source.host_organization_lineage_names	Association for the Advancement of Artificial Intelligence
best_oa_location.license
best_oa_location.pdf_url	https://ojs.aaai.org/index.php/AAAI/article/download/29022/29939
best_oa_location.version	publishedVersion
best_oa_location.raw_type	journal-article
best_oa_location.license_id
best_oa_location.is_accepted	True
best_oa_location.is_published	True
best_oa_location.raw_source_name	Proceedings of the AAAI Conference on Artificial Intelligence
best_oa_location.landing_page_url	https://doi.org/10.1609/aaai.v38i10.29022
primary_location.id	doi:10.1609/aaai.v38i10.29022
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4210191458
primary_location.source.issn	2159-5399, 2374-3468
primary_location.source.type	conference
primary_location.source.is_oa	True
primary_location.source.issn_l	2159-5399
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	Proceedings of the AAAI Conference on Artificial Intelligence
primary_location.source.host_organization	https://openalex.org/P4310320058
primary_location.source.host_organization_name	Association for the Advancement of Artificial Intelligence
primary_location.source.host_organization_lineage	https://openalex.org/P4310320058
primary_location.source.host_organization_lineage_names	Association for the Advancement of Artificial Intelligence
primary_location.license
primary_location.pdf_url	https://ojs.aaai.org/index.php/AAAI/article/download/29022/29939
primary_location.version	publishedVersion
primary_location.raw_type	journal-article
primary_location.license_id
primary_location.is_accepted	True
primary_location.is_published	True
primary_location.raw_source_name	Proceedings of the AAAI Conference on Artificial Intelligence
primary_location.landing_page_url	https://doi.org/10.1609/aaai.v38i10.29022
publication_date	2024-03-24
publication_year	2024
referenced_works	https://openalex.org/W6799319881, https://openalex.org/W6736209634, https://openalex.org/W2572659264, https://openalex.org/W6832754781, https://openalex.org/W6646531194, https://openalex.org/W2750990725, https://openalex.org/W3182614517, https://openalex.org/W2336954923, https://openalex.org/W2790483052, https://openalex.org/W2806473647, https://openalex.org/W2798302089, https://openalex.org/W2904453761, https://openalex.org/W6661649765, https://openalex.org/W3034925984, https://openalex.org/W6754300060, https://openalex.org/W6719547251, https://openalex.org/W3093206925, https://openalex.org/W6806471512, https://openalex.org/W3106395004, https://openalex.org/W2950526172, https://openalex.org/W2966386280, https://openalex.org/W2990062287, https://openalex.org/W2970912396, https://openalex.org/W2468613975, https://openalex.org/W6755758474, https://openalex.org/W3138939485, https://openalex.org/W2890912058, https://openalex.org/W2944264312, https://openalex.org/W2997293639, https://openalex.org/W6677916085, https://openalex.org/W6781003217, https://openalex.org/W3187326931, https://openalex.org/W4281897969, https://openalex.org/W3203484232, https://openalex.org/W3170697419, https://openalex.org/W3128125857, https://openalex.org/W2808828826, https://openalex.org/W3127429604, https://openalex.org/W3168988767, https://openalex.org/W3013816534, https://openalex.org/W4300352300, https://openalex.org/W4287028319, https://openalex.org/W3034593529, https://openalex.org/W4281918453, https://openalex.org/W4294183581, https://openalex.org/W3174612873, https://openalex.org/W3126729338, https://openalex.org/W2895814682, https://openalex.org/W4287102143, https://openalex.org/W3044681081, https://openalex.org/W3170241279, https://openalex.org/W3094261094, https://openalex.org/W3127580383, https://openalex.org/W2979429887, https://openalex.org/W2942689850, https://openalex.org/W4287330333, https://openalex.org/W4290858323, https://openalex.org/W4289815854, https://openalex.org/W2965685971, https://openalex.org/W3166645952, https://openalex.org/W4249736682, https://openalex.org/W4214717370, https://openalex.org/W2888960341, https://openalex.org/W4291238775, https://openalex.org/W3116146552, https://openalex.org/W4288358187, https://openalex.org/W2949103145, https://openalex.org/W2964013305, https://openalex.org/W2963351358
referenced_works_count	69
abstract_inverted_index.a	37, 51, 136
abstract_inverted_index.By	156
abstract_inverted_index.In	77, 107
abstract_inverted_index.RL	127
abstract_inverted_index.We	0, 28, 48
abstract_inverted_index.an	32
abstract_inverted_index.as	168, 170
abstract_inverted_index.be	71
abstract_inverted_index.by	183
abstract_inverted_index.in	45, 113, 139
abstract_inverted_index.is	174
abstract_inverted_index.of	40
abstract_inverted_index.to	23, 73, 79, 122
abstract_inverted_index.we	19, 88, 111, 165
abstract_inverted_index.(i)	13
abstract_inverted_index.and	10, 43, 69, 84, 118
abstract_inverted_index.any	149
abstract_inverted_index.can	34, 70
abstract_inverted_index.for	55, 100, 161
abstract_inverted_index.the	21, 41, 46, 81, 91, 95, 108, 114, 119, 124, 140, 145, 158, 162, 171, 177, 180, 186
abstract_inverted_index.(RL)	5
abstract_inverted_index.(ii)	27
abstract_inverted_index.PEVI	109, 184
abstract_inverted_index.bias	137, 172
abstract_inverted_index.data	11, 82, 133, 150
abstract_inverted_index.gap,	142
abstract_inverted_index.have	24
abstract_inverted_index.less	175
abstract_inverted_index.long	169
abstract_inverted_index.mean	102, 116
abstract_inverted_index.plug	112
abstract_inverted_index.rate	99
abstract_inverted_index.show	166
abstract_inverted_index.term	138, 173
abstract_inverted_index.than	176
abstract_inverted_index.that	90, 132, 148, 167
abstract_inverted_index.with	6, 65
abstract_inverted_index.PEVI,	164
abstract_inverted_index.Value	58
abstract_inverted_index.allow	20, 29
abstract_inverted_index.bound	121
abstract_inverted_index.data.	191
abstract_inverted_index.error	98
abstract_inverted_index.false	146
abstract_inverted_index.first	49
abstract_inverted_index.given	189
abstract_inverted_index.gives	144
abstract_inverted_index.order	78
abstract_inverted_index.prove	89
abstract_inverted_index.small	38
abstract_inverted_index.solve	123
abstract_inverted_index.study	1
abstract_inverted_index.under	104
abstract_inverted_index.using	157
abstract_inverted_index.value	188
abstract_inverted_index.where	31
abstract_inverted_index.which	61, 143
abstract_inverted_index.Moving	14
abstract_inverted_index.allows	62
abstract_inverted_index.beyond	15
abstract_inverted_index.bounds	68
abstract_inverted_index.derive	50
abstract_inverted_index.gap'',	179
abstract_inverted_index.handle	80
abstract_inverted_index.modify	36
abstract_inverted_index.policy	154, 181
abstract_inverted_index.proper	66
abstract_inverted_index.reward	8, 17, 86
abstract_inverted_index.robust	101, 125
abstract_inverted_index.(PEVI),	60
abstract_inverted_index.applied	72
abstract_inverted_index.induces	135
abstract_inverted_index.minimax	96
abstract_inverted_index.offline	2, 126
abstract_inverted_index.optimal	97, 153, 187
abstract_inverted_index.reveals	131
abstract_inverted_index.rewards	22, 42
abstract_inverted_index.trimmed	115
abstract_inverted_index.various	63
abstract_inverted_index.Standard	129
abstract_inverted_index.``action	178
abstract_inverted_index.achieves	94, 185
abstract_inverted_index.analysis	130
abstract_inverted_index.attacker	33
abstract_inverted_index.dataset.	47
abstract_inverted_index.fraction	39
abstract_inverted_index.infinite	25
abstract_inverted_index.learning	4, 75
abstract_inverted_index.multiple	74
abstract_inverted_index.prevents	152
abstract_inverted_index.problem.	128
abstract_inverted_index.returned	182
abstract_inverted_index.setting,	87
abstract_inverted_index.Iteration	59
abstract_inverted_index.condition	54, 160
abstract_inverted_index.learning.	155
abstract_inverted_index.settings.	76
abstract_inverted_index.algorithm,	110
abstract_inverted_index.confidence	67, 120
abstract_inverted_index.corruption	83, 134, 151
abstract_inverted_index.estimation	93, 103, 117
abstract_inverted_index.estimators	64
abstract_inverted_index.impression	147
abstract_inverted_index.optimality	53, 159
abstract_inverted_index.sufficient	52, 190
abstract_inverted_index.variances;	26
abstract_inverted_index.Pessimistic	57
abstract_inverted_index.arbitrarily	35
abstract_inverted_index.corruption:	12
abstract_inverted_index.corruptions	30
abstract_inverted_index.generalized	56, 163
abstract_inverted_index.subGaussian	16
abstract_inverted_index.transitions	44
abstract_inverted_index.distribution	9
abstract_inverted_index.heavy-tailed	7, 85, 105
abstract_inverted_index.trimmed-mean	92
abstract_inverted_index.distribution,	18
abstract_inverted_index.reinforcement	3
abstract_inverted_index.suboptimality	141
abstract_inverted_index.distributions.	106
cited_by_percentile_year
countries_distinct_count	1
institutions_distinct_count	4
sustainable_development_goals[0].id	https://metadata.un.org/sdg/16
sustainable_development_goals[0].score	0.8299999833106995
sustainable_development_goals[0].display_name	Peace, Justice and strong institutions
citation_normalized_percentile.value	0.04070796
citation_normalized_percentile.is_in_top_1_percent	False
citation_normalized_percentile.is_in_top_10_percent	False