Bayesian Nonparametrics for Offline Skill Discovery Article Swipe
Valentin Villecroze
,
Harry J. Braviner
,
Panteha Naderian
,
Chris J. Maddison
,
Gabriel Loaiza-Ganem
·
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2202.04675
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2202.04675
Skills or low-level policies in reinforcement learning are temporally extended actions that can speed up learning and enable complex behaviours. Recent work in offline reinforcement learning and imitation learning has proposed several techniques for skill discovery from a set of expert trajectories. While these methods are promising, the number K of skills to discover is always a fixed hyperparameter, which requires either prior knowledge about the environment or an additional parameter search to tune it. We first propose a method for offline learning of options (a particular skill framework) exploiting advances in variational inference and continuous relaxations. We then highlight an unexplored connection between Bayesian nonparametrics and offline skill discovery, and show how to obtain a nonparametric version of our model. This version is tractable thanks to a carefully structured approximate posterior with a dynamically-changing number of options, removing the need to specify K. We also show how our nonparametric extension can be applied in other skill frameworks, and empirically demonstrate that our method can outperform state-of-the-art offline skill learning algorithms across a variety of environments. Our code is available at https://github.com/layer6ai-labs/BNPO .
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2202.04675
- https://arxiv.org/pdf/2202.04675
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4221142095
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4221142095Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2202.04675Digital Object Identifier
- Title
-
Bayesian Nonparametrics for Offline Skill DiscoveryWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2022Year of publication
- Publication date
-
2022-02-09Full publication date if available
- Authors
-
Valentin Villecroze, Harry J. Braviner, Panteha Naderian, Chris J. Maddison, Gabriel Loaiza-GanemList of authors in order
- Landing page
-
https://arxiv.org/abs/2202.04675Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2202.04675Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2202.04675Direct OA link when available
- Concepts
-
Computer science, Reinforcement learning, Machine learning, Hyperparameter, Inference, Artificial intelligence, Variety (cybernetics), Set (abstract data type), Bayesian inference, Bayesian probability, Programming languageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4221142095 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2202.04675 |
| ids.doi | https://doi.org/10.48550/arxiv.2202.04675 |
| ids.openalex | https://openalex.org/W4221142095 |
| fwci | |
| type | preprint |
| title | Bayesian Nonparametrics for Offline Skill Discovery |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10462 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9973000288009644 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Reinforcement Learning in Robotics |
| topics[1].id | https://openalex.org/T12535 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9970999956130981 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Machine Learning and Data Classification |
| topics[2].id | https://openalex.org/T10848 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9828000068664551 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1703 |
| topics[2].subfield.display_name | Computational Theory and Mathematics |
| topics[2].display_name | Advanced Multi-Objective Optimization Algorithms |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.7542203664779663 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C97541855 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7377997636795044 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q830687 |
| concepts[1].display_name | Reinforcement learning |
| concepts[2].id | https://openalex.org/C119857082 |
| concepts[2].level | 1 |
| concepts[2].score | 0.6395944952964783 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[2].display_name | Machine learning |
| concepts[3].id | https://openalex.org/C8642999 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5868500471115112 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q4171168 |
| concepts[3].display_name | Hyperparameter |
| concepts[4].id | https://openalex.org/C2776214188 |
| concepts[4].level | 2 |
| concepts[4].score | 0.562639594078064 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q408386 |
| concepts[4].display_name | Inference |
| concepts[5].id | https://openalex.org/C154945302 |
| concepts[5].level | 1 |
| concepts[5].score | 0.5545663237571716 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[5].display_name | Artificial intelligence |
| concepts[6].id | https://openalex.org/C136197465 |
| concepts[6].level | 2 |
| concepts[6].score | 0.5114089846611023 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q1729295 |
| concepts[6].display_name | Variety (cybernetics) |
| concepts[7].id | https://openalex.org/C177264268 |
| concepts[7].level | 2 |
| concepts[7].score | 0.5102226734161377 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q1514741 |
| concepts[7].display_name | Set (abstract data type) |
| concepts[8].id | https://openalex.org/C160234255 |
| concepts[8].level | 3 |
| concepts[8].score | 0.48900970816612244 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q812535 |
| concepts[8].display_name | Bayesian inference |
| concepts[9].id | https://openalex.org/C107673813 |
| concepts[9].level | 2 |
| concepts[9].score | 0.44630593061447144 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q812534 |
| concepts[9].display_name | Bayesian probability |
| concepts[10].id | https://openalex.org/C199360897 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[10].display_name | Programming language |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.7542203664779663 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/reinforcement-learning |
| keywords[1].score | 0.7377997636795044 |
| keywords[1].display_name | Reinforcement learning |
| keywords[2].id | https://openalex.org/keywords/machine-learning |
| keywords[2].score | 0.6395944952964783 |
| keywords[2].display_name | Machine learning |
| keywords[3].id | https://openalex.org/keywords/hyperparameter |
| keywords[3].score | 0.5868500471115112 |
| keywords[3].display_name | Hyperparameter |
| keywords[4].id | https://openalex.org/keywords/inference |
| keywords[4].score | 0.562639594078064 |
| keywords[4].display_name | Inference |
| keywords[5].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[5].score | 0.5545663237571716 |
| keywords[5].display_name | Artificial intelligence |
| keywords[6].id | https://openalex.org/keywords/variety |
| keywords[6].score | 0.5114089846611023 |
| keywords[6].display_name | Variety (cybernetics) |
| keywords[7].id | https://openalex.org/keywords/set |
| keywords[7].score | 0.5102226734161377 |
| keywords[7].display_name | Set (abstract data type) |
| keywords[8].id | https://openalex.org/keywords/bayesian-inference |
| keywords[8].score | 0.48900970816612244 |
| keywords[8].display_name | Bayesian inference |
| keywords[9].id | https://openalex.org/keywords/bayesian-probability |
| keywords[9].score | 0.44630593061447144 |
| keywords[9].display_name | Bayesian probability |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2202.04675 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2202.04675 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2202.04675 |
| locations[1].id | doi:10.48550/arxiv.2202.04675 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2202.04675 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5005089622 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Valentin Villecroze |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Villecroze, Valentin |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5043056301 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Harry J. Braviner |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Braviner, Harry J. |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5001393563 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Panteha Naderian |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Naderian, Panteha |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5054711904 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Chris J. Maddison |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Maddison, Chris J. |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5020477291 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Gabriel Loaiza-Ganem |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Loaiza-Ganem, Gabriel |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2202.04675 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Bayesian Nonparametrics for Offline Skill Discovery |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10462 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9973000288009644 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Reinforcement Learning in Robotics |
| related_works | https://openalex.org/W4281847915, https://openalex.org/W3128856671, https://openalex.org/W2372267530, https://openalex.org/W2969189870, https://openalex.org/W4303857162, https://openalex.org/W2965643117, https://openalex.org/W2407375987, https://openalex.org/W2505726097, https://openalex.org/W2010643158, https://openalex.org/W3049691116 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2202.04675 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2202.04675 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2202.04675 |
| primary_location.id | pmh:oai:arXiv.org:2202.04675 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2202.04675 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2202.04675 |
| publication_date | 2022-02-09 |
| publication_year | 2022 |
| referenced_works_count | 0 |
| abstract_inverted_index.. | 182 |
| abstract_inverted_index.K | 49 |
| abstract_inverted_index.a | 37, 56, 78, 115, 127, 133, 172 |
| abstract_inverted_index.(a | 85 |
| abstract_inverted_index.K. | 143 |
| abstract_inverted_index.We | 75, 97, 144 |
| abstract_inverted_index.an | 68, 100 |
| abstract_inverted_index.at | 180 |
| abstract_inverted_index.be | 152 |
| abstract_inverted_index.in | 4, 22, 91, 154 |
| abstract_inverted_index.is | 54, 123, 178 |
| abstract_inverted_index.of | 39, 50, 83, 118, 136, 174 |
| abstract_inverted_index.or | 1, 67 |
| abstract_inverted_index.to | 52, 72, 113, 126, 141 |
| abstract_inverted_index.up | 14 |
| abstract_inverted_index.Our | 176 |
| abstract_inverted_index.and | 16, 26, 94, 106, 110, 158 |
| abstract_inverted_index.are | 7, 45 |
| abstract_inverted_index.can | 12, 151, 164 |
| abstract_inverted_index.for | 33, 80 |
| abstract_inverted_index.has | 29 |
| abstract_inverted_index.how | 112, 147 |
| abstract_inverted_index.it. | 74 |
| abstract_inverted_index.our | 119, 148, 162 |
| abstract_inverted_index.set | 38 |
| abstract_inverted_index.the | 47, 65, 139 |
| abstract_inverted_index.This | 121 |
| abstract_inverted_index.also | 145 |
| abstract_inverted_index.code | 177 |
| abstract_inverted_index.from | 36 |
| abstract_inverted_index.need | 140 |
| abstract_inverted_index.show | 111, 146 |
| abstract_inverted_index.that | 11, 161 |
| abstract_inverted_index.then | 98 |
| abstract_inverted_index.tune | 73 |
| abstract_inverted_index.with | 132 |
| abstract_inverted_index.work | 21 |
| abstract_inverted_index.While | 42 |
| abstract_inverted_index.about | 64 |
| abstract_inverted_index.first | 76 |
| abstract_inverted_index.fixed | 57 |
| abstract_inverted_index.other | 155 |
| abstract_inverted_index.prior | 62 |
| abstract_inverted_index.skill | 34, 87, 108, 156, 168 |
| abstract_inverted_index.speed | 13 |
| abstract_inverted_index.these | 43 |
| abstract_inverted_index.which | 59 |
| abstract_inverted_index.Recent | 20 |
| abstract_inverted_index.Skills | 0 |
| abstract_inverted_index.across | 171 |
| abstract_inverted_index.always | 55 |
| abstract_inverted_index.either | 61 |
| abstract_inverted_index.enable | 17 |
| abstract_inverted_index.expert | 40 |
| abstract_inverted_index.method | 79, 163 |
| abstract_inverted_index.model. | 120 |
| abstract_inverted_index.number | 48, 135 |
| abstract_inverted_index.obtain | 114 |
| abstract_inverted_index.search | 71 |
| abstract_inverted_index.skills | 51 |
| abstract_inverted_index.thanks | 125 |
| abstract_inverted_index.actions | 10 |
| abstract_inverted_index.applied | 153 |
| abstract_inverted_index.between | 103 |
| abstract_inverted_index.complex | 18 |
| abstract_inverted_index.methods | 44 |
| abstract_inverted_index.offline | 23, 81, 107, 167 |
| abstract_inverted_index.options | 84 |
| abstract_inverted_index.propose | 77 |
| abstract_inverted_index.several | 31 |
| abstract_inverted_index.specify | 142 |
| abstract_inverted_index.variety | 173 |
| abstract_inverted_index.version | 117, 122 |
| abstract_inverted_index.Bayesian | 104 |
| abstract_inverted_index.advances | 90 |
| abstract_inverted_index.discover | 53 |
| abstract_inverted_index.extended | 9 |
| abstract_inverted_index.learning | 6, 15, 25, 28, 82, 169 |
| abstract_inverted_index.options, | 137 |
| abstract_inverted_index.policies | 3 |
| abstract_inverted_index.proposed | 30 |
| abstract_inverted_index.removing | 138 |
| abstract_inverted_index.requires | 60 |
| abstract_inverted_index.available | 179 |
| abstract_inverted_index.carefully | 128 |
| abstract_inverted_index.discovery | 35 |
| abstract_inverted_index.extension | 150 |
| abstract_inverted_index.highlight | 99 |
| abstract_inverted_index.imitation | 27 |
| abstract_inverted_index.inference | 93 |
| abstract_inverted_index.knowledge | 63 |
| abstract_inverted_index.low-level | 2 |
| abstract_inverted_index.parameter | 70 |
| abstract_inverted_index.posterior | 131 |
| abstract_inverted_index.tractable | 124 |
| abstract_inverted_index.additional | 69 |
| abstract_inverted_index.algorithms | 170 |
| abstract_inverted_index.connection | 102 |
| abstract_inverted_index.continuous | 95 |
| abstract_inverted_index.discovery, | 109 |
| abstract_inverted_index.exploiting | 89 |
| abstract_inverted_index.framework) | 88 |
| abstract_inverted_index.outperform | 165 |
| abstract_inverted_index.particular | 86 |
| abstract_inverted_index.promising, | 46 |
| abstract_inverted_index.structured | 129 |
| abstract_inverted_index.techniques | 32 |
| abstract_inverted_index.temporally | 8 |
| abstract_inverted_index.unexplored | 101 |
| abstract_inverted_index.approximate | 130 |
| abstract_inverted_index.behaviours. | 19 |
| abstract_inverted_index.demonstrate | 160 |
| abstract_inverted_index.empirically | 159 |
| abstract_inverted_index.environment | 66 |
| abstract_inverted_index.frameworks, | 157 |
| abstract_inverted_index.variational | 92 |
| abstract_inverted_index.relaxations. | 96 |
| abstract_inverted_index.environments. | 175 |
| abstract_inverted_index.nonparametric | 116, 149 |
| abstract_inverted_index.reinforcement | 5, 24 |
| abstract_inverted_index.trajectories. | 41 |
| abstract_inverted_index.nonparametrics | 105 |
| abstract_inverted_index.hyperparameter, | 58 |
| abstract_inverted_index.state-of-the-art | 166 |
| abstract_inverted_index.dynamically-changing | 134 |
| abstract_inverted_index.https://github.com/layer6ai-labs/BNPO | 181 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.7900000214576721 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile |