Polycraft World AI Lab (PAL): An Extensible Platform for Evaluating Artificial Intelligence Agents Article Swipe
Stephen A. Goss
,
Robert J. Steininger
,
Dhruv Narayanan
,
Daniel V. Olivença
,
Yutong Sun
,
Peng Qiu
,
Jim Amato
,
Eberhard O. Voit
,
Walter Voit
,
Eric Kildebeck
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2301.11891
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2301.11891
As artificial intelligence research advances, the platforms used to evaluate AI agents need to adapt and grow to continue to challenge them. We present the Polycraft World AI Lab (PAL), a task simulator with an API based on the Minecraft mod Polycraft World. Our platform is built to allow AI agents with different architectures to easily interact with the Minecraft world, train and be evaluated in multiple tasks. PAL enables the creation of tasks in a flexible manner as well as having the capability to manipulate any aspect of the task during an evaluation. All actions taken by AI agents and external actors (non-player-characters, NPCs) in the open-world environment are logged to streamline evaluation. Here we present two custom tasks on the PAL platform, one focused on multi-step planning and one focused on navigation, and evaluations of agents solving them. In summary, we report a versatile and extensible AI evaluation platform with a low barrier to entry for AI researchers to utilize.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2301.11891
- https://arxiv.org/pdf/2301.11891
- OA Status
- green
- Cited By
- 4
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4318621207
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4318621207Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2301.11891Digital Object Identifier
- Title
-
Polycraft World AI Lab (PAL): An Extensible Platform for Evaluating Artificial Intelligence AgentsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-01-27Full publication date if available
- Authors
-
Stephen A. Goss, Robert J. Steininger, Dhruv Narayanan, Daniel V. Olivença, Yutong Sun, Peng Qiu, Jim Amato, Eberhard O. Voit, Walter Voit, Eric KildebeckList of authors in order
- Landing page
-
https://arxiv.org/abs/2301.11891Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2301.11891Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2301.11891Direct OA link when available
- Concepts
-
Task (project management), Computer science, Artificial intelligence, Extensibility, Human–computer interaction, Software engineering, Engineering, Systems engineering, Programming languageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
4Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 1, 2024: 2, 2023: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4318621207 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2301.11891 |
| ids.doi | https://doi.org/10.48550/arxiv.2301.11891 |
| ids.openalex | https://openalex.org/W4318621207 |
| fwci | |
| type | preprint |
| title | Polycraft World AI Lab (PAL): An Extensible Platform for Evaluating Artificial Intelligence Agents |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12026 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9839000105857849 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Explainable Artificial Intelligence (XAI) |
| topics[1].id | https://openalex.org/T10028 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9733999967575073 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Topic Modeling |
| topics[2].id | https://openalex.org/T11986 |
| topics[2].field.id | https://openalex.org/fields/18 |
| topics[2].field.display_name | Decision Sciences |
| topics[2].score | 0.9508000016212463 |
| topics[2].domain.id | https://openalex.org/domains/2 |
| topics[2].domain.display_name | Social Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1802 |
| topics[2].subfield.display_name | Information Systems and Management |
| topics[2].display_name | Scientific Computing and Data Management |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2780451532 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7182974815368652 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q759676 |
| concepts[0].display_name | Task (project management) |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.6915229558944702 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C154945302 |
| concepts[2].level | 1 |
| concepts[2].score | 0.5349605679512024 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[2].display_name | Artificial intelligence |
| concepts[3].id | https://openalex.org/C32833848 |
| concepts[3].level | 2 |
| concepts[3].score | 0.47946369647979736 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q4115054 |
| concepts[3].display_name | Extensibility |
| concepts[4].id | https://openalex.org/C107457646 |
| concepts[4].level | 1 |
| concepts[4].score | 0.4572969377040863 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q207434 |
| concepts[4].display_name | Human–computer interaction |
| concepts[5].id | https://openalex.org/C115903868 |
| concepts[5].level | 1 |
| concepts[5].score | 0.36599138379096985 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q80993 |
| concepts[5].display_name | Software engineering |
| concepts[6].id | https://openalex.org/C127413603 |
| concepts[6].level | 0 |
| concepts[6].score | 0.1780126988887787 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[6].display_name | Engineering |
| concepts[7].id | https://openalex.org/C201995342 |
| concepts[7].level | 1 |
| concepts[7].score | 0.17167681455612183 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q682496 |
| concepts[7].display_name | Systems engineering |
| concepts[8].id | https://openalex.org/C199360897 |
| concepts[8].level | 1 |
| concepts[8].score | 0.09462776780128479 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[8].display_name | Programming language |
| keywords[0].id | https://openalex.org/keywords/task |
| keywords[0].score | 0.7182974815368652 |
| keywords[0].display_name | Task (project management) |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.6915229558944702 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[2].score | 0.5349605679512024 |
| keywords[2].display_name | Artificial intelligence |
| keywords[3].id | https://openalex.org/keywords/extensibility |
| keywords[3].score | 0.47946369647979736 |
| keywords[3].display_name | Extensibility |
| keywords[4].id | https://openalex.org/keywords/human–computer-interaction |
| keywords[4].score | 0.4572969377040863 |
| keywords[4].display_name | Human–computer interaction |
| keywords[5].id | https://openalex.org/keywords/software-engineering |
| keywords[5].score | 0.36599138379096985 |
| keywords[5].display_name | Software engineering |
| keywords[6].id | https://openalex.org/keywords/engineering |
| keywords[6].score | 0.1780126988887787 |
| keywords[6].display_name | Engineering |
| keywords[7].id | https://openalex.org/keywords/systems-engineering |
| keywords[7].score | 0.17167681455612183 |
| keywords[7].display_name | Systems engineering |
| keywords[8].id | https://openalex.org/keywords/programming-language |
| keywords[8].score | 0.09462776780128479 |
| keywords[8].display_name | Programming language |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2301.11891 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2301.11891 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2301.11891 |
| locations[1].id | doi:10.48550/arxiv.2301.11891 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2301.11891 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5114041375 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Stephen A. Goss |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Goss, Stephen A. |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5079258586 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Robert J. Steininger |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Steininger, Robert J. |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5082924355 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Dhruv Narayanan |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Narayanan, Dhruv |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5048280648 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-5474-2657 |
| authorships[3].author.display_name | Daniel V. Olivença |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Olivença, Daniel V. |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5100587792 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | Yutong Sun |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Sun, Yutong |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5101449039 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-2509-3683 |
| authorships[5].author.display_name | Peng Qiu |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Qiu, Peng |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5077265353 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Jim Amato |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Amato, Jim |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5063233199 |
| authorships[7].author.orcid | https://orcid.org/0000-0003-1378-3043 |
| authorships[7].author.display_name | Eberhard O. Voit |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Voit, Eberhard O. |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5075556499 |
| authorships[8].author.orcid | https://orcid.org/0000-0003-0135-0531 |
| authorships[8].author.display_name | Walter Voit |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Voit, Walter E. |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5051966888 |
| authorships[9].author.orcid | https://orcid.org/0000-0001-7209-0226 |
| authorships[9].author.display_name | Eric Kildebeck |
| authorships[9].author_position | last |
| authorships[9].raw_author_name | Kildebeck, Eric J. |
| authorships[9].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2301.11891 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Polycraft World AI Lab (PAL): An Extensible Platform for Evaluating Artificial Intelligence Agents |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T12026 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9839000105857849 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Explainable Artificial Intelligence (XAI) |
| related_works | https://openalex.org/W1948607442, https://openalex.org/W3004004161, https://openalex.org/W2044615423, https://openalex.org/W4247766898, https://openalex.org/W4244765761, https://openalex.org/W2361584951, https://openalex.org/W2365327041, https://openalex.org/W4225348249, https://openalex.org/W31122515, https://openalex.org/W3196817267 |
| cited_by_count | 4 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 1 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 2 |
| counts_by_year[2].year | 2023 |
| counts_by_year[2].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2301.11891 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2301.11891 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2301.11891 |
| primary_location.id | pmh:oai:arXiv.org:2301.11891 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2301.11891 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2301.11891 |
| publication_date | 2023-01-27 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 30, 75, 144, 152 |
| abstract_inverted_index.AI | 10, 27, 49, 98, 148, 158 |
| abstract_inverted_index.As | 0 |
| abstract_inverted_index.In | 140 |
| abstract_inverted_index.We | 22 |
| abstract_inverted_index.an | 34, 92 |
| abstract_inverted_index.as | 78, 80 |
| abstract_inverted_index.be | 63 |
| abstract_inverted_index.by | 97 |
| abstract_inverted_index.in | 65, 74, 105 |
| abstract_inverted_index.is | 45 |
| abstract_inverted_index.of | 72, 88, 136 |
| abstract_inverted_index.on | 37, 120, 126, 132 |
| abstract_inverted_index.to | 8, 13, 17, 19, 47, 54, 84, 111, 155, 160 |
| abstract_inverted_index.we | 115, 142 |
| abstract_inverted_index.API | 35 |
| abstract_inverted_index.All | 94 |
| abstract_inverted_index.Lab | 28 |
| abstract_inverted_index.Our | 43 |
| abstract_inverted_index.PAL | 68, 122 |
| abstract_inverted_index.and | 15, 62, 100, 129, 134, 146 |
| abstract_inverted_index.any | 86 |
| abstract_inverted_index.are | 109 |
| abstract_inverted_index.for | 157 |
| abstract_inverted_index.low | 153 |
| abstract_inverted_index.mod | 40 |
| abstract_inverted_index.one | 124, 130 |
| abstract_inverted_index.the | 5, 24, 38, 58, 70, 82, 89, 106, 121 |
| abstract_inverted_index.two | 117 |
| abstract_inverted_index.Here | 114 |
| abstract_inverted_index.grow | 16 |
| abstract_inverted_index.need | 12 |
| abstract_inverted_index.task | 31, 90 |
| abstract_inverted_index.used | 7 |
| abstract_inverted_index.well | 79 |
| abstract_inverted_index.with | 33, 51, 57, 151 |
| abstract_inverted_index.NPCs) | 104 |
| abstract_inverted_index.World | 26 |
| abstract_inverted_index.adapt | 14 |
| abstract_inverted_index.allow | 48 |
| abstract_inverted_index.based | 36 |
| abstract_inverted_index.built | 46 |
| abstract_inverted_index.entry | 156 |
| abstract_inverted_index.taken | 96 |
| abstract_inverted_index.tasks | 73, 119 |
| abstract_inverted_index.them. | 21, 139 |
| abstract_inverted_index.train | 61 |
| abstract_inverted_index.(PAL), | 29 |
| abstract_inverted_index.World. | 42 |
| abstract_inverted_index.actors | 102 |
| abstract_inverted_index.agents | 11, 50, 99, 137 |
| abstract_inverted_index.aspect | 87 |
| abstract_inverted_index.custom | 118 |
| abstract_inverted_index.during | 91 |
| abstract_inverted_index.easily | 55 |
| abstract_inverted_index.having | 81 |
| abstract_inverted_index.logged | 110 |
| abstract_inverted_index.manner | 77 |
| abstract_inverted_index.report | 143 |
| abstract_inverted_index.tasks. | 67 |
| abstract_inverted_index.world, | 60 |
| abstract_inverted_index.actions | 95 |
| abstract_inverted_index.barrier | 154 |
| abstract_inverted_index.enables | 69 |
| abstract_inverted_index.focused | 125, 131 |
| abstract_inverted_index.present | 23, 116 |
| abstract_inverted_index.solving | 138 |
| abstract_inverted_index.continue | 18 |
| abstract_inverted_index.creation | 71 |
| abstract_inverted_index.evaluate | 9 |
| abstract_inverted_index.external | 101 |
| abstract_inverted_index.flexible | 76 |
| abstract_inverted_index.interact | 56 |
| abstract_inverted_index.multiple | 66 |
| abstract_inverted_index.planning | 128 |
| abstract_inverted_index.platform | 44, 150 |
| abstract_inverted_index.research | 3 |
| abstract_inverted_index.summary, | 141 |
| abstract_inverted_index.utilize. | 161 |
| abstract_inverted_index.Minecraft | 39, 59 |
| abstract_inverted_index.Polycraft | 25, 41 |
| abstract_inverted_index.advances, | 4 |
| abstract_inverted_index.challenge | 20 |
| abstract_inverted_index.different | 52 |
| abstract_inverted_index.evaluated | 64 |
| abstract_inverted_index.platform, | 123 |
| abstract_inverted_index.platforms | 6 |
| abstract_inverted_index.simulator | 32 |
| abstract_inverted_index.versatile | 145 |
| abstract_inverted_index.artificial | 1 |
| abstract_inverted_index.capability | 83 |
| abstract_inverted_index.evaluation | 149 |
| abstract_inverted_index.extensible | 147 |
| abstract_inverted_index.manipulate | 85 |
| abstract_inverted_index.multi-step | 127 |
| abstract_inverted_index.open-world | 107 |
| abstract_inverted_index.streamline | 112 |
| abstract_inverted_index.environment | 108 |
| abstract_inverted_index.evaluation. | 93, 113 |
| abstract_inverted_index.evaluations | 135 |
| abstract_inverted_index.navigation, | 133 |
| abstract_inverted_index.researchers | 159 |
| abstract_inverted_index.intelligence | 2 |
| abstract_inverted_index.architectures | 53 |
| abstract_inverted_index.(non-player-characters, | 103 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 10 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/9 |
| sustainable_development_goals[0].score | 0.4099999964237213 |
| sustainable_development_goals[0].display_name | Industry, innovation and infrastructure |
| citation_normalized_percentile |