Dialogue Evaluation with Offline Reinforcement Learning Article Swipe
Nurul Lubis
,
Christian Geishauser
,
Hsien-chin Lin
,
Carel van Niekerk
,
Michael Heck
,
Shutong Feng
,
Milica Gašić
·
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2209.00876
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2209.00876
Task-oriented dialogue systems aim to fulfill user goals through natural language interactions. They are ideally evaluated with human users, which however is unattainable to do at every iteration of the development phase. Simulated users could be an alternative, however their development is nontrivial. Therefore, researchers resort to offline metrics on existing human-human corpora, which are more practical and easily reproducible. They are unfortunately limited in reflecting real performance of dialogue systems. BLEU for instance is poorly correlated with human judgment, and existing corpus-based metrics such as success rate overlook dialogue context mismatches. There is still a need for a reliable metric for task-oriented systems with good generalization and strong correlation with human judgements. In this paper, we propose the use of offline reinforcement learning for dialogue evaluation based on a static corpus. Such an evaluator is typically called a critic and utilized for policy optimization. We go one step further and show that offline RL critics can be trained on a static corpus for any dialogue system as external evaluators, allowing dialogue performance comparisons across various types of systems. This approach has the benefit of being corpus- and model-independent, while attaining strong correlation with human judgements, which we confirm via an interactive user trial.
Related Topics
Concepts
Computer science
Reinforcement learning
Metric (unit)
Task (project management)
Generalization
Artificial intelligence
Context (archaeology)
Natural language processing
Machine learning
Human–computer interaction
Management
Mathematical analysis
Economics
Operations management
Biology
Mathematics
Paleontology
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2209.00876
- https://arxiv.org/pdf/2209.00876
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4294753080
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4294753080Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2209.00876Digital Object Identifier
- Title
-
Dialogue Evaluation with Offline Reinforcement LearningWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2022Year of publication
- Publication date
-
2022-09-02Full publication date if available
- Authors
-
Nurul Lubis, Christian Geishauser, Hsien-chin Lin, Carel van Niekerk, Michael Heck, Shutong Feng, Milica GašićList of authors in order
- Landing page
-
https://arxiv.org/abs/2209.00876Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2209.00876Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2209.00876Direct OA link when available
- Concepts
-
Computer science, Reinforcement learning, Metric (unit), Task (project management), Generalization, Artificial intelligence, Context (archaeology), Natural language processing, Machine learning, Human–computer interaction, Management, Mathematical analysis, Economics, Operations management, Biology, Mathematics, PaleontologyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4294753080 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2209.00876 |
| ids.doi | https://doi.org/10.48550/arxiv.2209.00876 |
| ids.openalex | https://openalex.org/W4294753080 |
| fwci | |
| type | preprint |
| title | Dialogue Evaluation with Offline Reinforcement Learning |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12031 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9984999895095825 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Speech and dialogue systems |
| topics[1].id | https://openalex.org/T10028 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9904999732971191 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Topic Modeling |
| topics[2].id | https://openalex.org/T11902 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9832000136375427 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Intelligent Tutoring Systems and Adaptive Learning |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.8456389904022217 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C97541855 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7443925738334656 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q830687 |
| concepts[1].display_name | Reinforcement learning |
| concepts[2].id | https://openalex.org/C176217482 |
| concepts[2].level | 2 |
| concepts[2].score | 0.6730756163597107 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q860554 |
| concepts[2].display_name | Metric (unit) |
| concepts[3].id | https://openalex.org/C2780451532 |
| concepts[3].level | 2 |
| concepts[3].score | 0.6643487215042114 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q759676 |
| concepts[3].display_name | Task (project management) |
| concepts[4].id | https://openalex.org/C177148314 |
| concepts[4].level | 2 |
| concepts[4].score | 0.6505661010742188 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q170084 |
| concepts[4].display_name | Generalization |
| concepts[5].id | https://openalex.org/C154945302 |
| concepts[5].level | 1 |
| concepts[5].score | 0.6302498579025269 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[5].display_name | Artificial intelligence |
| concepts[6].id | https://openalex.org/C2779343474 |
| concepts[6].level | 2 |
| concepts[6].score | 0.5505887269973755 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q3109175 |
| concepts[6].display_name | Context (archaeology) |
| concepts[7].id | https://openalex.org/C204321447 |
| concepts[7].level | 1 |
| concepts[7].score | 0.48113930225372314 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[7].display_name | Natural language processing |
| concepts[8].id | https://openalex.org/C119857082 |
| concepts[8].level | 1 |
| concepts[8].score | 0.4807400703430176 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[8].display_name | Machine learning |
| concepts[9].id | https://openalex.org/C107457646 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3755989074707031 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q207434 |
| concepts[9].display_name | Human–computer interaction |
| concepts[10].id | https://openalex.org/C187736073 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q2920921 |
| concepts[10].display_name | Management |
| concepts[11].id | https://openalex.org/C134306372 |
| concepts[11].level | 1 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q7754 |
| concepts[11].display_name | Mathematical analysis |
| concepts[12].id | https://openalex.org/C162324750 |
| concepts[12].level | 0 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q8134 |
| concepts[12].display_name | Economics |
| concepts[13].id | https://openalex.org/C21547014 |
| concepts[13].level | 1 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q1423657 |
| concepts[13].display_name | Operations management |
| concepts[14].id | https://openalex.org/C86803240 |
| concepts[14].level | 0 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q420 |
| concepts[14].display_name | Biology |
| concepts[15].id | https://openalex.org/C33923547 |
| concepts[15].level | 0 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[15].display_name | Mathematics |
| concepts[16].id | https://openalex.org/C151730666 |
| concepts[16].level | 1 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q7205 |
| concepts[16].display_name | Paleontology |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.8456389904022217 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/reinforcement-learning |
| keywords[1].score | 0.7443925738334656 |
| keywords[1].display_name | Reinforcement learning |
| keywords[2].id | https://openalex.org/keywords/metric |
| keywords[2].score | 0.6730756163597107 |
| keywords[2].display_name | Metric (unit) |
| keywords[3].id | https://openalex.org/keywords/task |
| keywords[3].score | 0.6643487215042114 |
| keywords[3].display_name | Task (project management) |
| keywords[4].id | https://openalex.org/keywords/generalization |
| keywords[4].score | 0.6505661010742188 |
| keywords[4].display_name | Generalization |
| keywords[5].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[5].score | 0.6302498579025269 |
| keywords[5].display_name | Artificial intelligence |
| keywords[6].id | https://openalex.org/keywords/context |
| keywords[6].score | 0.5505887269973755 |
| keywords[6].display_name | Context (archaeology) |
| keywords[7].id | https://openalex.org/keywords/natural-language-processing |
| keywords[7].score | 0.48113930225372314 |
| keywords[7].display_name | Natural language processing |
| keywords[8].id | https://openalex.org/keywords/machine-learning |
| keywords[8].score | 0.4807400703430176 |
| keywords[8].display_name | Machine learning |
| keywords[9].id | https://openalex.org/keywords/human–computer-interaction |
| keywords[9].score | 0.3755989074707031 |
| keywords[9].display_name | Human–computer interaction |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2209.00876 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2209.00876 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2209.00876 |
| locations[1].id | doi:10.48550/arxiv.2209.00876 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2209.00876 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5044076701 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-4461-7243 |
| authorships[0].author.display_name | Nurul Lubis |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Lubis, Nurul |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5057058723 |
| authorships[1].author.orcid | https://orcid.org/0009-0004-6369-3438 |
| authorships[1].author.display_name | Christian Geishauser |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Geishauser, Christian |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5103130759 |
| authorships[2].author.orcid | https://orcid.org/0009-0006-0027-226X |
| authorships[2].author.display_name | Hsien-chin Lin |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Lin, Hsien-Chin |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5023425411 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-4551-7447 |
| authorships[3].author.display_name | Carel van Niekerk |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | van Niekerk, Carel |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5015873217 |
| authorships[4].author.orcid | https://orcid.org/0000-0001-9841-5025 |
| authorships[4].author.display_name | Michael Heck |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Heck, Michael |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5031650617 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-1307-4223 |
| authorships[5].author.display_name | Shutong Feng |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Feng, Shutong |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5051889115 |
| authorships[6].author.orcid | https://orcid.org/0000-0003-0318-9147 |
| authorships[6].author.display_name | Milica Gašić |
| authorships[6].author_position | last |
| authorships[6].raw_author_name | Gašić, Milica |
| authorships[6].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2209.00876 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Dialogue Evaluation with Offline Reinforcement Learning |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T12031 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9984999895095825 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Speech and dialogue systems |
| related_works | https://openalex.org/W4306904969, https://openalex.org/W3162204513, https://openalex.org/W2138720691, https://openalex.org/W4362501864, https://openalex.org/W4380318855, https://openalex.org/W2031695474, https://openalex.org/W2024136090, https://openalex.org/W2586732548, https://openalex.org/W2075445622, https://openalex.org/W3196817267 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2209.00876 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2209.00876 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2209.00876 |
| primary_location.id | pmh:oai:arXiv.org:2209.00876 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2209.00876 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2209.00876 |
| publication_date | 2022-09-02 |
| publication_year | 2022 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 95, 98, 129, 138, 160 |
| abstract_inverted_index.In | 113 |
| abstract_inverted_index.RL | 154 |
| abstract_inverted_index.We | 145 |
| abstract_inverted_index.an | 36, 133, 200 |
| abstract_inverted_index.as | 85, 167 |
| abstract_inverted_index.at | 25 |
| abstract_inverted_index.be | 35, 157 |
| abstract_inverted_index.do | 24 |
| abstract_inverted_index.go | 146 |
| abstract_inverted_index.in | 64 |
| abstract_inverted_index.is | 21, 41, 74, 93, 135 |
| abstract_inverted_index.of | 28, 68, 120, 177, 184 |
| abstract_inverted_index.on | 49, 128, 159 |
| abstract_inverted_index.to | 4, 23, 46 |
| abstract_inverted_index.we | 116, 197 |
| abstract_inverted_index.aim | 3 |
| abstract_inverted_index.and | 57, 80, 107, 140, 150, 187 |
| abstract_inverted_index.any | 164 |
| abstract_inverted_index.are | 13, 54, 61 |
| abstract_inverted_index.can | 156 |
| abstract_inverted_index.for | 72, 97, 101, 124, 142, 163 |
| abstract_inverted_index.has | 181 |
| abstract_inverted_index.one | 147 |
| abstract_inverted_index.the | 29, 118, 182 |
| abstract_inverted_index.use | 119 |
| abstract_inverted_index.via | 199 |
| abstract_inverted_index.BLEU | 71 |
| abstract_inverted_index.Such | 132 |
| abstract_inverted_index.They | 12, 60 |
| abstract_inverted_index.This | 179 |
| abstract_inverted_index.good | 105 |
| abstract_inverted_index.more | 55 |
| abstract_inverted_index.need | 96 |
| abstract_inverted_index.rate | 87 |
| abstract_inverted_index.real | 66 |
| abstract_inverted_index.show | 151 |
| abstract_inverted_index.step | 148 |
| abstract_inverted_index.such | 84 |
| abstract_inverted_index.that | 152 |
| abstract_inverted_index.this | 114 |
| abstract_inverted_index.user | 6, 202 |
| abstract_inverted_index.with | 16, 77, 104, 110, 193 |
| abstract_inverted_index.There | 92 |
| abstract_inverted_index.based | 127 |
| abstract_inverted_index.being | 185 |
| abstract_inverted_index.could | 34 |
| abstract_inverted_index.every | 26 |
| abstract_inverted_index.goals | 7 |
| abstract_inverted_index.human | 17, 78, 111, 194 |
| abstract_inverted_index.still | 94 |
| abstract_inverted_index.their | 39 |
| abstract_inverted_index.types | 176 |
| abstract_inverted_index.users | 33 |
| abstract_inverted_index.which | 19, 53, 196 |
| abstract_inverted_index.while | 189 |
| abstract_inverted_index.across | 174 |
| abstract_inverted_index.called | 137 |
| abstract_inverted_index.corpus | 162 |
| abstract_inverted_index.critic | 139 |
| abstract_inverted_index.easily | 58 |
| abstract_inverted_index.metric | 100 |
| abstract_inverted_index.paper, | 115 |
| abstract_inverted_index.phase. | 31 |
| abstract_inverted_index.policy | 143 |
| abstract_inverted_index.poorly | 75 |
| abstract_inverted_index.resort | 45 |
| abstract_inverted_index.static | 130, 161 |
| abstract_inverted_index.strong | 108, 191 |
| abstract_inverted_index.system | 166 |
| abstract_inverted_index.trial. | 203 |
| abstract_inverted_index.users, | 18 |
| abstract_inverted_index.benefit | 183 |
| abstract_inverted_index.confirm | 198 |
| abstract_inverted_index.context | 90 |
| abstract_inverted_index.corpus- | 186 |
| abstract_inverted_index.corpus. | 131 |
| abstract_inverted_index.critics | 155 |
| abstract_inverted_index.fulfill | 5 |
| abstract_inverted_index.further | 149 |
| abstract_inverted_index.however | 20, 38 |
| abstract_inverted_index.ideally | 14 |
| abstract_inverted_index.limited | 63 |
| abstract_inverted_index.metrics | 48, 83 |
| abstract_inverted_index.natural | 9 |
| abstract_inverted_index.offline | 47, 121, 153 |
| abstract_inverted_index.propose | 117 |
| abstract_inverted_index.success | 86 |
| abstract_inverted_index.systems | 2, 103 |
| abstract_inverted_index.through | 8 |
| abstract_inverted_index.trained | 158 |
| abstract_inverted_index.various | 175 |
| abstract_inverted_index.allowing | 170 |
| abstract_inverted_index.approach | 180 |
| abstract_inverted_index.corpora, | 52 |
| abstract_inverted_index.dialogue | 1, 69, 89, 125, 165, 171 |
| abstract_inverted_index.existing | 50, 81 |
| abstract_inverted_index.external | 168 |
| abstract_inverted_index.instance | 73 |
| abstract_inverted_index.language | 10 |
| abstract_inverted_index.learning | 123 |
| abstract_inverted_index.overlook | 88 |
| abstract_inverted_index.reliable | 99 |
| abstract_inverted_index.systems. | 70, 178 |
| abstract_inverted_index.utilized | 141 |
| abstract_inverted_index.Simulated | 32 |
| abstract_inverted_index.attaining | 190 |
| abstract_inverted_index.evaluated | 15 |
| abstract_inverted_index.evaluator | 134 |
| abstract_inverted_index.iteration | 27 |
| abstract_inverted_index.judgment, | 79 |
| abstract_inverted_index.practical | 56 |
| abstract_inverted_index.typically | 136 |
| abstract_inverted_index.Therefore, | 43 |
| abstract_inverted_index.correlated | 76 |
| abstract_inverted_index.evaluation | 126 |
| abstract_inverted_index.reflecting | 65 |
| abstract_inverted_index.comparisons | 173 |
| abstract_inverted_index.correlation | 109, 192 |
| abstract_inverted_index.development | 30, 40 |
| abstract_inverted_index.evaluators, | 169 |
| abstract_inverted_index.human-human | 51 |
| abstract_inverted_index.interactive | 201 |
| abstract_inverted_index.judgements, | 195 |
| abstract_inverted_index.judgements. | 112 |
| abstract_inverted_index.mismatches. | 91 |
| abstract_inverted_index.nontrivial. | 42 |
| abstract_inverted_index.performance | 67, 172 |
| abstract_inverted_index.researchers | 44 |
| abstract_inverted_index.alternative, | 37 |
| abstract_inverted_index.corpus-based | 82 |
| abstract_inverted_index.unattainable | 22 |
| abstract_inverted_index.Task-oriented | 0 |
| abstract_inverted_index.interactions. | 11 |
| abstract_inverted_index.optimization. | 144 |
| abstract_inverted_index.reinforcement | 122 |
| abstract_inverted_index.reproducible. | 59 |
| abstract_inverted_index.task-oriented | 102 |
| abstract_inverted_index.unfortunately | 62 |
| abstract_inverted_index.generalization | 106 |
| abstract_inverted_index.model-independent, | 188 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 7 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/16 |
| sustainable_development_goals[0].score | 0.550000011920929 |
| sustainable_development_goals[0].display_name | Peace, Justice and strong institutions |
| citation_normalized_percentile |