Meta-evaluation of Conversational Search Evaluation Metrics Article Swipe
Zeyang Liu
,
Ke Zhou
,
Max L. Wilson
·
YOU?
·
· 2021
· Open Access
·
· DOI: https://doi.org/10.1145/3445029
YOU?
·
· 2021
· Open Access
·
· DOI: https://doi.org/10.1145/3445029
Conversational search systems, such as Google assistant and Microsoft Cortana, enable users to interact with search systems in multiple rounds through natural language dialogues. Evaluating such systems is very challenging, given that any natural language responses could be generated, and users commonly interact for multiple semantically coherent rounds to accomplish a search task. Although prior studies proposed many evaluation metrics, the extent of how those measures effectively capture user preference remain to be investigated. In this article, we systematically meta-evaluate a variety of conversational search metrics. We specifically study three perspectives on those metrics: (1) reliability : the ability to detect “actual” performance differences as opposed to those observed by chance; (2) fidelity : the ability to agree with ultimate user preference; and (3) intuitiveness : the ability to capture any property deemed important: adequacy, informativeness, and fluency in the context of conversational search. By conducting experiments on two test collections, we find that the performance of different metrics vary significantly across different scenarios, whereas consistent with prior studies, existing metrics only achieve weak correlation with ultimate user preference and satisfaction. METEOR is, comparatively speaking, the best existing single-turn metric considering all three perspectives. We also demonstrate that adapted session-based evaluation metrics can be used to measure multi-turn conversational search, achieving moderate concordance with user satisfaction. To our knowledge, our work establishes the most comprehensive meta-evaluation for conversational search to date.
Related Topics
Concepts
Computer science
Metric (unit)
Context (archaeology)
Task (project management)
Preference
Natural language understanding
Information retrieval
Fidelity
Relevance (law)
Natural language
Exploratory search
Consistency (knowledge bases)
Human–computer interaction
Artificial intelligence
Natural language processing
Law
Biology
Economics
Paleontology
Microeconomics
Telecommunications
Political science
Operations management
Management
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.1145/3445029
- OA Status
- green
- Cited By
- 39
- References
- 126
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W3157974261
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W3157974261Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.1145/3445029Digital Object Identifier
- Title
-
Meta-evaluation of Conversational Search Evaluation MetricsWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2021Year of publication
- Publication date
-
2021-09-01Full publication date if available
- Authors
-
Zeyang Liu, Ke Zhou, Max L. WilsonList of authors in order
- Landing page
-
https://doi.org/10.1145/3445029Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2104.13453Direct OA link when available
- Concepts
-
Computer science, Metric (unit), Context (archaeology), Task (project management), Preference, Natural language understanding, Information retrieval, Fidelity, Relevance (law), Natural language, Exploratory search, Consistency (knowledge bases), Human–computer interaction, Artificial intelligence, Natural language processing, Law, Biology, Economics, Paleontology, Microeconomics, Telecommunications, Political science, Operations management, ManagementTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
39Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 2, 2024: 25, 2023: 4, 2022: 3, 2021: 5Per-year citation counts (last 5 years)
- References (count)
-
126Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W3157974261 |
|---|---|
| doi | https://doi.org/10.1145/3445029 |
| ids.doi | https://doi.org/10.1145/3445029 |
| ids.mag | 3157974261 |
| ids.openalex | https://openalex.org/W3157974261 |
| fwci | 5.22083228 |
| type | article |
| title | Meta-evaluation of Conversational Search Evaluation Metrics |
| biblio.issue | 4 |
| biblio.volume | 39 |
| biblio.last_page | 42 |
| biblio.first_page | 1 |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9997000098228455 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10286 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9993000030517578 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1710 |
| topics[1].subfield.display_name | Information Systems |
| topics[1].display_name | Information Retrieval and Search Behavior |
| topics[2].id | https://openalex.org/T10203 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9984999895095825 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1710 |
| topics[2].subfield.display_name | Information Systems |
| topics[2].display_name | Recommender Systems and Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.7720790505409241 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C176217482 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5739269852638245 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q860554 |
| concepts[1].display_name | Metric (unit) |
| concepts[2].id | https://openalex.org/C2779343474 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5520659685134888 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q3109175 |
| concepts[2].display_name | Context (archaeology) |
| concepts[3].id | https://openalex.org/C2780451532 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5340285301208496 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q759676 |
| concepts[3].display_name | Task (project management) |
| concepts[4].id | https://openalex.org/C2781249084 |
| concepts[4].level | 2 |
| concepts[4].score | 0.4790983498096466 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q908656 |
| concepts[4].display_name | Preference |
| concepts[5].id | https://openalex.org/C2779439875 |
| concepts[5].level | 3 |
| concepts[5].score | 0.4685918688774109 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q1078276 |
| concepts[5].display_name | Natural language understanding |
| concepts[6].id | https://openalex.org/C23123220 |
| concepts[6].level | 1 |
| concepts[6].score | 0.46340513229370117 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q816826 |
| concepts[6].display_name | Information retrieval |
| concepts[7].id | https://openalex.org/C2776459999 |
| concepts[7].level | 2 |
| concepts[7].score | 0.45503804087638855 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q2119376 |
| concepts[7].display_name | Fidelity |
| concepts[8].id | https://openalex.org/C158154518 |
| concepts[8].level | 2 |
| concepts[8].score | 0.4433172941207886 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q7310970 |
| concepts[8].display_name | Relevance (law) |
| concepts[9].id | https://openalex.org/C195324797 |
| concepts[9].level | 2 |
| concepts[9].score | 0.4410404860973358 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q33742 |
| concepts[9].display_name | Natural language |
| concepts[10].id | https://openalex.org/C2777866876 |
| concepts[10].level | 2 |
| concepts[10].score | 0.4405614137649536 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q5421358 |
| concepts[10].display_name | Exploratory search |
| concepts[11].id | https://openalex.org/C2776436953 |
| concepts[11].level | 2 |
| concepts[11].score | 0.4152718782424927 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q5163215 |
| concepts[11].display_name | Consistency (knowledge bases) |
| concepts[12].id | https://openalex.org/C107457646 |
| concepts[12].level | 1 |
| concepts[12].score | 0.4071623682975769 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q207434 |
| concepts[12].display_name | Human–computer interaction |
| concepts[13].id | https://openalex.org/C154945302 |
| concepts[13].level | 1 |
| concepts[13].score | 0.33881425857543945 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[13].display_name | Artificial intelligence |
| concepts[14].id | https://openalex.org/C204321447 |
| concepts[14].level | 1 |
| concepts[14].score | 0.338152676820755 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[14].display_name | Natural language processing |
| concepts[15].id | https://openalex.org/C199539241 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q7748 |
| concepts[15].display_name | Law |
| concepts[16].id | https://openalex.org/C86803240 |
| concepts[16].level | 0 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q420 |
| concepts[16].display_name | Biology |
| concepts[17].id | https://openalex.org/C162324750 |
| concepts[17].level | 0 |
| concepts[17].score | 0.0 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q8134 |
| concepts[17].display_name | Economics |
| concepts[18].id | https://openalex.org/C151730666 |
| concepts[18].level | 1 |
| concepts[18].score | 0.0 |
| concepts[18].wikidata | https://www.wikidata.org/wiki/Q7205 |
| concepts[18].display_name | Paleontology |
| concepts[19].id | https://openalex.org/C175444787 |
| concepts[19].level | 1 |
| concepts[19].score | 0.0 |
| concepts[19].wikidata | https://www.wikidata.org/wiki/Q39072 |
| concepts[19].display_name | Microeconomics |
| concepts[20].id | https://openalex.org/C76155785 |
| concepts[20].level | 1 |
| concepts[20].score | 0.0 |
| concepts[20].wikidata | https://www.wikidata.org/wiki/Q418 |
| concepts[20].display_name | Telecommunications |
| concepts[21].id | https://openalex.org/C17744445 |
| concepts[21].level | 0 |
| concepts[21].score | 0.0 |
| concepts[21].wikidata | https://www.wikidata.org/wiki/Q36442 |
| concepts[21].display_name | Political science |
| concepts[22].id | https://openalex.org/C21547014 |
| concepts[22].level | 1 |
| concepts[22].score | 0.0 |
| concepts[22].wikidata | https://www.wikidata.org/wiki/Q1423657 |
| concepts[22].display_name | Operations management |
| concepts[23].id | https://openalex.org/C187736073 |
| concepts[23].level | 1 |
| concepts[23].score | 0.0 |
| concepts[23].wikidata | https://www.wikidata.org/wiki/Q2920921 |
| concepts[23].display_name | Management |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.7720790505409241 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/metric |
| keywords[1].score | 0.5739269852638245 |
| keywords[1].display_name | Metric (unit) |
| keywords[2].id | https://openalex.org/keywords/context |
| keywords[2].score | 0.5520659685134888 |
| keywords[2].display_name | Context (archaeology) |
| keywords[3].id | https://openalex.org/keywords/task |
| keywords[3].score | 0.5340285301208496 |
| keywords[3].display_name | Task (project management) |
| keywords[4].id | https://openalex.org/keywords/preference |
| keywords[4].score | 0.4790983498096466 |
| keywords[4].display_name | Preference |
| keywords[5].id | https://openalex.org/keywords/natural-language-understanding |
| keywords[5].score | 0.4685918688774109 |
| keywords[5].display_name | Natural language understanding |
| keywords[6].id | https://openalex.org/keywords/information-retrieval |
| keywords[6].score | 0.46340513229370117 |
| keywords[6].display_name | Information retrieval |
| keywords[7].id | https://openalex.org/keywords/fidelity |
| keywords[7].score | 0.45503804087638855 |
| keywords[7].display_name | Fidelity |
| keywords[8].id | https://openalex.org/keywords/relevance |
| keywords[8].score | 0.4433172941207886 |
| keywords[8].display_name | Relevance (law) |
| keywords[9].id | https://openalex.org/keywords/natural-language |
| keywords[9].score | 0.4410404860973358 |
| keywords[9].display_name | Natural language |
| keywords[10].id | https://openalex.org/keywords/exploratory-search |
| keywords[10].score | 0.4405614137649536 |
| keywords[10].display_name | Exploratory search |
| keywords[11].id | https://openalex.org/keywords/consistency |
| keywords[11].score | 0.4152718782424927 |
| keywords[11].display_name | Consistency (knowledge bases) |
| keywords[12].id | https://openalex.org/keywords/human–computer-interaction |
| keywords[12].score | 0.4071623682975769 |
| keywords[12].display_name | Human–computer interaction |
| keywords[13].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[13].score | 0.33881425857543945 |
| keywords[13].display_name | Artificial intelligence |
| keywords[14].id | https://openalex.org/keywords/natural-language-processing |
| keywords[14].score | 0.338152676820755 |
| keywords[14].display_name | Natural language processing |
| language | en |
| locations[0].id | doi:10.1145/3445029 |
| locations[0].is_oa | False |
| locations[0].source.id | https://openalex.org/S4394735545 |
| locations[0].source.issn | 1046-8188, 1558-2868 |
| locations[0].source.type | journal |
| locations[0].source.is_oa | False |
| locations[0].source.issn_l | 1046-8188 |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | ACM Transactions on Information Systems |
| locations[0].source.host_organization | |
| locations[0].source.host_organization_name | |
| locations[0].license | |
| locations[0].pdf_url | |
| locations[0].version | publishedVersion |
| locations[0].raw_type | journal-article |
| locations[0].license_id | |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | ACM Transactions on Information Systems |
| locations[0].landing_page_url | https://doi.org/10.1145/3445029 |
| locations[1].id | pmh:oai:arXiv.org:2104.13453 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | https://arxiv.org/pdf/2104.13453 |
| locations[1].version | submittedVersion |
| locations[1].raw_type | text |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | False |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | http://arxiv.org/abs/2104.13453 |
| indexed_in | arxiv, crossref |
| authorships[0].author.id | https://openalex.org/A5054589937 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-3553-3793 |
| authorships[0].author.display_name | Zeyang Liu |
| authorships[0].countries | GB |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I142263535 |
| authorships[0].affiliations[0].raw_affiliation_string | University of Nottingham, Nottingham, UK |
| authorships[0].institutions[0].id | https://openalex.org/I142263535 |
| authorships[0].institutions[0].ror | https://ror.org/01ee9ar58 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I142263535 |
| authorships[0].institutions[0].country_code | GB |
| authorships[0].institutions[0].display_name | University of Nottingham |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Zeyang Liu |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | University of Nottingham, Nottingham, UK |
| authorships[1].author.id | https://openalex.org/A5101500495 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Ke Zhou |
| authorships[1].countries | GB |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I142263535, https://openalex.org/I4210098141 |
| authorships[1].affiliations[0].raw_affiliation_string | University of Nottingham & Nokia Bell Labs, Cambridge, UK |
| authorships[1].institutions[0].id | https://openalex.org/I4210098141 |
| authorships[1].institutions[0].ror | https://ror.org/00zpf0626 |
| authorships[1].institutions[0].type | company |
| authorships[1].institutions[0].lineage | https://openalex.org/I2738502077, https://openalex.org/I4210098141 |
| authorships[1].institutions[0].country_code | GB |
| authorships[1].institutions[0].display_name | Nokia (United Kingdom) |
| authorships[1].institutions[1].id | https://openalex.org/I142263535 |
| authorships[1].institutions[1].ror | https://ror.org/01ee9ar58 |
| authorships[1].institutions[1].type | education |
| authorships[1].institutions[1].lineage | https://openalex.org/I142263535 |
| authorships[1].institutions[1].country_code | GB |
| authorships[1].institutions[1].display_name | University of Nottingham |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Ke Zhou |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | University of Nottingham & Nokia Bell Labs, Cambridge, UK |
| authorships[2].author.id | https://openalex.org/A5088221028 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-3515-6633 |
| authorships[2].author.display_name | Max L. Wilson |
| authorships[2].countries | GB |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I142263535 |
| authorships[2].affiliations[0].raw_affiliation_string | University of Nottingham, Nottingham, UK |
| authorships[2].institutions[0].id | https://openalex.org/I142263535 |
| authorships[2].institutions[0].ror | https://ror.org/01ee9ar58 |
| authorships[2].institutions[0].type | education |
| authorships[2].institutions[0].lineage | https://openalex.org/I142263535 |
| authorships[2].institutions[0].country_code | GB |
| authorships[2].institutions[0].display_name | University of Nottingham |
| authorships[2].author_position | last |
| authorships[2].raw_author_name | Max L. Wilson |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | University of Nottingham, Nottingham, UK |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2104.13453 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Meta-evaluation of Conversational Search Evaluation Metrics |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9997000098228455 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W2367925007, https://openalex.org/W4288263119, https://openalex.org/W3015724364, https://openalex.org/W2967994095, https://openalex.org/W4285240985, https://openalex.org/W2900126711, https://openalex.org/W4225162083, https://openalex.org/W4286930972, https://openalex.org/W3202115945, https://openalex.org/W2542958340 |
| cited_by_count | 39 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 2 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 25 |
| counts_by_year[2].year | 2023 |
| counts_by_year[2].cited_by_count | 4 |
| counts_by_year[3].year | 2022 |
| counts_by_year[3].cited_by_count | 3 |
| counts_by_year[4].year | 2021 |
| counts_by_year[4].cited_by_count | 5 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2104.13453 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2104.13453 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2104.13453 |
| primary_location.id | doi:10.1145/3445029 |
| primary_location.is_oa | False |
| primary_location.source.id | https://openalex.org/S4394735545 |
| primary_location.source.issn | 1046-8188, 1558-2868 |
| primary_location.source.type | journal |
| primary_location.source.is_oa | False |
| primary_location.source.issn_l | 1046-8188 |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | ACM Transactions on Information Systems |
| primary_location.source.host_organization | |
| primary_location.source.host_organization_name | |
| primary_location.license | |
| primary_location.pdf_url | |
| primary_location.version | publishedVersion |
| primary_location.raw_type | journal-article |
| primary_location.license_id | |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | ACM Transactions on Information Systems |
| primary_location.landing_page_url | https://doi.org/10.1145/3445029 |
| publication_date | 2021-09-01 |
| publication_year | 2021 |
| referenced_works | https://openalex.org/W1551409364, https://openalex.org/W2105157020, https://openalex.org/W3131173775, https://openalex.org/W2017292914, https://openalex.org/W2113640060, https://openalex.org/W2751698296, https://openalex.org/W2740916867, https://openalex.org/W2983435475, https://openalex.org/W1571268342, https://openalex.org/W1999822363, https://openalex.org/W2963681593, https://openalex.org/W2944069152, https://openalex.org/W2089150068, https://openalex.org/W2080068076, https://openalex.org/W2898031744, https://openalex.org/W2069870183, https://openalex.org/W2324619102, https://openalex.org/W2158254843, https://openalex.org/W2889448364, https://openalex.org/W1999444901, https://openalex.org/W2035569891, https://openalex.org/W2341328702, https://openalex.org/W2331728731, https://openalex.org/W1983578042, https://openalex.org/W2976801151, https://openalex.org/W2798482129, https://openalex.org/W1968927634, https://openalex.org/W2101105183, https://openalex.org/W2799195065, https://openalex.org/W2160471965, https://openalex.org/W2590822507, https://openalex.org/W2024932032, https://openalex.org/W2058896506, https://openalex.org/W2110202502, https://openalex.org/W1988619666, https://openalex.org/W2004918107, https://openalex.org/W2955792106, https://openalex.org/W2116008435, https://openalex.org/W2517428641, https://openalex.org/W2024635814, https://openalex.org/W1552182777, https://openalex.org/W2012430612, https://openalex.org/W2741363662, https://openalex.org/W2791808303, https://openalex.org/W2169783907, https://openalex.org/W3203363246, https://openalex.org/W2798392716, https://openalex.org/W2251818205, https://openalex.org/W2986867746, https://openalex.org/W2163632538, https://openalex.org/W1984827663, https://openalex.org/W2561368124, https://openalex.org/W2600463316, https://openalex.org/W2963790827, https://openalex.org/W2130570284, https://openalex.org/W2251251208, https://openalex.org/W3022041859, https://openalex.org/W2123301721, https://openalex.org/W3022187094, https://openalex.org/W2963527228, https://openalex.org/W3104405162, https://openalex.org/W2154652894, https://openalex.org/W2798526799, https://openalex.org/W245306944, https://openalex.org/W2951813108, https://openalex.org/W2131133093, https://openalex.org/W836999996, https://openalex.org/W3022592851, https://openalex.org/W2892228078, https://openalex.org/W1518951372, https://openalex.org/W2949888546, https://openalex.org/W2936695845, https://openalex.org/W2996403597, https://openalex.org/W2949954138, https://openalex.org/W3035656139, https://openalex.org/W3023366413, https://openalex.org/W3034808773, https://openalex.org/W2986292373, https://openalex.org/W2890969459, https://openalex.org/W2765617518, https://openalex.org/W2427527485, https://openalex.org/W2964039645, https://openalex.org/W2962854379, https://openalex.org/W2786983967, https://openalex.org/W2898875342, https://openalex.org/W2964006684, https://openalex.org/W2761590056, https://openalex.org/W2896457183, https://openalex.org/W2539809671, https://openalex.org/W2328886022, https://openalex.org/W2963748441, https://openalex.org/W3014461195, https://openalex.org/W2952630045, https://openalex.org/W3102132500, https://openalex.org/W2563351168, https://openalex.org/W3125764608, https://openalex.org/W4301406666, https://openalex.org/W2950902819, https://openalex.org/W3102777668, https://openalex.org/W2962786758, https://openalex.org/W4287991009, https://openalex.org/W3101567279, https://openalex.org/W4287870915, https://openalex.org/W2181584843, https://openalex.org/W1967370444, https://openalex.org/W2130942839, https://openalex.org/W2963672599, https://openalex.org/W3005445152, https://openalex.org/W2963341956, https://openalex.org/W2137274315, https://openalex.org/W2584220694, https://openalex.org/W2140054881, https://openalex.org/W2250645967, https://openalex.org/W2964178377, https://openalex.org/W2941500661, https://openalex.org/W2964703418, https://openalex.org/W1654173042, https://openalex.org/W2564590796, https://openalex.org/W10957333, https://openalex.org/W2963903950, https://openalex.org/W2995951327, https://openalex.org/W2137607259, https://openalex.org/W2891416139, https://openalex.org/W2399060250, https://openalex.org/W2891732163, https://openalex.org/W2951883832 |
| referenced_works_count | 126 |
| abstract_inverted_index.: | 96, 113, 125 |
| abstract_inverted_index.a | 50, 80 |
| abstract_inverted_index.By | 144 |
| abstract_inverted_index.In | 74 |
| abstract_inverted_index.To | 216 |
| abstract_inverted_index.We | 86, 194 |
| abstract_inverted_index.as | 4, 104 |
| abstract_inverted_index.be | 37, 72, 203 |
| abstract_inverted_index.by | 109 |
| abstract_inverted_index.in | 17, 138 |
| abstract_inverted_index.is | 27 |
| abstract_inverted_index.of | 62, 82, 141, 156 |
| abstract_inverted_index.on | 91, 147 |
| abstract_inverted_index.to | 12, 48, 71, 99, 106, 116, 128, 205, 229 |
| abstract_inverted_index.we | 77, 151 |
| abstract_inverted_index.(1) | 94 |
| abstract_inverted_index.(2) | 111 |
| abstract_inverted_index.(3) | 123 |
| abstract_inverted_index.all | 191 |
| abstract_inverted_index.and | 7, 39, 122, 136, 179 |
| abstract_inverted_index.any | 32, 130 |
| abstract_inverted_index.can | 202 |
| abstract_inverted_index.for | 43, 226 |
| abstract_inverted_index.how | 63 |
| abstract_inverted_index.is, | 182 |
| abstract_inverted_index.our | 217, 219 |
| abstract_inverted_index.the | 60, 97, 114, 126, 139, 154, 185, 222 |
| abstract_inverted_index.two | 148 |
| abstract_inverted_index.also | 195 |
| abstract_inverted_index.best | 186 |
| abstract_inverted_index.find | 152 |
| abstract_inverted_index.many | 57 |
| abstract_inverted_index.most | 223 |
| abstract_inverted_index.only | 171 |
| abstract_inverted_index.such | 3, 25 |
| abstract_inverted_index.test | 149 |
| abstract_inverted_index.that | 31, 153, 197 |
| abstract_inverted_index.this | 75 |
| abstract_inverted_index.used | 204 |
| abstract_inverted_index.user | 68, 120, 177, 214 |
| abstract_inverted_index.vary | 159 |
| abstract_inverted_index.very | 28 |
| abstract_inverted_index.weak | 173 |
| abstract_inverted_index.with | 14, 118, 166, 175, 213 |
| abstract_inverted_index.work | 220 |
| abstract_inverted_index.agree | 117 |
| abstract_inverted_index.could | 36 |
| abstract_inverted_index.date. | 230 |
| abstract_inverted_index.given | 30 |
| abstract_inverted_index.prior | 54, 167 |
| abstract_inverted_index.study | 88 |
| abstract_inverted_index.task. | 52 |
| abstract_inverted_index.those | 64, 92, 107 |
| abstract_inverted_index.three | 89, 192 |
| abstract_inverted_index.users | 11, 40 |
| abstract_inverted_index.Google | 5 |
| abstract_inverted_index.METEOR | 181 |
| abstract_inverted_index.across | 161 |
| abstract_inverted_index.deemed | 132 |
| abstract_inverted_index.detect | 100 |
| abstract_inverted_index.enable | 10 |
| abstract_inverted_index.extent | 61 |
| abstract_inverted_index.metric | 189 |
| abstract_inverted_index.remain | 70 |
| abstract_inverted_index.rounds | 19, 47 |
| abstract_inverted_index.search | 1, 15, 51, 84, 228 |
| abstract_inverted_index.ability | 98, 115, 127 |
| abstract_inverted_index.achieve | 172 |
| abstract_inverted_index.adapted | 198 |
| abstract_inverted_index.capture | 67, 129 |
| abstract_inverted_index.chance; | 110 |
| abstract_inverted_index.context | 140 |
| abstract_inverted_index.fluency | 137 |
| abstract_inverted_index.measure | 206 |
| abstract_inverted_index.metrics | 158, 170, 201 |
| abstract_inverted_index.natural | 21, 33 |
| abstract_inverted_index.opposed | 105 |
| abstract_inverted_index.search, | 209 |
| abstract_inverted_index.search. | 143 |
| abstract_inverted_index.studies | 55 |
| abstract_inverted_index.systems | 16, 26 |
| abstract_inverted_index.through | 20 |
| abstract_inverted_index.variety | 81 |
| abstract_inverted_index.whereas | 164 |
| abstract_inverted_index.Although | 53 |
| abstract_inverted_index.Cortana, | 9 |
| abstract_inverted_index.article, | 76 |
| abstract_inverted_index.coherent | 46 |
| abstract_inverted_index.commonly | 41 |
| abstract_inverted_index.existing | 169, 187 |
| abstract_inverted_index.fidelity | 112 |
| abstract_inverted_index.interact | 13, 42 |
| abstract_inverted_index.language | 22, 34 |
| abstract_inverted_index.measures | 65 |
| abstract_inverted_index.metrics, | 59 |
| abstract_inverted_index.metrics. | 85 |
| abstract_inverted_index.metrics: | 93 |
| abstract_inverted_index.moderate | 211 |
| abstract_inverted_index.multiple | 18, 44 |
| abstract_inverted_index.observed | 108 |
| abstract_inverted_index.property | 131 |
| abstract_inverted_index.proposed | 56 |
| abstract_inverted_index.studies, | 168 |
| abstract_inverted_index.systems, | 2 |
| abstract_inverted_index.ultimate | 119, 176 |
| abstract_inverted_index.Microsoft | 8 |
| abstract_inverted_index.achieving | 210 |
| abstract_inverted_index.adequacy, | 134 |
| abstract_inverted_index.assistant | 6 |
| abstract_inverted_index.different | 157, 162 |
| abstract_inverted_index.responses | 35 |
| abstract_inverted_index.speaking, | 184 |
| abstract_inverted_index.Evaluating | 24 |
| abstract_inverted_index.accomplish | 49 |
| abstract_inverted_index.conducting | 145 |
| abstract_inverted_index.consistent | 165 |
| abstract_inverted_index.dialogues. | 23 |
| abstract_inverted_index.evaluation | 58, 200 |
| abstract_inverted_index.generated, | 38 |
| abstract_inverted_index.important: | 133 |
| abstract_inverted_index.knowledge, | 218 |
| abstract_inverted_index.multi-turn | 207 |
| abstract_inverted_index.preference | 69, 178 |
| abstract_inverted_index.scenarios, | 163 |
| abstract_inverted_index.concordance | 212 |
| abstract_inverted_index.considering | 190 |
| abstract_inverted_index.correlation | 174 |
| abstract_inverted_index.demonstrate | 196 |
| abstract_inverted_index.differences | 103 |
| abstract_inverted_index.effectively | 66 |
| abstract_inverted_index.establishes | 221 |
| abstract_inverted_index.experiments | 146 |
| abstract_inverted_index.performance | 102, 155 |
| abstract_inverted_index.preference; | 121 |
| abstract_inverted_index.reliability | 95 |
| abstract_inverted_index.single-turn | 188 |
| abstract_inverted_index.challenging, | 29 |
| abstract_inverted_index.collections, | 150 |
| abstract_inverted_index.perspectives | 90 |
| abstract_inverted_index.semantically | 45 |
| abstract_inverted_index.specifically | 87 |
| abstract_inverted_index.“actual” | 101 |
| abstract_inverted_index.comparatively | 183 |
| abstract_inverted_index.comprehensive | 224 |
| abstract_inverted_index.intuitiveness | 124 |
| abstract_inverted_index.investigated. | 73 |
| abstract_inverted_index.meta-evaluate | 79 |
| abstract_inverted_index.perspectives. | 193 |
| abstract_inverted_index.satisfaction. | 180, 215 |
| abstract_inverted_index.session-based | 199 |
| abstract_inverted_index.significantly | 160 |
| abstract_inverted_index.Conversational | 0 |
| abstract_inverted_index.conversational | 83, 142, 208, 227 |
| abstract_inverted_index.systematically | 78 |
| abstract_inverted_index.meta-evaluation | 225 |
| abstract_inverted_index.informativeness, | 135 |
| cited_by_percentile_year.max | 100 |
| cited_by_percentile_year.min | 95 |
| countries_distinct_count | 1 |
| institutions_distinct_count | 3 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.8199999928474426 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile.value | 0.96229667 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | True |