RISE: Leveraging Retrieval Techniques for Summarization Evaluation Article Swipe
David Uthus
,
Jianmo Ni
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.18653/v1/2023.findings-acl.865
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.18653/v1/2023.findings-acl.865
Evaluating automatically-generated text summaries is a challenging task. While there have been many interesting approaches, they still fall short of human evaluations. We present RISE, a new approach for evaluating summaries by leveraging techniques from information retrieval. RISE is first trained as a retrieval task using a dual-encoder retrieval setup, and can then be subsequently utilized for evaluating a generated summary given an input document, without gold reference summaries. RISE is especially well suited when working on new datasets where one may not have reference summaries available for evaluation. We conduct comprehensive experiments on the SummEval benchmark (Fabbri et al., 2021) and a long document summarization benchmark. The results show that RISE consistently achieves higher correlation with human evaluations compared to many past approaches to summarization evaluation. Furthermore, RISE also demonstrates data-efficiency and generalizability across languages.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.18653/v1/2023.findings-acl.865
- https://aclanthology.org/2023.findings-acl.865.pdf
- OA Status
- gold
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4385570309
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4385570309Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.18653/v1/2023.findings-acl.865Digital Object Identifier
- Title
-
RISE: Leveraging Retrieval Techniques for Summarization EvaluationWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-01-01Full publication date if available
- Authors
-
David Uthus, Jianmo NiList of authors in order
- Landing page
-
https://doi.org/10.18653/v1/2023.findings-acl.865Publisher landing page
- PDF URL
-
https://aclanthology.org/2023.findings-acl.865.pdfDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://aclanthology.org/2023.findings-acl.865.pdfDirect OA link when available
- Concepts
-
Automatic summarization, Computer science, Generalizability theory, Benchmark (surveying), Task (project management), Information retrieval, Encoder, Multi-document summarization, Artificial intelligence, Data mining, Geodesy, Statistics, Economics, Geography, Mathematics, Operating system, ManagementTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4385570309 |
|---|---|
| doi | https://doi.org/10.18653/v1/2023.findings-acl.865 |
| ids.doi | https://doi.org/10.18653/v1/2023.findings-acl.865 |
| ids.openalex | https://openalex.org/W4385570309 |
| fwci | 0.0 |
| type | article |
| title | RISE: Leveraging Retrieval Techniques for Summarization Evaluation |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | 13709 |
| biblio.first_page | 13697 |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9998000264167786 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9993000030517578 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T13083 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9930999875068665 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Advanced Text Analysis Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C170858558 |
| concepts[0].level | 2 |
| concepts[0].score | 0.9581271409988403 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q1394144 |
| concepts[0].display_name | Automatic summarization |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.8721643686294556 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C27158222 |
| concepts[2].level | 2 |
| concepts[2].score | 0.8561527729034424 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q5532422 |
| concepts[2].display_name | Generalizability theory |
| concepts[3].id | https://openalex.org/C185798385 |
| concepts[3].level | 2 |
| concepts[3].score | 0.8110018968582153 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q1161707 |
| concepts[3].display_name | Benchmark (surveying) |
| concepts[4].id | https://openalex.org/C2780451532 |
| concepts[4].level | 2 |
| concepts[4].score | 0.6691621541976929 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q759676 |
| concepts[4].display_name | Task (project management) |
| concepts[5].id | https://openalex.org/C23123220 |
| concepts[5].level | 1 |
| concepts[5].score | 0.6586121916770935 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q816826 |
| concepts[5].display_name | Information retrieval |
| concepts[6].id | https://openalex.org/C118505674 |
| concepts[6].level | 2 |
| concepts[6].score | 0.47341281175613403 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q42586063 |
| concepts[6].display_name | Encoder |
| concepts[7].id | https://openalex.org/C134714966 |
| concepts[7].level | 3 |
| concepts[7].score | 0.4635557234287262 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q6934448 |
| concepts[7].display_name | Multi-document summarization |
| concepts[8].id | https://openalex.org/C154945302 |
| concepts[8].level | 1 |
| concepts[8].score | 0.4361902177333832 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[8].display_name | Artificial intelligence |
| concepts[9].id | https://openalex.org/C124101348 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3746695816516876 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q172491 |
| concepts[9].display_name | Data mining |
| concepts[10].id | https://openalex.org/C13280743 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q131089 |
| concepts[10].display_name | Geodesy |
| concepts[11].id | https://openalex.org/C105795698 |
| concepts[11].level | 1 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[11].display_name | Statistics |
| concepts[12].id | https://openalex.org/C162324750 |
| concepts[12].level | 0 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q8134 |
| concepts[12].display_name | Economics |
| concepts[13].id | https://openalex.org/C205649164 |
| concepts[13].level | 0 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q1071 |
| concepts[13].display_name | Geography |
| concepts[14].id | https://openalex.org/C33923547 |
| concepts[14].level | 0 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[14].display_name | Mathematics |
| concepts[15].id | https://openalex.org/C111919701 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q9135 |
| concepts[15].display_name | Operating system |
| concepts[16].id | https://openalex.org/C187736073 |
| concepts[16].level | 1 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q2920921 |
| concepts[16].display_name | Management |
| keywords[0].id | https://openalex.org/keywords/automatic-summarization |
| keywords[0].score | 0.9581271409988403 |
| keywords[0].display_name | Automatic summarization |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.8721643686294556 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/generalizability-theory |
| keywords[2].score | 0.8561527729034424 |
| keywords[2].display_name | Generalizability theory |
| keywords[3].id | https://openalex.org/keywords/benchmark |
| keywords[3].score | 0.8110018968582153 |
| keywords[3].display_name | Benchmark (surveying) |
| keywords[4].id | https://openalex.org/keywords/task |
| keywords[4].score | 0.6691621541976929 |
| keywords[4].display_name | Task (project management) |
| keywords[5].id | https://openalex.org/keywords/information-retrieval |
| keywords[5].score | 0.6586121916770935 |
| keywords[5].display_name | Information retrieval |
| keywords[6].id | https://openalex.org/keywords/encoder |
| keywords[6].score | 0.47341281175613403 |
| keywords[6].display_name | Encoder |
| keywords[7].id | https://openalex.org/keywords/multi-document-summarization |
| keywords[7].score | 0.4635557234287262 |
| keywords[7].display_name | Multi-document summarization |
| keywords[8].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[8].score | 0.4361902177333832 |
| keywords[8].display_name | Artificial intelligence |
| keywords[9].id | https://openalex.org/keywords/data-mining |
| keywords[9].score | 0.3746695816516876 |
| keywords[9].display_name | Data mining |
| language | en |
| locations[0].id | doi:10.18653/v1/2023.findings-acl.865 |
| locations[0].is_oa | True |
| locations[0].source | |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://aclanthology.org/2023.findings-acl.865.pdf |
| locations[0].version | publishedVersion |
| locations[0].raw_type | proceedings-article |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | Findings of the Association for Computational Linguistics: ACL 2023 |
| locations[0].landing_page_url | https://doi.org/10.18653/v1/2023.findings-acl.865 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5026906002 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | David Uthus |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | David Uthus |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5077817759 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-6863-8073 |
| authorships[1].author.display_name | Jianmo Ni |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Jianmo Ni |
| authorships[1].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://aclanthology.org/2023.findings-acl.865.pdf |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | RISE: Leveraging Retrieval Techniques for Summarization Evaluation |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9998000264167786 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W2589098947, https://openalex.org/W2380641910, https://openalex.org/W2104677027, https://openalex.org/W65350537, https://openalex.org/W4200075185, https://openalex.org/W2093597205, https://openalex.org/W2389846579, https://openalex.org/W2392495745, https://openalex.org/W132250100, https://openalex.org/W2725657302 |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.18653/v1/2023.findings-acl.865 |
| best_oa_location.is_oa | True |
| best_oa_location.source | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://aclanthology.org/2023.findings-acl.865.pdf |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | proceedings-article |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | Findings of the Association for Computational Linguistics: ACL 2023 |
| best_oa_location.landing_page_url | https://doi.org/10.18653/v1/2023.findings-acl.865 |
| primary_location.id | doi:10.18653/v1/2023.findings-acl.865 |
| primary_location.is_oa | True |
| primary_location.source | |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://aclanthology.org/2023.findings-acl.865.pdf |
| primary_location.version | publishedVersion |
| primary_location.raw_type | proceedings-article |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | Findings of the Association for Computational Linguistics: ACL 2023 |
| primary_location.landing_page_url | https://doi.org/10.18653/v1/2023.findings-acl.865 |
| publication_date | 2023-01-01 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 5, 25, 42, 46, 58, 102 |
| abstract_inverted_index.We | 22, 89 |
| abstract_inverted_index.an | 62 |
| abstract_inverted_index.as | 41 |
| abstract_inverted_index.be | 53 |
| abstract_inverted_index.by | 31 |
| abstract_inverted_index.et | 98 |
| abstract_inverted_index.is | 4, 38, 70 |
| abstract_inverted_index.of | 19 |
| abstract_inverted_index.on | 76, 93 |
| abstract_inverted_index.to | 120, 124 |
| abstract_inverted_index.The | 107 |
| abstract_inverted_index.and | 50, 101, 132 |
| abstract_inverted_index.can | 51 |
| abstract_inverted_index.for | 28, 56, 87 |
| abstract_inverted_index.may | 81 |
| abstract_inverted_index.new | 26, 77 |
| abstract_inverted_index.not | 82 |
| abstract_inverted_index.one | 80 |
| abstract_inverted_index.the | 94 |
| abstract_inverted_index.RISE | 37, 69, 111, 128 |
| abstract_inverted_index.al., | 99 |
| abstract_inverted_index.also | 129 |
| abstract_inverted_index.been | 11 |
| abstract_inverted_index.fall | 17 |
| abstract_inverted_index.from | 34 |
| abstract_inverted_index.gold | 66 |
| abstract_inverted_index.have | 10, 83 |
| abstract_inverted_index.long | 103 |
| abstract_inverted_index.many | 12, 121 |
| abstract_inverted_index.past | 122 |
| abstract_inverted_index.show | 109 |
| abstract_inverted_index.task | 44 |
| abstract_inverted_index.text | 2 |
| abstract_inverted_index.that | 110 |
| abstract_inverted_index.then | 52 |
| abstract_inverted_index.they | 15 |
| abstract_inverted_index.well | 72 |
| abstract_inverted_index.when | 74 |
| abstract_inverted_index.with | 116 |
| abstract_inverted_index.2021) | 100 |
| abstract_inverted_index.RISE, | 24 |
| abstract_inverted_index.While | 8 |
| abstract_inverted_index.first | 39 |
| abstract_inverted_index.given | 61 |
| abstract_inverted_index.human | 20, 117 |
| abstract_inverted_index.input | 63 |
| abstract_inverted_index.short | 18 |
| abstract_inverted_index.still | 16 |
| abstract_inverted_index.task. | 7 |
| abstract_inverted_index.there | 9 |
| abstract_inverted_index.using | 45 |
| abstract_inverted_index.where | 79 |
| abstract_inverted_index.across | 134 |
| abstract_inverted_index.higher | 114 |
| abstract_inverted_index.setup, | 49 |
| abstract_inverted_index.suited | 73 |
| abstract_inverted_index.(Fabbri | 97 |
| abstract_inverted_index.conduct | 90 |
| abstract_inverted_index.present | 23 |
| abstract_inverted_index.results | 108 |
| abstract_inverted_index.summary | 60 |
| abstract_inverted_index.trained | 40 |
| abstract_inverted_index.without | 65 |
| abstract_inverted_index.working | 75 |
| abstract_inverted_index.SummEval | 95 |
| abstract_inverted_index.achieves | 113 |
| abstract_inverted_index.approach | 27 |
| abstract_inverted_index.compared | 119 |
| abstract_inverted_index.datasets | 78 |
| abstract_inverted_index.document | 104 |
| abstract_inverted_index.utilized | 55 |
| abstract_inverted_index.available | 86 |
| abstract_inverted_index.benchmark | 96 |
| abstract_inverted_index.document, | 64 |
| abstract_inverted_index.generated | 59 |
| abstract_inverted_index.reference | 67, 84 |
| abstract_inverted_index.retrieval | 43, 48 |
| abstract_inverted_index.summaries | 3, 30, 85 |
| abstract_inverted_index.Evaluating | 0 |
| abstract_inverted_index.approaches | 123 |
| abstract_inverted_index.benchmark. | 106 |
| abstract_inverted_index.especially | 71 |
| abstract_inverted_index.evaluating | 29, 57 |
| abstract_inverted_index.languages. | 135 |
| abstract_inverted_index.leveraging | 32 |
| abstract_inverted_index.retrieval. | 36 |
| abstract_inverted_index.summaries. | 68 |
| abstract_inverted_index.techniques | 33 |
| abstract_inverted_index.approaches, | 14 |
| abstract_inverted_index.challenging | 6 |
| abstract_inverted_index.correlation | 115 |
| abstract_inverted_index.evaluation. | 88, 126 |
| abstract_inverted_index.evaluations | 118 |
| abstract_inverted_index.experiments | 92 |
| abstract_inverted_index.information | 35 |
| abstract_inverted_index.interesting | 13 |
| abstract_inverted_index.Furthermore, | 127 |
| abstract_inverted_index.consistently | 112 |
| abstract_inverted_index.demonstrates | 130 |
| abstract_inverted_index.dual-encoder | 47 |
| abstract_inverted_index.evaluations. | 21 |
| abstract_inverted_index.subsequently | 54 |
| abstract_inverted_index.comprehensive | 91 |
| abstract_inverted_index.summarization | 105, 125 |
| abstract_inverted_index.data-efficiency | 131 |
| abstract_inverted_index.generalizability | 133 |
| abstract_inverted_index.automatically-generated | 1 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 2 |
| citation_normalized_percentile.value | 0.10488044 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |