When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications Article Swipe
Kevin Pei
,
Ishan Jindal
,
Kevin Chen–Chuan Chang
,
ChengXiang Zhai
,
Yunyao Li
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.18653/v1/2023.acl-long.53
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.18653/v1/2023.acl-long.53
Open Information Extraction (OpenIE) has been used in the pipelines of various NLP tasks. Unfortunately, there is no clear consensus on which models to use in which tasks. Muddying things further is the lack of comparisons that take differing training sets into account. In this paper, we present an application-focused empirical survey of neural OpenIE models, training sets, and benchmarks in an effort to help users choose the most suitable OpenIE systems for their applications. We find that the different assumptions made by different models and datasets have a statistically significant effect on performance, making it important to choose the most appropriate model for one’s applications. We demonstrate the applicability of our recommendations on a downstream Complex QA application.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.18653/v1/2023.acl-long.53
- https://aclanthology.org/2023.acl-long.53.pdf
- OA Status
- gold
- Cited By
- 2
- References
- 33
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4385571433
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4385571433Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.18653/v1/2023.acl-long.53Digital Object Identifier
- Title
-
When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream ApplicationsWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-01-01Full publication date if available
- Authors
-
Kevin Pei, Ishan Jindal, Kevin Chen–Chuan Chang, ChengXiang Zhai, Yunyao LiList of authors in order
- Landing page
-
https://doi.org/10.18653/v1/2023.acl-long.53Publisher landing page
- PDF URL
-
https://aclanthology.org/2023.acl-long.53.pdfDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://aclanthology.org/2023.acl-long.53.pdfDirect OA link when available
- Concepts
-
Computer science, Downstream (manufacturing), Empirical research, Machine learning, Artificial intelligence, Pipeline transport, Data science, Data mining, Engineering, Operations management, Philosophy, Epistemology, Environmental engineeringTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
2Total citation count in OpenAlex
- Citations by year (recent)
-
2023: 2Per-year citation counts (last 5 years)
- References (count)
-
33Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4385571433 |
|---|---|
| doi | https://doi.org/10.18653/v1/2023.acl-long.53 |
| ids.doi | https://doi.org/10.18653/v1/2023.acl-long.53 |
| ids.openalex | https://openalex.org/W4385571433 |
| fwci | 0.51088578 |
| type | article |
| title | When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | 949 |
| biblio.first_page | 929 |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9975000023841858 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9854000210762024 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T10260 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9815999865531921 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1710 |
| topics[2].subfield.display_name | Information Systems |
| topics[2].display_name | Software Engineering Research |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.8249739408493042 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C2776207758 |
| concepts[1].level | 2 |
| concepts[1].score | 0.8222993612289429 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q5303302 |
| concepts[1].display_name | Downstream (manufacturing) |
| concepts[2].id | https://openalex.org/C120936955 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5256938934326172 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q2155640 |
| concepts[2].display_name | Empirical research |
| concepts[3].id | https://openalex.org/C119857082 |
| concepts[3].level | 1 |
| concepts[3].score | 0.4929814338684082 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[3].display_name | Machine learning |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.48867467045783997 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C175309249 |
| concepts[5].level | 2 |
| concepts[5].score | 0.4186646342277527 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q725864 |
| concepts[5].display_name | Pipeline transport |
| concepts[6].id | https://openalex.org/C2522767166 |
| concepts[6].level | 1 |
| concepts[6].score | 0.36386409401893616 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q2374463 |
| concepts[6].display_name | Data science |
| concepts[7].id | https://openalex.org/C124101348 |
| concepts[7].level | 1 |
| concepts[7].score | 0.36002352833747864 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q172491 |
| concepts[7].display_name | Data mining |
| concepts[8].id | https://openalex.org/C127413603 |
| concepts[8].level | 0 |
| concepts[8].score | 0.08750954270362854 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[8].display_name | Engineering |
| concepts[9].id | https://openalex.org/C21547014 |
| concepts[9].level | 1 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q1423657 |
| concepts[9].display_name | Operations management |
| concepts[10].id | https://openalex.org/C138885662 |
| concepts[10].level | 0 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[10].display_name | Philosophy |
| concepts[11].id | https://openalex.org/C111472728 |
| concepts[11].level | 1 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q9471 |
| concepts[11].display_name | Epistemology |
| concepts[12].id | https://openalex.org/C87717796 |
| concepts[12].level | 1 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q146326 |
| concepts[12].display_name | Environmental engineering |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.8249739408493042 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/downstream |
| keywords[1].score | 0.8222993612289429 |
| keywords[1].display_name | Downstream (manufacturing) |
| keywords[2].id | https://openalex.org/keywords/empirical-research |
| keywords[2].score | 0.5256938934326172 |
| keywords[2].display_name | Empirical research |
| keywords[3].id | https://openalex.org/keywords/machine-learning |
| keywords[3].score | 0.4929814338684082 |
| keywords[3].display_name | Machine learning |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.48867467045783997 |
| keywords[4].display_name | Artificial intelligence |
| keywords[5].id | https://openalex.org/keywords/pipeline-transport |
| keywords[5].score | 0.4186646342277527 |
| keywords[5].display_name | Pipeline transport |
| keywords[6].id | https://openalex.org/keywords/data-science |
| keywords[6].score | 0.36386409401893616 |
| keywords[6].display_name | Data science |
| keywords[7].id | https://openalex.org/keywords/data-mining |
| keywords[7].score | 0.36002352833747864 |
| keywords[7].display_name | Data mining |
| keywords[8].id | https://openalex.org/keywords/engineering |
| keywords[8].score | 0.08750954270362854 |
| keywords[8].display_name | Engineering |
| language | en |
| locations[0].id | doi:10.18653/v1/2023.acl-long.53 |
| locations[0].is_oa | True |
| locations[0].source | |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://aclanthology.org/2023.acl-long.53.pdf |
| locations[0].version | publishedVersion |
| locations[0].raw_type | proceedings-article |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |
| locations[0].landing_page_url | https://doi.org/10.18653/v1/2023.acl-long.53 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5108296434 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Kevin Pei |
| authorships[0].countries | US |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I157725225 |
| authorships[0].affiliations[0].raw_affiliation_string | University of Illinois at Urbana-Champaign, |
| authorships[0].institutions[0].id | https://openalex.org/I157725225 |
| authorships[0].institutions[0].ror | https://ror.org/047426m28 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I157725225 |
| authorships[0].institutions[0].country_code | US |
| authorships[0].institutions[0].display_name | University of Illinois Urbana-Champaign |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Kevin Pei |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | University of Illinois at Urbana-Champaign, |
| authorships[1].author.id | https://openalex.org/A5077046363 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-8854-4189 |
| authorships[1].author.display_name | Ishan Jindal |
| authorships[1].affiliations[0].raw_affiliation_string | IBM Research, |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Ishan Jindal |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | IBM Research, |
| authorships[2].author.id | https://openalex.org/A5101880377 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-0997-6803 |
| authorships[2].author.display_name | Kevin Chen–Chuan Chang |
| authorships[2].countries | US |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I157725225 |
| authorships[2].affiliations[0].raw_affiliation_string | University of Illinois at Urbana-Champaign, |
| authorships[2].institutions[0].id | https://openalex.org/I157725225 |
| authorships[2].institutions[0].ror | https://ror.org/047426m28 |
| authorships[2].institutions[0].type | education |
| authorships[2].institutions[0].lineage | https://openalex.org/I157725225 |
| authorships[2].institutions[0].country_code | US |
| authorships[2].institutions[0].display_name | University of Illinois Urbana-Champaign |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Kevin Chen-Chuan Chang |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | University of Illinois at Urbana-Champaign, |
| authorships[3].author.id | https://openalex.org/A5028518494 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-6434-3702 |
| authorships[3].author.display_name | ChengXiang Zhai |
| authorships[3].countries | US |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I157725225 |
| authorships[3].affiliations[0].raw_affiliation_string | University of Illinois at Urbana-Champaign, |
| authorships[3].institutions[0].id | https://openalex.org/I157725225 |
| authorships[3].institutions[0].ror | https://ror.org/047426m28 |
| authorships[3].institutions[0].type | education |
| authorships[3].institutions[0].lineage | https://openalex.org/I157725225 |
| authorships[3].institutions[0].country_code | US |
| authorships[3].institutions[0].display_name | University of Illinois Urbana-Champaign |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | ChengXiang Zhai |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | University of Illinois at Urbana-Champaign, |
| authorships[4].author.id | https://openalex.org/A5102944075 |
| authorships[4].author.orcid | https://orcid.org/0009-0002-0814-4634 |
| authorships[4].author.display_name | Yunyao Li |
| authorships[4].countries | GB |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I4210107260 |
| authorships[4].affiliations[0].raw_affiliation_string | Apple |
| authorships[4].institutions[0].id | https://openalex.org/I4210107260 |
| authorships[4].institutions[0].ror | https://ror.org/01vpeym60 |
| authorships[4].institutions[0].type | company |
| authorships[4].institutions[0].lineage | https://openalex.org/I4210107260 |
| authorships[4].institutions[0].country_code | GB |
| authorships[4].institutions[0].display_name | Apple (United Kingdom) |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Yunyao Li |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | Apple |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://aclanthology.org/2023.acl-long.53.pdf |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9975000023841858 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W1583765404, https://openalex.org/W4214653257, https://openalex.org/W3040203686, https://openalex.org/W4249524554, https://openalex.org/W2521424917, https://openalex.org/W2349021146, https://openalex.org/W35583307, https://openalex.org/W2436192316, https://openalex.org/W2970287324, https://openalex.org/W1995889332 |
| cited_by_count | 2 |
| counts_by_year[0].year | 2023 |
| counts_by_year[0].cited_by_count | 2 |
| locations_count | 1 |
| best_oa_location.id | doi:10.18653/v1/2023.acl-long.53 |
| best_oa_location.is_oa | True |
| best_oa_location.source | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://aclanthology.org/2023.acl-long.53.pdf |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | proceedings-article |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |
| best_oa_location.landing_page_url | https://doi.org/10.18653/v1/2023.acl-long.53 |
| primary_location.id | doi:10.18653/v1/2023.acl-long.53 |
| primary_location.is_oa | True |
| primary_location.source | |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://aclanthology.org/2023.acl-long.53.pdf |
| primary_location.version | publishedVersion |
| primary_location.raw_type | proceedings-article |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers) |
| primary_location.landing_page_url | https://doi.org/10.18653/v1/2023.acl-long.53 |
| publication_date | 2023-01-01 |
| publication_year | 2023 |
| referenced_works | https://openalex.org/W3086172114, https://openalex.org/W3103433205, https://openalex.org/W2127978399, https://openalex.org/W2252139350, https://openalex.org/W2888389346, https://openalex.org/W2807205034, https://openalex.org/W3126096026, https://openalex.org/W2563163303, https://openalex.org/W4242510480, https://openalex.org/W4200367537, https://openalex.org/W2251913848, https://openalex.org/W2768957049, https://openalex.org/W2406663039, https://openalex.org/W3099004917, https://openalex.org/W3199733080, https://openalex.org/W3153140595, https://openalex.org/W2131726681, https://openalex.org/W3202897922, https://openalex.org/W2608309533, https://openalex.org/W3034302623, https://openalex.org/W2756465743, https://openalex.org/W2794325560, https://openalex.org/W2971204996, https://openalex.org/W2966867036, https://openalex.org/W2620968868, https://openalex.org/W2251397412, https://openalex.org/W3105609801, https://openalex.org/W2917128112, https://openalex.org/W3103849289, https://openalex.org/W2897509371, https://openalex.org/W2090243146, https://openalex.org/W3207077137, https://openalex.org/W2810987158 |
| referenced_works_count | 33 |
| abstract_inverted_index.a | 88, 114 |
| abstract_inverted_index.In | 43 |
| abstract_inverted_index.QA | 117 |
| abstract_inverted_index.We | 75, 106 |
| abstract_inverted_index.an | 48, 61 |
| abstract_inverted_index.by | 82 |
| abstract_inverted_index.in | 7, 25, 60 |
| abstract_inverted_index.is | 16, 31 |
| abstract_inverted_index.it | 95 |
| abstract_inverted_index.no | 17 |
| abstract_inverted_index.of | 10, 34, 52, 110 |
| abstract_inverted_index.on | 20, 92, 113 |
| abstract_inverted_index.to | 23, 63, 97 |
| abstract_inverted_index.we | 46 |
| abstract_inverted_index.NLP | 12 |
| abstract_inverted_index.and | 58, 85 |
| abstract_inverted_index.for | 72, 103 |
| abstract_inverted_index.has | 4 |
| abstract_inverted_index.our | 111 |
| abstract_inverted_index.the | 8, 32, 67, 78, 99, 108 |
| abstract_inverted_index.use | 24 |
| abstract_inverted_index.Open | 0 |
| abstract_inverted_index.been | 5 |
| abstract_inverted_index.find | 76 |
| abstract_inverted_index.have | 87 |
| abstract_inverted_index.help | 64 |
| abstract_inverted_index.into | 41 |
| abstract_inverted_index.lack | 33 |
| abstract_inverted_index.made | 81 |
| abstract_inverted_index.most | 68, 100 |
| abstract_inverted_index.sets | 40 |
| abstract_inverted_index.take | 37 |
| abstract_inverted_index.that | 36, 77 |
| abstract_inverted_index.this | 44 |
| abstract_inverted_index.used | 6 |
| abstract_inverted_index.clear | 18 |
| abstract_inverted_index.model | 102 |
| abstract_inverted_index.sets, | 57 |
| abstract_inverted_index.their | 73 |
| abstract_inverted_index.there | 15 |
| abstract_inverted_index.users | 65 |
| abstract_inverted_index.which | 21, 26 |
| abstract_inverted_index.OpenIE | 54, 70 |
| abstract_inverted_index.choose | 66, 98 |
| abstract_inverted_index.effect | 91 |
| abstract_inverted_index.effort | 62 |
| abstract_inverted_index.making | 94 |
| abstract_inverted_index.models | 22, 84 |
| abstract_inverted_index.neural | 53 |
| abstract_inverted_index.paper, | 45 |
| abstract_inverted_index.survey | 51 |
| abstract_inverted_index.tasks. | 13, 27 |
| abstract_inverted_index.things | 29 |
| abstract_inverted_index.Complex | 116 |
| abstract_inverted_index.further | 30 |
| abstract_inverted_index.models, | 55 |
| abstract_inverted_index.one’s | 104 |
| abstract_inverted_index.present | 47 |
| abstract_inverted_index.systems | 71 |
| abstract_inverted_index.various | 11 |
| abstract_inverted_index.(OpenIE) | 3 |
| abstract_inverted_index.Muddying | 28 |
| abstract_inverted_index.account. | 42 |
| abstract_inverted_index.datasets | 86 |
| abstract_inverted_index.suitable | 69 |
| abstract_inverted_index.training | 39, 56 |
| abstract_inverted_index.consensus | 19 |
| abstract_inverted_index.different | 79, 83 |
| abstract_inverted_index.differing | 38 |
| abstract_inverted_index.empirical | 50 |
| abstract_inverted_index.important | 96 |
| abstract_inverted_index.pipelines | 9 |
| abstract_inverted_index.Extraction | 2 |
| abstract_inverted_index.benchmarks | 59 |
| abstract_inverted_index.downstream | 115 |
| abstract_inverted_index.Information | 1 |
| abstract_inverted_index.appropriate | 101 |
| abstract_inverted_index.assumptions | 80 |
| abstract_inverted_index.comparisons | 35 |
| abstract_inverted_index.demonstrate | 107 |
| abstract_inverted_index.significant | 90 |
| abstract_inverted_index.application. | 118 |
| abstract_inverted_index.performance, | 93 |
| abstract_inverted_index.applicability | 109 |
| abstract_inverted_index.applications. | 74, 105 |
| abstract_inverted_index.statistically | 89 |
| abstract_inverted_index.Unfortunately, | 14 |
| abstract_inverted_index.recommendations | 112 |
| abstract_inverted_index.application-focused | 49 |
| cited_by_percentile_year.max | 96 |
| cited_by_percentile_year.min | 94 |
| countries_distinct_count | 2 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile.value | 0.67254179 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |