When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications Article Swipe
Kevin Pei
,
Ishan Jindal
,
Kevin Chen–Chuan Chang
,
ChengXiang Zhai
,
Yunyao Li
·
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2211.08228
YOU?
·
· 2022
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2211.08228
Open Information Extraction (OpenIE) has been used in the pipelines of various NLP tasks. Unfortunately, there is no clear consensus on which models to use in which tasks. Muddying things further is the lack of comparisons that take differing training sets into account. In this paper, we present an application-focused empirical survey of neural OpenIE models, training sets, and benchmarks in an effort to help users choose the most suitable OpenIE systems for their applications. We find that the different assumptions made by different models and datasets have a statistically significant effect on performance, making it important to choose the most appropriate model for one's applications. We demonstrate the applicability of our recommendations on a downstream Complex QA application.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2211.08228
- https://arxiv.org/pdf/2211.08228
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4309211221
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4309211221Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2211.08228Digital Object Identifier
- Title
-
When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream ApplicationsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2022Year of publication
- Publication date
-
2022-11-15Full publication date if available
- Authors
-
Kevin Pei, Ishan Jindal, Kevin Chen–Chuan Chang, ChengXiang Zhai, Yunyao LiList of authors in order
- Landing page
-
https://arxiv.org/abs/2211.08228Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2211.08228Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2211.08228Direct OA link when available
- Concepts
-
Downstream (manufacturing), Computer science, Empirical research, Artificial intelligence, Data science, Machine learning, Data mining, Engineering, Operations management, Epistemology, PhilosophyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4309211221 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2211.08228 |
| ids.doi | https://doi.org/10.48550/arxiv.2211.08228 |
| ids.openalex | https://openalex.org/W4309211221 |
| fwci | |
| type | preprint |
| title | When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9979000091552734 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T10181 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9882000088691711 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Natural Language Processing Techniques |
| topics[2].id | https://openalex.org/T10260 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9821000099182129 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1710 |
| topics[2].subfield.display_name | Information Systems |
| topics[2].display_name | Software Engineering Research |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2776207758 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8288152813911438 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q5303302 |
| concepts[0].display_name | Downstream (manufacturing) |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.7959704995155334 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C120936955 |
| concepts[2].level | 2 |
| concepts[2].score | 0.49799323081970215 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q2155640 |
| concepts[2].display_name | Empirical research |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.46513810753822327 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C2522767166 |
| concepts[4].level | 1 |
| concepts[4].score | 0.45761600136756897 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q2374463 |
| concepts[4].display_name | Data science |
| concepts[5].id | https://openalex.org/C119857082 |
| concepts[5].level | 1 |
| concepts[5].score | 0.45635345578193665 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[5].display_name | Machine learning |
| concepts[6].id | https://openalex.org/C124101348 |
| concepts[6].level | 1 |
| concepts[6].score | 0.36230355501174927 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q172491 |
| concepts[6].display_name | Data mining |
| concepts[7].id | https://openalex.org/C127413603 |
| concepts[7].level | 0 |
| concepts[7].score | 0.09766751527786255 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[7].display_name | Engineering |
| concepts[8].id | https://openalex.org/C21547014 |
| concepts[8].level | 1 |
| concepts[8].score | 0.0793558657169342 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q1423657 |
| concepts[8].display_name | Operations management |
| concepts[9].id | https://openalex.org/C111472728 |
| concepts[9].level | 1 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q9471 |
| concepts[9].display_name | Epistemology |
| concepts[10].id | https://openalex.org/C138885662 |
| concepts[10].level | 0 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[10].display_name | Philosophy |
| keywords[0].id | https://openalex.org/keywords/downstream |
| keywords[0].score | 0.8288152813911438 |
| keywords[0].display_name | Downstream (manufacturing) |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.7959704995155334 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/empirical-research |
| keywords[2].score | 0.49799323081970215 |
| keywords[2].display_name | Empirical research |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.46513810753822327 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/data-science |
| keywords[4].score | 0.45761600136756897 |
| keywords[4].display_name | Data science |
| keywords[5].id | https://openalex.org/keywords/machine-learning |
| keywords[5].score | 0.45635345578193665 |
| keywords[5].display_name | Machine learning |
| keywords[6].id | https://openalex.org/keywords/data-mining |
| keywords[6].score | 0.36230355501174927 |
| keywords[6].display_name | Data mining |
| keywords[7].id | https://openalex.org/keywords/engineering |
| keywords[7].score | 0.09766751527786255 |
| keywords[7].display_name | Engineering |
| keywords[8].id | https://openalex.org/keywords/operations-management |
| keywords[8].score | 0.0793558657169342 |
| keywords[8].display_name | Operations management |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2211.08228 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2211.08228 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2211.08228 |
| locations[1].id | doi:10.48550/arxiv.2211.08228 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2211.08228 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5108296434 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Kevin Pei |
| authorships[0].countries | US |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I157725225 |
| authorships[0].affiliations[0].raw_affiliation_string | Grainger College of Engineering, University of Illinois at Urbana-Champaign |
| authorships[0].institutions[0].id | https://openalex.org/I157725225 |
| authorships[0].institutions[0].ror | https://ror.org/047426m28 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I157725225 |
| authorships[0].institutions[0].country_code | US |
| authorships[0].institutions[0].display_name | University of Illinois Urbana-Champaign |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Pei, Kevin |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | Grainger College of Engineering, University of Illinois at Urbana-Champaign |
| authorships[1].author.id | https://openalex.org/A5077046363 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-8854-4189 |
| authorships[1].author.display_name | Ishan Jindal |
| authorships[1].affiliations[0].raw_affiliation_string | IBM Research |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Jindal, Ishan |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | IBM Research |
| authorships[2].author.id | https://openalex.org/A5101880377 |
| authorships[2].author.orcid | https://orcid.org/0000-0003-0997-6803 |
| authorships[2].author.display_name | Kevin Chen–Chuan Chang |
| authorships[2].countries | US |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I157725225 |
| authorships[2].affiliations[0].raw_affiliation_string | Grainger College of Engineering, University of Illinois at Urbana-Champaign |
| authorships[2].institutions[0].id | https://openalex.org/I157725225 |
| authorships[2].institutions[0].ror | https://ror.org/047426m28 |
| authorships[2].institutions[0].type | education |
| authorships[2].institutions[0].lineage | https://openalex.org/I157725225 |
| authorships[2].institutions[0].country_code | US |
| authorships[2].institutions[0].display_name | University of Illinois Urbana-Champaign |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Chang, Kevin Chen-Chuan |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Grainger College of Engineering, University of Illinois at Urbana-Champaign |
| authorships[3].author.id | https://openalex.org/A5028518494 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-6434-3702 |
| authorships[3].author.display_name | ChengXiang Zhai |
| authorships[3].countries | US |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I157725225 |
| authorships[3].affiliations[0].raw_affiliation_string | Grainger College of Engineering, University of Illinois at Urbana-Champaign |
| authorships[3].institutions[0].id | https://openalex.org/I157725225 |
| authorships[3].institutions[0].ror | https://ror.org/047426m28 |
| authorships[3].institutions[0].type | education |
| authorships[3].institutions[0].lineage | https://openalex.org/I157725225 |
| authorships[3].institutions[0].country_code | US |
| authorships[3].institutions[0].display_name | University of Illinois Urbana-Champaign |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Zhai, Chengxiang |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | Grainger College of Engineering, University of Illinois at Urbana-Champaign |
| authorships[4].author.id | https://openalex.org/A5102944075 |
| authorships[4].author.orcid | https://orcid.org/0009-0002-0814-4634 |
| authorships[4].author.display_name | Yunyao Li |
| authorships[4].countries | IL |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I1311269955 |
| authorships[4].affiliations[0].raw_affiliation_string | Apple Knowledge Platform |
| authorships[4].institutions[0].id | https://openalex.org/I1311269955 |
| authorships[4].institutions[0].ror | https://ror.org/04ehjr030 |
| authorships[4].institutions[0].type | company |
| authorships[4].institutions[0].lineage | https://openalex.org/I1311269955, https://openalex.org/I4210153776 |
| authorships[4].institutions[0].country_code | IL |
| authorships[4].institutions[0].display_name | Apple (Israel) |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Li, Yunyao |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | Apple Knowledge Platform |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2211.08228 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2022-11-24T00:00:00 |
| display_name | When to Use What: An In-Depth Comparative Empirical Analysis of OpenIE Systems for Downstream Applications |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9979000091552734 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W1583765404, https://openalex.org/W4214653257, https://openalex.org/W2055438207, https://openalex.org/W2521424917, https://openalex.org/W3040203686, https://openalex.org/W4249524554, https://openalex.org/W2349021146, https://openalex.org/W35583307, https://openalex.org/W2436192316, https://openalex.org/W2970287324 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2211.08228 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2211.08228 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2211.08228 |
| primary_location.id | pmh:oai:arXiv.org:2211.08228 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2211.08228 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2211.08228 |
| publication_date | 2022-11-15 |
| publication_year | 2022 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 88, 114 |
| abstract_inverted_index.In | 43 |
| abstract_inverted_index.QA | 117 |
| abstract_inverted_index.We | 75, 106 |
| abstract_inverted_index.an | 48, 61 |
| abstract_inverted_index.by | 82 |
| abstract_inverted_index.in | 7, 25, 60 |
| abstract_inverted_index.is | 16, 31 |
| abstract_inverted_index.it | 95 |
| abstract_inverted_index.no | 17 |
| abstract_inverted_index.of | 10, 34, 52, 110 |
| abstract_inverted_index.on | 20, 92, 113 |
| abstract_inverted_index.to | 23, 63, 97 |
| abstract_inverted_index.we | 46 |
| abstract_inverted_index.NLP | 12 |
| abstract_inverted_index.and | 58, 85 |
| abstract_inverted_index.for | 72, 103 |
| abstract_inverted_index.has | 4 |
| abstract_inverted_index.our | 111 |
| abstract_inverted_index.the | 8, 32, 67, 78, 99, 108 |
| abstract_inverted_index.use | 24 |
| abstract_inverted_index.Open | 0 |
| abstract_inverted_index.been | 5 |
| abstract_inverted_index.find | 76 |
| abstract_inverted_index.have | 87 |
| abstract_inverted_index.help | 64 |
| abstract_inverted_index.into | 41 |
| abstract_inverted_index.lack | 33 |
| abstract_inverted_index.made | 81 |
| abstract_inverted_index.most | 68, 100 |
| abstract_inverted_index.sets | 40 |
| abstract_inverted_index.take | 37 |
| abstract_inverted_index.that | 36, 77 |
| abstract_inverted_index.this | 44 |
| abstract_inverted_index.used | 6 |
| abstract_inverted_index.clear | 18 |
| abstract_inverted_index.model | 102 |
| abstract_inverted_index.one's | 104 |
| abstract_inverted_index.sets, | 57 |
| abstract_inverted_index.their | 73 |
| abstract_inverted_index.there | 15 |
| abstract_inverted_index.users | 65 |
| abstract_inverted_index.which | 21, 26 |
| abstract_inverted_index.OpenIE | 54, 70 |
| abstract_inverted_index.choose | 66, 98 |
| abstract_inverted_index.effect | 91 |
| abstract_inverted_index.effort | 62 |
| abstract_inverted_index.making | 94 |
| abstract_inverted_index.models | 22, 84 |
| abstract_inverted_index.neural | 53 |
| abstract_inverted_index.paper, | 45 |
| abstract_inverted_index.survey | 51 |
| abstract_inverted_index.tasks. | 13, 27 |
| abstract_inverted_index.things | 29 |
| abstract_inverted_index.Complex | 116 |
| abstract_inverted_index.further | 30 |
| abstract_inverted_index.models, | 55 |
| abstract_inverted_index.present | 47 |
| abstract_inverted_index.systems | 71 |
| abstract_inverted_index.various | 11 |
| abstract_inverted_index.(OpenIE) | 3 |
| abstract_inverted_index.Muddying | 28 |
| abstract_inverted_index.account. | 42 |
| abstract_inverted_index.datasets | 86 |
| abstract_inverted_index.suitable | 69 |
| abstract_inverted_index.training | 39, 56 |
| abstract_inverted_index.consensus | 19 |
| abstract_inverted_index.different | 79, 83 |
| abstract_inverted_index.differing | 38 |
| abstract_inverted_index.empirical | 50 |
| abstract_inverted_index.important | 96 |
| abstract_inverted_index.pipelines | 9 |
| abstract_inverted_index.Extraction | 2 |
| abstract_inverted_index.benchmarks | 59 |
| abstract_inverted_index.downstream | 115 |
| abstract_inverted_index.Information | 1 |
| abstract_inverted_index.appropriate | 101 |
| abstract_inverted_index.assumptions | 80 |
| abstract_inverted_index.comparisons | 35 |
| abstract_inverted_index.demonstrate | 107 |
| abstract_inverted_index.significant | 90 |
| abstract_inverted_index.application. | 118 |
| abstract_inverted_index.performance, | 93 |
| abstract_inverted_index.applicability | 109 |
| abstract_inverted_index.applications. | 74, 105 |
| abstract_inverted_index.statistically | 89 |
| abstract_inverted_index.Unfortunately, | 14 |
| abstract_inverted_index.recommendations | 112 |
| abstract_inverted_index.application-focused | 49 |
| cited_by_percentile_year | |
| countries_distinct_count | 2 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |