LLM-IE: A Python Package for Generative Information Extraction with Large Language Models Article Swipe
Enshuo Hsu
,
Kirk Roberts
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2411.11779
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2411.11779
Objectives: Despite the recent adoption of large language models (LLMs) for biomedical information extraction, challenges in prompt engineering and algorithms persist, with no dedicated software available. To address this, we developed LLM-IE: a Python package for building complete information extraction pipelines. Our key innovation is an interactive LLM agent to support schema definition and prompt design. Materials and Methods: The LLM-IE supports named entity recognition, entity attribute extraction, and relation extraction tasks. We benchmarked on the i2b2 datasets and conducted a system evaluation. Results: The sentence-based prompting algorithm resulted in the best performance while requiring a longer inference time. System evaluation provided intuitive visualization. Discussion: LLM-IE was designed from practical NLP experience in healthcare and has been adopted in internal projects. It should hold great value to the biomedical NLP community. Conclusion: We developed a Python package, LLM-IE, that provides building blocks for robust information extraction pipeline construction.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2411.11779
- https://arxiv.org/pdf/2411.11779
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4404571232
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4404571232Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2411.11779Digital Object Identifier
- Title
-
LLM-IE: A Python Package for Generative Information Extraction with Large Language ModelsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-11-18Full publication date if available
- Authors
-
Enshuo Hsu, Kirk RobertsList of authors in order
- Landing page
-
https://arxiv.org/abs/2411.11779Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2411.11779Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2411.11779Direct OA link when available
- Concepts
-
Python (programming language), Generative grammar, Computer science, Programming language, Natural language processing, Artificial intelligenceTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4404571232 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2411.11779 |
| ids.doi | https://doi.org/10.48550/arxiv.2411.11779 |
| ids.openalex | https://openalex.org/W4404571232 |
| fwci | 0.0 |
| type | preprint |
| title | LLM-IE: A Python Package for Generative Information Extraction with Large Language Models |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10181 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9973999857902527 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Natural Language Processing Techniques |
| topics[1].id | https://openalex.org/T10028 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9868999719619751 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Topic Modeling |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C519991488 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8181297779083252 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q28865 |
| concepts[0].display_name | Python (programming language) |
| concepts[1].id | https://openalex.org/C39890363 |
| concepts[1].level | 2 |
| concepts[1].score | 0.6996393203735352 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q36108 |
| concepts[1].display_name | Generative grammar |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.5977216362953186 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C199360897 |
| concepts[3].level | 1 |
| concepts[3].score | 0.5215151906013489 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[3].display_name | Programming language |
| concepts[4].id | https://openalex.org/C204321447 |
| concepts[4].level | 1 |
| concepts[4].score | 0.45865345001220703 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[4].display_name | Natural language processing |
| concepts[5].id | https://openalex.org/C154945302 |
| concepts[5].level | 1 |
| concepts[5].score | 0.37531065940856934 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[5].display_name | Artificial intelligence |
| keywords[0].id | https://openalex.org/keywords/python |
| keywords[0].score | 0.8181297779083252 |
| keywords[0].display_name | Python (programming language) |
| keywords[1].id | https://openalex.org/keywords/generative-grammar |
| keywords[1].score | 0.6996393203735352 |
| keywords[1].display_name | Generative grammar |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.5977216362953186 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/programming-language |
| keywords[3].score | 0.5215151906013489 |
| keywords[3].display_name | Programming language |
| keywords[4].id | https://openalex.org/keywords/natural-language-processing |
| keywords[4].score | 0.45865345001220703 |
| keywords[4].display_name | Natural language processing |
| keywords[5].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[5].score | 0.37531065940856934 |
| keywords[5].display_name | Artificial intelligence |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2411.11779 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2411.11779 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2411.11779 |
| locations[1].id | doi:10.48550/arxiv.2411.11779 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article-journal |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2411.11779 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101522835 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-8137-0171 |
| authorships[0].author.display_name | Enshuo Hsu |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Hsu, Enshuo |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5046709245 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-6525-5213 |
| authorships[1].author.display_name | Kirk Roberts |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Roberts, Kirk |
| authorships[1].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2411.11779 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | LLM-IE: A Python Package for Generative Information Extraction with Large Language Models |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10181 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9973999857902527 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Natural Language Processing Techniques |
| related_works | https://openalex.org/W2341492732, https://openalex.org/W3187193180, https://openalex.org/W106542691, https://openalex.org/W1699080303, https://openalex.org/W4297799326, https://openalex.org/W3116064965, https://openalex.org/W4287027380, https://openalex.org/W2207495067, https://openalex.org/W1906486629, https://openalex.org/W3204019825 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2411.11779 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2411.11779 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2411.11779 |
| primary_location.id | pmh:oai:arXiv.org:2411.11779 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2411.11779 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2411.11779 |
| publication_date | 2024-11-18 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 32, 80, 95, 134 |
| abstract_inverted_index.It | 121 |
| abstract_inverted_index.To | 26 |
| abstract_inverted_index.We | 72, 132 |
| abstract_inverted_index.an | 45 |
| abstract_inverted_index.in | 15, 89, 112, 118 |
| abstract_inverted_index.is | 44 |
| abstract_inverted_index.no | 22 |
| abstract_inverted_index.of | 5 |
| abstract_inverted_index.on | 74 |
| abstract_inverted_index.to | 49, 126 |
| abstract_inverted_index.we | 29 |
| abstract_inverted_index.LLM | 47 |
| abstract_inverted_index.NLP | 110, 129 |
| abstract_inverted_index.Our | 41 |
| abstract_inverted_index.The | 59, 84 |
| abstract_inverted_index.and | 18, 53, 57, 68, 78, 114 |
| abstract_inverted_index.for | 10, 35, 142 |
| abstract_inverted_index.has | 115 |
| abstract_inverted_index.key | 42 |
| abstract_inverted_index.the | 2, 75, 90, 127 |
| abstract_inverted_index.was | 106 |
| abstract_inverted_index.been | 116 |
| abstract_inverted_index.best | 91 |
| abstract_inverted_index.from | 108 |
| abstract_inverted_index.hold | 123 |
| abstract_inverted_index.i2b2 | 76 |
| abstract_inverted_index.that | 138 |
| abstract_inverted_index.with | 21 |
| abstract_inverted_index.agent | 48 |
| abstract_inverted_index.great | 124 |
| abstract_inverted_index.large | 6 |
| abstract_inverted_index.named | 62 |
| abstract_inverted_index.this, | 28 |
| abstract_inverted_index.time. | 98 |
| abstract_inverted_index.value | 125 |
| abstract_inverted_index.while | 93 |
| abstract_inverted_index.(LLMs) | 9 |
| abstract_inverted_index.LLM-IE | 60, 105 |
| abstract_inverted_index.Python | 33, 135 |
| abstract_inverted_index.System | 99 |
| abstract_inverted_index.blocks | 141 |
| abstract_inverted_index.entity | 63, 65 |
| abstract_inverted_index.longer | 96 |
| abstract_inverted_index.models | 8 |
| abstract_inverted_index.prompt | 16, 54 |
| abstract_inverted_index.recent | 3 |
| abstract_inverted_index.robust | 143 |
| abstract_inverted_index.schema | 51 |
| abstract_inverted_index.should | 122 |
| abstract_inverted_index.system | 81 |
| abstract_inverted_index.tasks. | 71 |
| abstract_inverted_index.Despite | 1 |
| abstract_inverted_index.LLM-IE, | 137 |
| abstract_inverted_index.LLM-IE: | 31 |
| abstract_inverted_index.address | 27 |
| abstract_inverted_index.adopted | 117 |
| abstract_inverted_index.design. | 55 |
| abstract_inverted_index.package | 34 |
| abstract_inverted_index.support | 50 |
| abstract_inverted_index.Methods: | 58 |
| abstract_inverted_index.Results: | 83 |
| abstract_inverted_index.adoption | 4 |
| abstract_inverted_index.building | 36, 140 |
| abstract_inverted_index.complete | 37 |
| abstract_inverted_index.datasets | 77 |
| abstract_inverted_index.designed | 107 |
| abstract_inverted_index.internal | 119 |
| abstract_inverted_index.language | 7 |
| abstract_inverted_index.package, | 136 |
| abstract_inverted_index.persist, | 20 |
| abstract_inverted_index.pipeline | 146 |
| abstract_inverted_index.provided | 101 |
| abstract_inverted_index.provides | 139 |
| abstract_inverted_index.relation | 69 |
| abstract_inverted_index.resulted | 88 |
| abstract_inverted_index.software | 24 |
| abstract_inverted_index.supports | 61 |
| abstract_inverted_index.Materials | 56 |
| abstract_inverted_index.algorithm | 87 |
| abstract_inverted_index.attribute | 66 |
| abstract_inverted_index.conducted | 79 |
| abstract_inverted_index.dedicated | 23 |
| abstract_inverted_index.developed | 30, 133 |
| abstract_inverted_index.inference | 97 |
| abstract_inverted_index.intuitive | 102 |
| abstract_inverted_index.practical | 109 |
| abstract_inverted_index.projects. | 120 |
| abstract_inverted_index.prompting | 86 |
| abstract_inverted_index.requiring | 94 |
| abstract_inverted_index.algorithms | 19 |
| abstract_inverted_index.available. | 25 |
| abstract_inverted_index.biomedical | 11, 128 |
| abstract_inverted_index.challenges | 14 |
| abstract_inverted_index.community. | 130 |
| abstract_inverted_index.definition | 52 |
| abstract_inverted_index.evaluation | 100 |
| abstract_inverted_index.experience | 111 |
| abstract_inverted_index.extraction | 39, 70, 145 |
| abstract_inverted_index.healthcare | 113 |
| abstract_inverted_index.innovation | 43 |
| abstract_inverted_index.pipelines. | 40 |
| abstract_inverted_index.Conclusion: | 131 |
| abstract_inverted_index.Discussion: | 104 |
| abstract_inverted_index.Objectives: | 0 |
| abstract_inverted_index.benchmarked | 73 |
| abstract_inverted_index.engineering | 17 |
| abstract_inverted_index.evaluation. | 82 |
| abstract_inverted_index.extraction, | 13, 67 |
| abstract_inverted_index.information | 12, 38, 144 |
| abstract_inverted_index.interactive | 46 |
| abstract_inverted_index.performance | 92 |
| abstract_inverted_index.recognition, | 64 |
| abstract_inverted_index.construction. | 147 |
| abstract_inverted_index.sentence-based | 85 |
| abstract_inverted_index.visualization. | 103 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 2 |
| citation_normalized_percentile.value | 0.23069357 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |