Streamlining Systematic Reviews: A Novel Application of Large Language Models Article Swipe

PDF

Fouad Trad , Ryan Yammine , Jana Charafeddine , Marlene Chakhtoura , Maya Rahme , Ghada El‐Hajj Fuleihan , Ali Chehab ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2412.15247

Systematic reviews (SRs) are essential for evidence-based guidelines but are often limited by the time-consuming nature of literature screening. We propose and evaluate an in-house system based on Large Language Models (LLMs) for automating both title/abstract and full-text screening, addressing a critical gap in the literature. Using a completed SR on Vitamin D and falls (14,439 articles), the LLM-based system employed prompt engineering for title/abstract screening and Retrieval-Augmented Generation (RAG) for full-text screening. The system achieved an article exclusion rate (AER) of 99.5%, specificity of 99.6%, a false negative rate (FNR) of 0%, and a negative predictive value (NPV) of 100%. After screening, only 78 articles required manual review, including all 20 identified by traditional methods, reducing manual screening time by 95.5%. For comparison, Rayyan, a commercial tool for title/abstract screening, achieved an AER of 72.1% and FNR of 5% when including articles Rayyan considered as undecided or likely to include. Lowering Rayyan's inclusion thresholds improved FNR to 0% but increased screening time. By addressing both screening phases, the LLM-based system significantly outperformed Rayyan and traditional methods, reducing total screening time to 25.5 hours while maintaining high accuracy. These findings highlight the transformative potential of LLMs in SR workflows by offering a scalable, efficient, and accurate solution, particularly for the full-text screening phase, which has lacked automation tools.

Related Topics

Concepts

Computer science Systematic review Chemistry MEDLINE Biochemistry

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2412.15247
PDF: https://arxiv.org/pdf/2412.15247
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4405713874

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4405713874

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2412.15247

Digital Object Identifier
Title: Streamlining Systematic Reviews: A Novel Application of Large Language Models

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-12-14

Full publication date if available
Authors: Fouad Trad, Ryan Yammine, Jana Charafeddine, Marlene Chakhtoura, Maya Rahme, Ghada El‐Hajj Fuleihan, Ali Chehab

List of authors in order
Landing page: https://arxiv.org/abs/2412.15247

Publisher landing page
PDF URL: https://arxiv.org/pdf/2412.15247

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2412.15247

Direct OA link when available
Concepts: Computer science, Systematic review, Chemistry, MEDLINE, Biochemistry

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4405713874
doi	https://doi.org/10.48550/arxiv.2412.15247
ids.doi	https://doi.org/10.48550/arxiv.2412.15247
ids.openalex	https://openalex.org/W4405713874
fwci	0.0
type	preprint
title	Streamlining Systematic Reviews: A Novel Application of Large Language Models
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T13910
topics[0].field.id	https://openalex.org/fields/33
topics[0].field.display_name	Social Sciences
topics[0].score	0.47049999237060547
topics[0].domain.id	https://openalex.org/domains/2
topics[0].domain.display_name	Social Sciences
topics[0].subfield.id	https://openalex.org/subfields/3300
topics[0].subfield.display_name	General Social Sciences
topics[0].display_name	Computational and Text Analysis Methods
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.5212714672088623
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
concepts[1].id	https://openalex.org/C189708586
concepts[1].level	3
concepts[1].score	0.48064127564430237
concepts[1].wikidata	https://www.wikidata.org/wiki/Q1504425
concepts[1].display_name	Systematic review
concepts[2].id	https://openalex.org/C185592680
concepts[2].level	0
concepts[2].score	0.08996716141700745
concepts[2].wikidata	https://www.wikidata.org/wiki/Q2329
concepts[2].display_name	Chemistry
concepts[3].id	https://openalex.org/C2779473830
concepts[3].level	2
concepts[3].score	0.05224794149398804
concepts[3].wikidata	https://www.wikidata.org/wiki/Q1540899
concepts[3].display_name	MEDLINE
concepts[4].id	https://openalex.org/C55493867
concepts[4].level	1
concepts[4].score	0.0
concepts[4].wikidata	https://www.wikidata.org/wiki/Q7094
concepts[4].display_name	Biochemistry
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.5212714672088623
keywords[0].display_name	Computer science
keywords[1].id	https://openalex.org/keywords/systematic-review
keywords[1].score	0.48064127564430237
keywords[1].display_name	Systematic review
keywords[2].id	https://openalex.org/keywords/chemistry
keywords[2].score	0.08996716141700745
keywords[2].display_name	Chemistry
keywords[3].id	https://openalex.org/keywords/medline
keywords[3].score	0.05224794149398804
keywords[3].display_name	MEDLINE
language	en
locations[0].id	pmh:oai:arXiv.org:2412.15247
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2412.15247
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2412.15247
locations[1].id	doi:10.48550/arxiv.2412.15247
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article-journal
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2412.15247
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5086668805
authorships[0].author.orcid	https://orcid.org/0000-0003-2241-8195
authorships[0].author.display_name	Fouad Trad
authorships[0].author_position	first
authorships[0].raw_author_name	Trad, Fouad
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5018463301
authorships[1].author.orcid	https://orcid.org/0009-0005-9183-7395
authorships[1].author.display_name	Ryan Yammine
authorships[1].author_position	middle
authorships[1].raw_author_name	Yammine, Ryan
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5115615963
authorships[2].author.orcid
authorships[2].author.display_name	Jana Charafeddine
authorships[2].author_position	middle
authorships[2].raw_author_name	Charafeddine, Jana
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5073122583
authorships[3].author.orcid	https://orcid.org/0000-0003-0012-986X
authorships[3].author.display_name	Marlene Chakhtoura
authorships[3].author_position	middle
authorships[3].raw_author_name	Chakhtoura, Marlene
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5014581406
authorships[4].author.orcid	https://orcid.org/0000-0002-9467-5062
authorships[4].author.display_name	Maya Rahme
authorships[4].author_position	middle
authorships[4].raw_author_name	Rahme, Maya
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5083958215
authorships[5].author.orcid	https://orcid.org/0000-0002-2076-5858
authorships[5].author.display_name	Ghada El‐Hajj Fuleihan
authorships[5].author_position	middle
authorships[5].raw_author_name	Fuleihan, Ghada El-Hajj
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5115615964
authorships[6].author.orcid
authorships[6].author.display_name	Ali Chehab
authorships[6].author_position	last
authorships[6].raw_author_name	Chehab, Ali
authorships[6].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2412.15247
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Streamlining Systematic Reviews: A Novel Application of Large Language Models
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T13910
primary_topic.field.id	https://openalex.org/fields/33
primary_topic.field.display_name	Social Sciences
primary_topic.score	0.47049999237060547
primary_topic.domain.id	https://openalex.org/domains/2
primary_topic.domain.display_name	Social Sciences
primary_topic.subfield.id	https://openalex.org/subfields/3300
primary_topic.subfield.display_name	General Social Sciences
primary_topic.display_name	Computational and Text Analysis Methods
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2412.15247
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2412.15247
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2412.15247
primary_location.id	pmh:oai:arXiv.org:2412.15247
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2412.15247
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2412.15247
publication_date	2024-12-14
publication_year	2024
referenced_works_count	0
abstract_inverted_index.D	52
abstract_inverted_index.a	40, 47, 86, 94, 125, 201
abstract_inverted_index.0%	158
abstract_inverted_index.20	111
abstract_inverted_index.5%	139
abstract_inverted_index.78	104
abstract_inverted_index.By	163
abstract_inverted_index.SR	49, 197
abstract_inverted_index.We	19
abstract_inverted_index.an	23, 76, 132
abstract_inverted_index.as	145
abstract_inverted_index.by	12, 113, 120, 199
abstract_inverted_index.in	43, 196
abstract_inverted_index.of	16, 81, 84, 91, 99, 134, 138, 194
abstract_inverted_index.on	27, 50
abstract_inverted_index.or	147
abstract_inverted_index.to	149, 157, 181
abstract_inverted_index.0%,	92
abstract_inverted_index.AER	133
abstract_inverted_index.FNR	137, 156
abstract_inverted_index.For	122
abstract_inverted_index.The	73
abstract_inverted_index.all	110
abstract_inverted_index.and	21, 36, 53, 66, 93, 136, 174, 204
abstract_inverted_index.are	3, 9
abstract_inverted_index.but	8, 159
abstract_inverted_index.for	5, 32, 63, 70, 128, 208
abstract_inverted_index.gap	42
abstract_inverted_index.has	214
abstract_inverted_index.the	13, 44, 57, 168, 191, 209
abstract_inverted_index.25.5	182
abstract_inverted_index.LLMs	195
abstract_inverted_index.both	34, 165
abstract_inverted_index.high	186
abstract_inverted_index.only	103
abstract_inverted_index.rate	79, 89
abstract_inverted_index.time	119, 180
abstract_inverted_index.tool	127
abstract_inverted_index.when	140
abstract_inverted_index.(AER)	80
abstract_inverted_index.(FNR)	90
abstract_inverted_index.(NPV)	98
abstract_inverted_index.(RAG)	69
abstract_inverted_index.(SRs)	2
abstract_inverted_index.100%.	100
abstract_inverted_index.72.1%	135
abstract_inverted_index.After	101
abstract_inverted_index.Large	28
abstract_inverted_index.These	188
abstract_inverted_index.Using	46
abstract_inverted_index.based	26
abstract_inverted_index.falls	54
abstract_inverted_index.false	87
abstract_inverted_index.hours	183
abstract_inverted_index.often	10
abstract_inverted_index.time.	162
abstract_inverted_index.total	178
abstract_inverted_index.value	97
abstract_inverted_index.which	213
abstract_inverted_index.while	184
abstract_inverted_index.(LLMs)	31
abstract_inverted_index.95.5%.	121
abstract_inverted_index.99.5%,	82
abstract_inverted_index.99.6%,	85
abstract_inverted_index.Models	30
abstract_inverted_index.Rayyan	143, 173
abstract_inverted_index.lacked	215
abstract_inverted_index.likely	148
abstract_inverted_index.manual	107, 117
abstract_inverted_index.nature	15
abstract_inverted_index.phase,	212
abstract_inverted_index.prompt	61
abstract_inverted_index.system	25, 59, 74, 170
abstract_inverted_index.tools.	217
abstract_inverted_index.(14,439	55
abstract_inverted_index.Rayyan,	124
abstract_inverted_index.Vitamin	51
abstract_inverted_index.article	77
abstract_inverted_index.limited	11
abstract_inverted_index.phases,	167
abstract_inverted_index.propose	20
abstract_inverted_index.review,	108
abstract_inverted_index.reviews	1
abstract_inverted_index.Language	29
abstract_inverted_index.Lowering	151
abstract_inverted_index.Rayyan's	152
abstract_inverted_index.accurate	205
abstract_inverted_index.achieved	75, 131
abstract_inverted_index.articles	105, 142
abstract_inverted_index.critical	41
abstract_inverted_index.employed	60
abstract_inverted_index.evaluate	22
abstract_inverted_index.findings	189
abstract_inverted_index.improved	155
abstract_inverted_index.in-house	24
abstract_inverted_index.include.	150
abstract_inverted_index.methods,	115, 176
abstract_inverted_index.negative	88, 95
abstract_inverted_index.offering	200
abstract_inverted_index.reducing	116, 177
abstract_inverted_index.required	106
abstract_inverted_index.LLM-based	58, 169
abstract_inverted_index.accuracy.	187
abstract_inverted_index.completed	48
abstract_inverted_index.essential	4
abstract_inverted_index.exclusion	78
abstract_inverted_index.full-text	37, 71, 210
abstract_inverted_index.highlight	190
abstract_inverted_index.including	109, 141
abstract_inverted_index.inclusion	153
abstract_inverted_index.increased	160
abstract_inverted_index.potential	193
abstract_inverted_index.scalable,	202
abstract_inverted_index.screening	65, 118, 161, 166, 179, 211
abstract_inverted_index.solution,	206
abstract_inverted_index.undecided	146
abstract_inverted_index.workflows	198
abstract_inverted_index.Generation	68
abstract_inverted_index.Systematic	0
abstract_inverted_index.addressing	39, 164
abstract_inverted_index.articles),	56
abstract_inverted_index.automating	33
abstract_inverted_index.automation	216
abstract_inverted_index.commercial	126
abstract_inverted_index.considered	144
abstract_inverted_index.efficient,	203
abstract_inverted_index.guidelines	7
abstract_inverted_index.identified	112
abstract_inverted_index.literature	17
abstract_inverted_index.predictive	96
abstract_inverted_index.screening,	38, 102, 130
abstract_inverted_index.screening.	18, 72
abstract_inverted_index.thresholds	154
abstract_inverted_index.comparison,	123
abstract_inverted_index.engineering	62
abstract_inverted_index.literature.	45
abstract_inverted_index.maintaining	185
abstract_inverted_index.specificity	83
abstract_inverted_index.traditional	114, 175
abstract_inverted_index.outperformed	172
abstract_inverted_index.particularly	207
abstract_inverted_index.significantly	171
abstract_inverted_index.evidence-based	6
abstract_inverted_index.time-consuming	14
abstract_inverted_index.title/abstract	35, 64, 129
abstract_inverted_index.transformative	192
abstract_inverted_index.Retrieval-Augmented	67
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	7
citation_normalized_percentile.value	0.46242106
citation_normalized_percentile.is_in_top_1_percent	False
citation_normalized_percentile.is_in_top_10_percent	True