Customizing GPT-4 for clinical information retrieval from standard operating procedures Article Swipe
Hannah Sophie Muti
,
Chiara Maria Lavinia Löffler
,
Marie-Elisabeth Leßmann
,
Esther Helene Stüker
,
Johanna Kirchberg
,
Malte von Bonin
,
Martin Kolditz
,
Dyke Ferber
,
Katharina Egger‐Heidrich
,
Felix Merboth
,
Daniel E. Stange
,
Marius Distler
,
Jakob Nikolas Kather
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.1101/2024.06.24.24309221
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.1101/2024.06.24.24309221
Background The increasing complexity of medical knowledge necessitates efficient and reliable information access systems in clinical settings. For quality purposes, most hospitals use standard operating procedures (SOPs) for information management and implementation of local treatment standards. However, in clinical routine, this information is not always easily accessible. Customized Large Language Models (LLMs) may offer a tailored solution, but need thorough evaluation prior to clinical implementation. Objective To customize an LLM to retrieve information from hospital-specific SOPs, to evaluate its accuracy for clinical use and to compare different prompting strategies and large language models. Methods We customized GPT-4 with a predefined system prompt and 10 SOPs from four departments at the University Hospital Dresden. The model’s performance was evaluated through 30 predefined clinical questions of varying degree of detail, which were assessed by five observers with different levels of medical expertise through simple and interactive question-and-answering (Q&A). We assessed answer completeness, correctness and sufficiency for clinical use and the impact of prompt design on model performance. Finally, we compared the performance of GPT-4 with Claude-3-opus. Results Interactive Q&A yielded the highest rate of completeness (80%), correctness (83%) and sufficiency (60%). Acceptance of the LLM’s answer was higher among early-career medical staff. Degree of detail of the question prompt influenced answer accuracy, with intermediate-detail prompts achieving the highest sufficiency rates. Comparing LLMs, Claude-3-opus outperformed GPT-4 in providing sufficient answers (70.0% vs. 36.7%) and required fewer iterations for satisfactory responses. Both models adhered to the system prompt more effectively in the self-coded pipeline than in the browser application. All observers showed discrepancies between correctness and accuracy of the answers, which rooted in the representation of information in the SOPs. Conclusion Interactively querying customized LLMs can enhance clinical information retrieval, though expert oversight remains essential to ensure a safe application of this technology. After broader evaluation and with basic knowledge in prompt engineering, customized LLMs can be an efficient, clinically applicable tool.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- https://doi.org/10.1101/2024.06.24.24309221
- https://www.medrxiv.org/content/medrxiv/early/2024/06/24/2024.06.24.24309221.full.pdf
- OA Status
- green
- Cited By
- 1
- References
- 30
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4399933304
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4399933304Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.1101/2024.06.24.24309221Digital Object Identifier
- Title
-
Customizing GPT-4 for clinical information retrieval from standard operating proceduresWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-06-24Full publication date if available
- Authors
-
Hannah Sophie Muti, Chiara Maria Lavinia Löffler, Marie-Elisabeth Leßmann, Esther Helene Stüker, Johanna Kirchberg, Malte von Bonin, Martin Kolditz, Dyke Ferber, Katharina Egger‐Heidrich, Felix Merboth, Daniel E. Stange, Marius Distler, Jakob Nikolas KatherList of authors in order
- Landing page
-
https://doi.org/10.1101/2024.06.24.24309221Publisher landing page
- PDF URL
-
https://www.medrxiv.org/content/medrxiv/early/2024/06/24/2024.06.24.24309221.full.pdfDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://www.medrxiv.org/content/medrxiv/early/2024/06/24/2024.06.24.24309221.full.pdfDirect OA link when available
- Concepts
-
Information retrieval, Computer scienceTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
1Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 1Per-year citation counts (last 5 years)
- References (count)
-
30Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4399933304 |
|---|---|
| doi | https://doi.org/10.1101/2024.06.24.24309221 |
| ids.doi | https://doi.org/10.1101/2024.06.24.24309221 |
| ids.openalex | https://openalex.org/W4399933304 |
| fwci | 0.24339971 |
| type | preprint |
| title | Customizing GPT-4 for clinical information retrieval from standard operating procedures |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11636 |
| topics[0].field.id | https://openalex.org/fields/27 |
| topics[0].field.display_name | Medicine |
| topics[0].score | 0.753600001335144 |
| topics[0].domain.id | https://openalex.org/domains/4 |
| topics[0].domain.display_name | Health Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/2718 |
| topics[0].subfield.display_name | Health Informatics |
| topics[0].display_name | Artificial Intelligence in Healthcare and Education |
| topics[1].id | https://openalex.org/T13702 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.6628000140190125 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Machine Learning in Healthcare |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C23123220 |
| concepts[0].level | 1 |
| concepts[0].score | 0.6172921061515808 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q816826 |
| concepts[0].display_name | Information retrieval |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.5402685403823853 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| keywords[0].id | https://openalex.org/keywords/information-retrieval |
| keywords[0].score | 0.6172921061515808 |
| keywords[0].display_name | Information retrieval |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.5402685403823853 |
| keywords[1].display_name | Computer science |
| language | en |
| locations[0].id | doi:10.1101/2024.06.24.24309221 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306402567 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | False |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | bioRxiv (Cold Spring Harbor Laboratory) |
| locations[0].source.host_organization | https://openalex.org/I2750212522 |
| locations[0].source.host_organization_name | Cold Spring Harbor Laboratory |
| locations[0].source.host_organization_lineage | https://openalex.org/I2750212522 |
| locations[0].license | cc-by-nc |
| locations[0].pdf_url | https://www.medrxiv.org/content/medrxiv/early/2024/06/24/2024.06.24.24309221.full.pdf |
| locations[0].version | acceptedVersion |
| locations[0].raw_type | posted-content |
| locations[0].license_id | https://openalex.org/licenses/cc-by-nc |
| locations[0].is_accepted | True |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://doi.org/10.1101/2024.06.24.24309221 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5007050711 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-3134-3837 |
| authorships[0].author.display_name | Hannah Sophie Muti |
| authorships[0].countries | DE |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[0].affiliations[0].raw_affiliation_string | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[0].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[0].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[0].institutions[0].type | healthcare |
| authorships[0].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[0].institutions[0].country_code | DE |
| authorships[0].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Hannah Sophie Muti |
| authorships[0].is_corresponding | True |
| authorships[0].raw_affiliation_strings | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[1].author.id | https://openalex.org/A5093036710 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Chiara Maria Lavinia Löffler |
| authorships[1].countries | DE |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[1].affiliations[0].raw_affiliation_string | Else Kroener Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[1].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[1].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[1].institutions[0].type | healthcare |
| authorships[1].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[1].institutions[0].country_code | DE |
| authorships[1].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Chiara Maria Lavinia Löffler |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | Else Kroener Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[2].author.id | https://openalex.org/A5007284791 |
| authorships[2].author.orcid | https://orcid.org/0009-0005-7029-0028 |
| authorships[2].author.display_name | Marie-Elisabeth Leßmann |
| authorships[2].countries | DE |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[2].affiliations[0].raw_affiliation_string | Else Kroener Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[2].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[2].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[2].institutions[0].type | healthcare |
| authorships[2].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[2].institutions[0].country_code | DE |
| authorships[2].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Marie-Elisabeth Leßmann |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Else Kroener Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[3].author.id | https://openalex.org/A5099430585 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Esther Helene Stüker |
| authorships[3].countries | DE |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[3].affiliations[0].raw_affiliation_string | Else Kroener Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[3].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[3].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[3].institutions[0].type | healthcare |
| authorships[3].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[3].institutions[0].country_code | DE |
| authorships[3].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Esther Helene Stüker |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | Else Kroener Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[4].author.id | https://openalex.org/A5051317183 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-2786-4269 |
| authorships[4].author.display_name | Johanna Kirchberg |
| authorships[4].countries | DE |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[4].affiliations[0].raw_affiliation_string | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[4].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[4].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[4].institutions[0].type | healthcare |
| authorships[4].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[4].institutions[0].country_code | DE |
| authorships[4].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Johanna Kirchberg |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[5].author.id | https://openalex.org/A5102843648 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-2995-3230 |
| authorships[5].author.display_name | Malte von Bonin |
| authorships[5].countries | DE |
| authorships[5].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[5].affiliations[0].raw_affiliation_string | Medical Department 1, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[5].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[5].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[5].institutions[0].type | healthcare |
| authorships[5].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[5].institutions[0].country_code | DE |
| authorships[5].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Malte von Bonin |
| authorships[5].is_corresponding | False |
| authorships[5].raw_affiliation_strings | Medical Department 1, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[6].author.id | https://openalex.org/A5017733593 |
| authorships[6].author.orcid | https://orcid.org/0000-0001-6022-6827 |
| authorships[6].author.display_name | Martin Kolditz |
| authorships[6].countries | DE |
| authorships[6].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[6].affiliations[0].raw_affiliation_string | Medical Department 1, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[6].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[6].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[6].institutions[0].type | healthcare |
| authorships[6].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[6].institutions[0].country_code | DE |
| authorships[6].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Martin Kolditz |
| authorships[6].is_corresponding | False |
| authorships[6].raw_affiliation_strings | Medical Department 1, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[7].author.id | https://openalex.org/A5048917650 |
| authorships[7].author.orcid | https://orcid.org/0009-0006-6195-9276 |
| authorships[7].author.display_name | Dyke Ferber |
| authorships[7].countries | DE |
| authorships[7].affiliations[0].institution_ids | https://openalex.org/I223822909, https://openalex.org/I2802164966, https://openalex.org/I4210111460 |
| authorships[7].affiliations[0].raw_affiliation_string | Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany |
| authorships[7].institutions[0].id | https://openalex.org/I223822909 |
| authorships[7].institutions[0].ror | https://ror.org/038t36y30 |
| authorships[7].institutions[0].type | education |
| authorships[7].institutions[0].lineage | https://openalex.org/I223822909 |
| authorships[7].institutions[0].country_code | DE |
| authorships[7].institutions[0].display_name | Heidelberg University |
| authorships[7].institutions[1].id | https://openalex.org/I4210111460 |
| authorships[7].institutions[1].ror | https://ror.org/01txwsw02 |
| authorships[7].institutions[1].type | healthcare |
| authorships[7].institutions[1].lineage | https://openalex.org/I4210111460 |
| authorships[7].institutions[1].country_code | DE |
| authorships[7].institutions[1].display_name | National Center for Tumor Diseases |
| authorships[7].institutions[2].id | https://openalex.org/I2802164966 |
| authorships[7].institutions[2].ror | https://ror.org/013czdx64 |
| authorships[7].institutions[2].type | healthcare |
| authorships[7].institutions[2].lineage | https://openalex.org/I2802164966 |
| authorships[7].institutions[2].country_code | DE |
| authorships[7].institutions[2].display_name | University Hospital Heidelberg |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Dyke Ferber |
| authorships[7].is_corresponding | False |
| authorships[7].raw_affiliation_strings | Department of Medical Oncology, National Center for Tumor Diseases (NCT), Heidelberg University Hospital, Heidelberg, Germany |
| authorships[8].author.id | https://openalex.org/A5041105208 |
| authorships[8].author.orcid | https://orcid.org/0000-0002-0472-4335 |
| authorships[8].author.display_name | Katharina Egger‐Heidrich |
| authorships[8].countries | DE |
| authorships[8].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[8].affiliations[0].raw_affiliation_string | Medical Department 1, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[8].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[8].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[8].institutions[0].type | healthcare |
| authorships[8].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[8].institutions[0].country_code | DE |
| authorships[8].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Katharina Egger-Heidrich |
| authorships[8].is_corresponding | False |
| authorships[8].raw_affiliation_strings | Medical Department 1, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[9].author.id | https://openalex.org/A5016926654 |
| authorships[9].author.orcid | https://orcid.org/0000-0003-2540-6421 |
| authorships[9].author.display_name | Felix Merboth |
| authorships[9].countries | DE |
| authorships[9].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[9].affiliations[0].raw_affiliation_string | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[9].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[9].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[9].institutions[0].type | healthcare |
| authorships[9].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[9].institutions[0].country_code | DE |
| authorships[9].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Felix Merboth |
| authorships[9].is_corresponding | False |
| authorships[9].raw_affiliation_strings | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[10].author.id | https://openalex.org/A5004040583 |
| authorships[10].author.orcid | https://orcid.org/0000-0003-4246-2230 |
| authorships[10].author.display_name | Daniel E. Stange |
| authorships[10].countries | DE |
| authorships[10].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[10].affiliations[0].raw_affiliation_string | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[10].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[10].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[10].institutions[0].type | healthcare |
| authorships[10].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[10].institutions[0].country_code | DE |
| authorships[10].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[10].author_position | middle |
| authorships[10].raw_author_name | Daniel E. Stange |
| authorships[10].is_corresponding | False |
| authorships[10].raw_affiliation_strings | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[11].author.id | https://openalex.org/A5020380620 |
| authorships[11].author.orcid | https://orcid.org/0000-0002-3164-5755 |
| authorships[11].author.display_name | Marius Distler |
| authorships[11].countries | DE |
| authorships[11].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[11].affiliations[0].raw_affiliation_string | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[11].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[11].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[11].institutions[0].type | healthcare |
| authorships[11].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[11].institutions[0].country_code | DE |
| authorships[11].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[11].author_position | middle |
| authorships[11].raw_author_name | Marius Distler |
| authorships[11].is_corresponding | False |
| authorships[11].raw_affiliation_strings | Department for Visceral, Thoracic and Vascular Surgery, University Hospital and Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[12].author.id | https://openalex.org/A5073483894 |
| authorships[12].author.orcid | https://orcid.org/0000-0002-3730-5348 |
| authorships[12].author.display_name | Jakob Nikolas Kather |
| authorships[12].countries | DE |
| authorships[12].affiliations[0].institution_ids | https://openalex.org/I4210162051 |
| authorships[12].affiliations[0].raw_affiliation_string | Else Kroener Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| authorships[12].institutions[0].id | https://openalex.org/I4210162051 |
| authorships[12].institutions[0].ror | https://ror.org/04za5zm41 |
| authorships[12].institutions[0].type | healthcare |
| authorships[12].institutions[0].lineage | https://openalex.org/I4210162051 |
| authorships[12].institutions[0].country_code | DE |
| authorships[12].institutions[0].display_name | University Hospital Carl Gustav Carus |
| authorships[12].author_position | last |
| authorships[12].raw_author_name | Jakob Nikolas Kather |
| authorships[12].is_corresponding | False |
| authorships[12].raw_affiliation_strings | Else Kroener Fresenius Center for Digital Health, Faculty of Medicine Carl Gustav Carus, Dresden University of Technology, Dresden, Germany |
| has_content.pdf | True |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://www.medrxiv.org/content/medrxiv/early/2024/06/24/2024.06.24.24309221.full.pdf |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Customizing GPT-4 for clinical information retrieval from standard operating procedures |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T11636 |
| primary_topic.field.id | https://openalex.org/fields/27 |
| primary_topic.field.display_name | Medicine |
| primary_topic.score | 0.753600001335144 |
| primary_topic.domain.id | https://openalex.org/domains/4 |
| primary_topic.domain.display_name | Health Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/2718 |
| primary_topic.subfield.display_name | Health Informatics |
| primary_topic.display_name | Artificial Intelligence in Healthcare and Education |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052, https://openalex.org/W2382290278, https://openalex.org/W4395014643 |
| cited_by_count | 1 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 1 |
| locations_count | 1 |
| best_oa_location.id | doi:10.1101/2024.06.24.24309221 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306402567 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | False |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | bioRxiv (Cold Spring Harbor Laboratory) |
| best_oa_location.source.host_organization | https://openalex.org/I2750212522 |
| best_oa_location.source.host_organization_name | Cold Spring Harbor Laboratory |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I2750212522 |
| best_oa_location.license | cc-by-nc |
| best_oa_location.pdf_url | https://www.medrxiv.org/content/medrxiv/early/2024/06/24/2024.06.24.24309221.full.pdf |
| best_oa_location.version | acceptedVersion |
| best_oa_location.raw_type | posted-content |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by-nc |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.1101/2024.06.24.24309221 |
| primary_location.id | doi:10.1101/2024.06.24.24309221 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306402567 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | False |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | bioRxiv (Cold Spring Harbor Laboratory) |
| primary_location.source.host_organization | https://openalex.org/I2750212522 |
| primary_location.source.host_organization_name | Cold Spring Harbor Laboratory |
| primary_location.source.host_organization_lineage | https://openalex.org/I2750212522 |
| primary_location.license | cc-by-nc |
| primary_location.pdf_url | https://www.medrxiv.org/content/medrxiv/early/2024/06/24/2024.06.24.24309221.full.pdf |
| primary_location.version | acceptedVersion |
| primary_location.raw_type | posted-content |
| primary_location.license_id | https://openalex.org/licenses/cc-by-nc |
| primary_location.is_accepted | True |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://doi.org/10.1101/2024.06.24.24309221 |
| publication_date | 2024-06-24 |
| publication_year | 2024 |
| referenced_works | https://openalex.org/W3114754085, https://openalex.org/W4398766116, https://openalex.org/W4384561707, https://openalex.org/W4362708127, https://openalex.org/W4387500346, https://openalex.org/W4386399156, https://openalex.org/W4392091326, https://openalex.org/W4368367885, https://openalex.org/W4392504747, https://openalex.org/W3027879771, https://openalex.org/W4391221150, https://openalex.org/W2084788486, https://openalex.org/W2093833831, https://openalex.org/W4385620388, https://openalex.org/W4397003633, https://openalex.org/W4392945187, https://openalex.org/W4387744047, https://openalex.org/W4225689423, https://openalex.org/W4387768520, https://openalex.org/W4385623654, https://openalex.org/W4324373918, https://openalex.org/W4395049441, https://openalex.org/W4388494284, https://openalex.org/W4385571264, https://openalex.org/W2028828949, https://openalex.org/W4394975332, https://openalex.org/W4392603239, https://openalex.org/W4392193048, https://openalex.org/W4380201210, https://openalex.org/W4223572763 |
| referenced_works_count | 30 |
| abstract_inverted_index.a | 55, 99, 294 |
| abstract_inverted_index.10 | 104 |
| abstract_inverted_index.30 | 120 |
| abstract_inverted_index.To | 67 |
| abstract_inverted_index.We | 95, 147 |
| abstract_inverted_index.an | 69, 314 |
| abstract_inverted_index.at | 109 |
| abstract_inverted_index.be | 313 |
| abstract_inverted_index.by | 132 |
| abstract_inverted_index.in | 15, 38, 224, 247, 252, 269, 274, 307 |
| abstract_inverted_index.is | 43 |
| abstract_inverted_index.of | 5, 33, 124, 127, 138, 160, 171, 182, 191, 202, 204, 264, 272, 297 |
| abstract_inverted_index.on | 163 |
| abstract_inverted_index.to | 63, 71, 77, 85, 241, 292 |
| abstract_inverted_index.we | 167 |
| abstract_inverted_index.All | 256 |
| abstract_inverted_index.For | 18 |
| abstract_inverted_index.LLM | 70 |
| abstract_inverted_index.The | 2, 114 |
| abstract_inverted_index.and | 10, 31, 84, 90, 103, 143, 152, 157, 187, 231, 262, 303 |
| abstract_inverted_index.but | 58 |
| abstract_inverted_index.can | 282, 312 |
| abstract_inverted_index.for | 28, 81, 154, 235 |
| abstract_inverted_index.its | 79 |
| abstract_inverted_index.may | 53 |
| abstract_inverted_index.not | 44 |
| abstract_inverted_index.the | 110, 158, 169, 179, 192, 205, 215, 242, 248, 253, 265, 270, 275 |
| abstract_inverted_index.use | 23, 83, 156 |
| abstract_inverted_index.vs. | 229 |
| abstract_inverted_index.was | 117, 195 |
| abstract_inverted_index.Both | 238 |
| abstract_inverted_index.LLMs | 281, 311 |
| abstract_inverted_index.SOPs | 105 |
| abstract_inverted_index.five | 133 |
| abstract_inverted_index.four | 107 |
| abstract_inverted_index.from | 74, 106 |
| abstract_inverted_index.more | 245 |
| abstract_inverted_index.most | 21 |
| abstract_inverted_index.need | 59 |
| abstract_inverted_index.rate | 181 |
| abstract_inverted_index.safe | 295 |
| abstract_inverted_index.than | 251 |
| abstract_inverted_index.this | 41, 298 |
| abstract_inverted_index.were | 130 |
| abstract_inverted_index.with | 98, 135, 173, 211, 304 |
| abstract_inverted_index.(83%) | 186 |
| abstract_inverted_index.After | 300 |
| abstract_inverted_index.GPT-4 | 97, 172, 223 |
| abstract_inverted_index.LLMs, | 220 |
| abstract_inverted_index.Large | 49 |
| abstract_inverted_index.SOPs, | 76 |
| abstract_inverted_index.SOPs. | 276 |
| abstract_inverted_index.among | 197 |
| abstract_inverted_index.basic | 305 |
| abstract_inverted_index.fewer | 233 |
| abstract_inverted_index.large | 91 |
| abstract_inverted_index.local | 34 |
| abstract_inverted_index.model | 164 |
| abstract_inverted_index.offer | 54 |
| abstract_inverted_index.prior | 62 |
| abstract_inverted_index.tool. | 318 |
| abstract_inverted_index.which | 129, 267 |
| abstract_inverted_index.(60%). | 189 |
| abstract_inverted_index.(70.0% | 228 |
| abstract_inverted_index.(80%), | 184 |
| abstract_inverted_index.(LLMs) | 52 |
| abstract_inverted_index.(SOPs) | 27 |
| abstract_inverted_index.36.7%) | 230 |
| abstract_inverted_index.Degree | 201 |
| abstract_inverted_index.Models | 51 |
| abstract_inverted_index.access | 13 |
| abstract_inverted_index.always | 45 |
| abstract_inverted_index.answer | 149, 194, 209 |
| abstract_inverted_index.degree | 126 |
| abstract_inverted_index.design | 162 |
| abstract_inverted_index.detail | 203 |
| abstract_inverted_index.easily | 46 |
| abstract_inverted_index.ensure | 293 |
| abstract_inverted_index.expert | 288 |
| abstract_inverted_index.higher | 196 |
| abstract_inverted_index.impact | 159 |
| abstract_inverted_index.levels | 137 |
| abstract_inverted_index.models | 239 |
| abstract_inverted_index.prompt | 102, 161, 207, 244, 308 |
| abstract_inverted_index.rates. | 218 |
| abstract_inverted_index.rooted | 268 |
| abstract_inverted_index.showed | 258 |
| abstract_inverted_index.simple | 142 |
| abstract_inverted_index.staff. | 200 |
| abstract_inverted_index.system | 101, 243 |
| abstract_inverted_index.though | 287 |
| abstract_inverted_index.LLM’s | 193 |
| abstract_inverted_index.Methods | 94 |
| abstract_inverted_index.Q&A | 177 |
| abstract_inverted_index.Results | 175 |
| abstract_inverted_index.adhered | 240 |
| abstract_inverted_index.answers | 227 |
| abstract_inverted_index.between | 260 |
| abstract_inverted_index.broader | 301 |
| abstract_inverted_index.browser | 254 |
| abstract_inverted_index.compare | 86 |
| abstract_inverted_index.detail, | 128 |
| abstract_inverted_index.enhance | 283 |
| abstract_inverted_index.highest | 180, 216 |
| abstract_inverted_index.medical | 6, 139, 199 |
| abstract_inverted_index.models. | 93 |
| abstract_inverted_index.prompts | 213 |
| abstract_inverted_index.quality | 19 |
| abstract_inverted_index.remains | 290 |
| abstract_inverted_index.systems | 14 |
| abstract_inverted_index.through | 119, 141 |
| abstract_inverted_index.varying | 125 |
| abstract_inverted_index.yielded | 178 |
| abstract_inverted_index.Abstract | 0 |
| abstract_inverted_index.Dresden. | 113 |
| abstract_inverted_index.Finally, | 166 |
| abstract_inverted_index.Hospital | 112 |
| abstract_inverted_index.However, | 37 |
| abstract_inverted_index.Language | 50 |
| abstract_inverted_index.accuracy | 80, 263 |
| abstract_inverted_index.answers, | 266 |
| abstract_inverted_index.assessed | 131, 148 |
| abstract_inverted_index.clinical | 16, 39, 64, 82, 122, 155, 284 |
| abstract_inverted_index.compared | 168 |
| abstract_inverted_index.evaluate | 78 |
| abstract_inverted_index.language | 92 |
| abstract_inverted_index.pipeline | 250 |
| abstract_inverted_index.querying | 279 |
| abstract_inverted_index.question | 206 |
| abstract_inverted_index.reliable | 11 |
| abstract_inverted_index.required | 232 |
| abstract_inverted_index.retrieve | 72 |
| abstract_inverted_index.routine, | 40 |
| abstract_inverted_index.standard | 24 |
| abstract_inverted_index.tailored | 56 |
| abstract_inverted_index.thorough | 60 |
| abstract_inverted_index.Comparing | 219 |
| abstract_inverted_index.Objective | 66 |
| abstract_inverted_index.accuracy, | 210 |
| abstract_inverted_index.achieving | 214 |
| abstract_inverted_index.customize | 68 |
| abstract_inverted_index.different | 87, 136 |
| abstract_inverted_index.efficient | 9 |
| abstract_inverted_index.essential | 291 |
| abstract_inverted_index.evaluated | 118 |
| abstract_inverted_index.expertise | 140 |
| abstract_inverted_index.hospitals | 22 |
| abstract_inverted_index.knowledge | 7, 306 |
| abstract_inverted_index.model’s | 115 |
| abstract_inverted_index.observers | 134, 257 |
| abstract_inverted_index.operating | 25 |
| abstract_inverted_index.oversight | 289 |
| abstract_inverted_index.prompting | 88 |
| abstract_inverted_index.providing | 225 |
| abstract_inverted_index.purposes, | 20 |
| abstract_inverted_index.questions | 123 |
| abstract_inverted_index.settings. | 17 |
| abstract_inverted_index.solution, | 57 |
| abstract_inverted_index.treatment | 35 |
| abstract_inverted_index.(Q&A). | 146 |
| abstract_inverted_index.Acceptance | 190 |
| abstract_inverted_index.Background | 1 |
| abstract_inverted_index.Conclusion | 277 |
| abstract_inverted_index.Customized | 48 |
| abstract_inverted_index.University | 111 |
| abstract_inverted_index.applicable | 317 |
| abstract_inverted_index.clinically | 316 |
| abstract_inverted_index.complexity | 4 |
| abstract_inverted_index.customized | 96, 280, 310 |
| abstract_inverted_index.efficient, | 315 |
| abstract_inverted_index.evaluation | 61, 302 |
| abstract_inverted_index.increasing | 3 |
| abstract_inverted_index.influenced | 208 |
| abstract_inverted_index.iterations | 234 |
| abstract_inverted_index.management | 30 |
| abstract_inverted_index.predefined | 100, 121 |
| abstract_inverted_index.procedures | 26 |
| abstract_inverted_index.responses. | 237 |
| abstract_inverted_index.retrieval, | 286 |
| abstract_inverted_index.self-coded | 249 |
| abstract_inverted_index.standards. | 36 |
| abstract_inverted_index.strategies | 89 |
| abstract_inverted_index.sufficient | 226 |
| abstract_inverted_index.Interactive | 176 |
| abstract_inverted_index.accessible. | 47 |
| abstract_inverted_index.application | 296 |
| abstract_inverted_index.correctness | 151, 185, 261 |
| abstract_inverted_index.departments | 108 |
| abstract_inverted_index.effectively | 246 |
| abstract_inverted_index.information | 12, 29, 42, 73, 273, 285 |
| abstract_inverted_index.interactive | 144 |
| abstract_inverted_index.performance | 116, 170 |
| abstract_inverted_index.sufficiency | 153, 188, 217 |
| abstract_inverted_index.technology. | 299 |
| abstract_inverted_index.application. | 255 |
| abstract_inverted_index.completeness | 183 |
| abstract_inverted_index.early-career | 198 |
| abstract_inverted_index.engineering, | 309 |
| abstract_inverted_index.necessitates | 8 |
| abstract_inverted_index.outperformed | 222 |
| abstract_inverted_index.performance. | 165 |
| abstract_inverted_index.satisfactory | 236 |
| abstract_inverted_index.Claude-3-opus | 221 |
| abstract_inverted_index.Interactively | 278 |
| abstract_inverted_index.completeness, | 150 |
| abstract_inverted_index.discrepancies | 259 |
| abstract_inverted_index.Claude-3-opus. | 174 |
| abstract_inverted_index.implementation | 32 |
| abstract_inverted_index.representation | 271 |
| abstract_inverted_index.implementation. | 65 |
| abstract_inverted_index.hospital-specific | 75 |
| abstract_inverted_index.intermediate-detail | 212 |
| abstract_inverted_index.question-and-answering | 145 |
| cited_by_percentile_year.max | 95 |
| cited_by_percentile_year.min | 91 |
| corresponding_author_ids | https://openalex.org/A5007050711 |
| countries_distinct_count | 1 |
| institutions_distinct_count | 13 |
| corresponding_institution_ids | https://openalex.org/I4210162051 |
| citation_normalized_percentile.value | 0.49929677 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |