Contrasting the performance of mainstream Large Language Models in Radiology Board Examinations Article Swipe
Boxiong Wei
,
Xiumei Zhang
,
Yuhong Shao
,
Xiuming Sun
,
Luzeng Chen
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.21203/rs.3.rs-4573702/v1
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.21203/rs.3.rs-4573702/v1
Objective This study evaluates the performance of mainstream Large Language Models, including GPT-4, Claude, Bard, Tongyi Qianwen, and Gemini Pro, in radiology board exams. Methods A comparative analysis of 150 multiple-choice questions from radiology board exams without images was conducted. Models were assessed on accuracy in text-based questions categorized by cognitive levels and medical specialties using chi-square tests and ANOVA. Results GPT-4 achieved the highest accuracy (83.3%), significantly outperforming others. Tongyi Qianwen also performed well (70.7%). Performance varied across question types and specialties, with GPT-4 excelling in both lower-order and higher-order questions, while Claude and Bard struggled with complex diagnostic questions. Conclusion GPT-4 and Tongyi Qianwen show promise in medical education and training. The study emphasizes the need for domain-specific training datasets to enhance large models' effectiveness in specialized fields like radiology.
Related Topics
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- https://doi.org/10.21203/rs.3.rs-4573702/v1
- OA Status
- gold
- References
- 25
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4400840172
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4400840172Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.21203/rs.3.rs-4573702/v1Digital Object Identifier
- Title
-
Contrasting the performance of mainstream Large Language Models in Radiology Board ExaminationsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-07-19Full publication date if available
- Authors
-
Boxiong Wei, Xiumei Zhang, Yuhong Shao, Xiuming Sun, Luzeng ChenList of authors in order
- Landing page
-
https://doi.org/10.21203/rs.3.rs-4573702/v1Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://doi.org/10.21203/rs.3.rs-4573702/v1Direct OA link when available
- Concepts
-
Mainstream, Computer science, Business, Political science, LawTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- References (count)
-
25Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4400840172 |
|---|---|
| doi | https://doi.org/10.21203/rs.3.rs-4573702/v1 |
| ids.doi | https://doi.org/10.21203/rs.3.rs-4573702/v1 |
| ids.openalex | https://openalex.org/W4400840172 |
| fwci | 0.0 |
| type | preprint |
| title | Contrasting the performance of mainstream Large Language Models in Radiology Board Examinations |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11894 |
| topics[0].field.id | https://openalex.org/fields/27 |
| topics[0].field.display_name | Medicine |
| topics[0].score | 0.9976999759674072 |
| topics[0].domain.id | https://openalex.org/domains/4 |
| topics[0].domain.display_name | Health Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/2741 |
| topics[0].subfield.display_name | Radiology, Nuclear Medicine and Imaging |
| topics[0].display_name | Radiology practices and education |
| topics[1].id | https://openalex.org/T11636 |
| topics[1].field.id | https://openalex.org/fields/27 |
| topics[1].field.display_name | Medicine |
| topics[1].score | 0.9976000189781189 |
| topics[1].domain.id | https://openalex.org/domains/4 |
| topics[1].domain.display_name | Health Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2718 |
| topics[1].subfield.display_name | Health Informatics |
| topics[1].display_name | Artificial Intelligence in Healthcare and Education |
| topics[2].id | https://openalex.org/T12422 |
| topics[2].field.id | https://openalex.org/fields/27 |
| topics[2].field.display_name | Medicine |
| topics[2].score | 0.9958000183105469 |
| topics[2].domain.id | https://openalex.org/domains/4 |
| topics[2].domain.display_name | Health Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/2741 |
| topics[2].subfield.display_name | Radiology, Nuclear Medicine and Imaging |
| topics[2].display_name | Radiomics and Machine Learning in Medical Imaging |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2777617010 |
| concepts[0].level | 2 |
| concepts[0].score | 0.847923755645752 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q18957 |
| concepts[0].display_name | Mainstream |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.34364044666290283 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C144133560 |
| concepts[2].level | 0 |
| concepts[2].score | 0.3245663046836853 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q4830453 |
| concepts[2].display_name | Business |
| concepts[3].id | https://openalex.org/C17744445 |
| concepts[3].level | 0 |
| concepts[3].score | 0.22947335243225098 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q36442 |
| concepts[3].display_name | Political science |
| concepts[4].id | https://openalex.org/C199539241 |
| concepts[4].level | 1 |
| concepts[4].score | 0.0 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q7748 |
| concepts[4].display_name | Law |
| keywords[0].id | https://openalex.org/keywords/mainstream |
| keywords[0].score | 0.847923755645752 |
| keywords[0].display_name | Mainstream |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.34364044666290283 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/business |
| keywords[2].score | 0.3245663046836853 |
| keywords[2].display_name | Business |
| keywords[3].id | https://openalex.org/keywords/political-science |
| keywords[3].score | 0.22947335243225098 |
| keywords[3].display_name | Political science |
| language | en |
| locations[0].id | doi:10.21203/rs.3.rs-4573702/v1 |
| locations[0].is_oa | True |
| locations[0].source | |
| locations[0].license | cc-by |
| locations[0].pdf_url | |
| locations[0].version | acceptedVersion |
| locations[0].raw_type | posted-content |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://doi.org/10.21203/rs.3.rs-4573702/v1 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5036285374 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-6206-9319 |
| authorships[0].author.display_name | Boxiong Wei |
| authorships[0].countries | CN |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I20231570, https://openalex.org/I4210130930 |
| authorships[0].affiliations[0].raw_affiliation_string | Department of ultrasound,Peking University First Hospital |
| authorships[0].institutions[0].id | https://openalex.org/I20231570 |
| authorships[0].institutions[0].ror | https://ror.org/02v51f717 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I20231570 |
| authorships[0].institutions[0].country_code | CN |
| authorships[0].institutions[0].display_name | Peking University |
| authorships[0].institutions[1].id | https://openalex.org/I4210130930 |
| authorships[0].institutions[1].ror | https://ror.org/02z1vqm45 |
| authorships[0].institutions[1].type | healthcare |
| authorships[0].institutions[1].lineage | https://openalex.org/I4210130930 |
| authorships[0].institutions[1].country_code | CN |
| authorships[0].institutions[1].display_name | Peking University First Hospital |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Boxiong Wei |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | Department of ultrasound,Peking University First Hospital |
| authorships[1].author.id | https://openalex.org/A5100729499 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-3168-7971 |
| authorships[1].author.display_name | Xiumei Zhang |
| authorships[1].countries | CN |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I20231570, https://openalex.org/I4210130930 |
| authorships[1].affiliations[0].raw_affiliation_string | Department of ultrasound,Peking University First Hospital |
| authorships[1].institutions[0].id | https://openalex.org/I20231570 |
| authorships[1].institutions[0].ror | https://ror.org/02v51f717 |
| authorships[1].institutions[0].type | education |
| authorships[1].institutions[0].lineage | https://openalex.org/I20231570 |
| authorships[1].institutions[0].country_code | CN |
| authorships[1].institutions[0].display_name | Peking University |
| authorships[1].institutions[1].id | https://openalex.org/I4210130930 |
| authorships[1].institutions[1].ror | https://ror.org/02z1vqm45 |
| authorships[1].institutions[1].type | healthcare |
| authorships[1].institutions[1].lineage | https://openalex.org/I4210130930 |
| authorships[1].institutions[1].country_code | CN |
| authorships[1].institutions[1].display_name | Peking University First Hospital |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Xiumei Zhang |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | Department of ultrasound,Peking University First Hospital |
| authorships[2].author.id | https://openalex.org/A5110832795 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Yuhong Shao |
| authorships[2].countries | CN |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I20231570, https://openalex.org/I4210130930 |
| authorships[2].affiliations[0].raw_affiliation_string | Department of ultrasound,Peking University First Hospital |
| authorships[2].institutions[0].id | https://openalex.org/I20231570 |
| authorships[2].institutions[0].ror | https://ror.org/02v51f717 |
| authorships[2].institutions[0].type | education |
| authorships[2].institutions[0].lineage | https://openalex.org/I20231570 |
| authorships[2].institutions[0].country_code | CN |
| authorships[2].institutions[0].display_name | Peking University |
| authorships[2].institutions[1].id | https://openalex.org/I4210130930 |
| authorships[2].institutions[1].ror | https://ror.org/02z1vqm45 |
| authorships[2].institutions[1].type | healthcare |
| authorships[2].institutions[1].lineage | https://openalex.org/I4210130930 |
| authorships[2].institutions[1].country_code | CN |
| authorships[2].institutions[1].display_name | Peking University First Hospital |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Yuhong Shao |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Department of ultrasound,Peking University First Hospital |
| authorships[3].author.id | https://openalex.org/A5109533921 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Xiuming Sun |
| authorships[3].countries | CN |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I20231570, https://openalex.org/I4210130930 |
| authorships[3].affiliations[0].raw_affiliation_string | Department of ultrasound,Peking University First Hospital |
| authorships[3].institutions[0].id | https://openalex.org/I20231570 |
| authorships[3].institutions[0].ror | https://ror.org/02v51f717 |
| authorships[3].institutions[0].type | education |
| authorships[3].institutions[0].lineage | https://openalex.org/I20231570 |
| authorships[3].institutions[0].country_code | CN |
| authorships[3].institutions[0].display_name | Peking University |
| authorships[3].institutions[1].id | https://openalex.org/I4210130930 |
| authorships[3].institutions[1].ror | https://ror.org/02z1vqm45 |
| authorships[3].institutions[1].type | healthcare |
| authorships[3].institutions[1].lineage | https://openalex.org/I4210130930 |
| authorships[3].institutions[1].country_code | CN |
| authorships[3].institutions[1].display_name | Peking University First Hospital |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Xiuming Sun |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | Department of ultrasound,Peking University First Hospital |
| authorships[4].author.id | https://openalex.org/A5010117357 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-3156-1092 |
| authorships[4].author.display_name | Luzeng Chen |
| authorships[4].countries | CN |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I20231570, https://openalex.org/I4210130930 |
| authorships[4].affiliations[0].raw_affiliation_string | Department of ultrasound,Peking University First Hospital |
| authorships[4].institutions[0].id | https://openalex.org/I20231570 |
| authorships[4].institutions[0].ror | https://ror.org/02v51f717 |
| authorships[4].institutions[0].type | education |
| authorships[4].institutions[0].lineage | https://openalex.org/I20231570 |
| authorships[4].institutions[0].country_code | CN |
| authorships[4].institutions[0].display_name | Peking University |
| authorships[4].institutions[1].id | https://openalex.org/I4210130930 |
| authorships[4].institutions[1].ror | https://ror.org/02z1vqm45 |
| authorships[4].institutions[1].type | healthcare |
| authorships[4].institutions[1].lineage | https://openalex.org/I4210130930 |
| authorships[4].institutions[1].country_code | CN |
| authorships[4].institutions[1].display_name | Peking University First Hospital |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Luzeng Chen |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | Department of ultrasound,Peking University First Hospital |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://doi.org/10.21203/rs.3.rs-4573702/v1 |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Contrasting the performance of mainstream Large Language Models in Radiology Board Examinations |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T11894 |
| primary_topic.field.id | https://openalex.org/fields/27 |
| primary_topic.field.display_name | Medicine |
| primary_topic.score | 0.9976999759674072 |
| primary_topic.domain.id | https://openalex.org/domains/4 |
| primary_topic.domain.display_name | Health Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/2741 |
| primary_topic.subfield.display_name | Radiology, Nuclear Medicine and Imaging |
| primary_topic.display_name | Radiology practices and education |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W1583826057, https://openalex.org/W2377237701, https://openalex.org/W2360099860, https://openalex.org/W4323893170, https://openalex.org/W2352463596, https://openalex.org/W2380850119, https://openalex.org/W2101450440, https://openalex.org/W2383675217 |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.21203/rs.3.rs-4573702/v1 |
| best_oa_location.is_oa | True |
| best_oa_location.source | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | |
| best_oa_location.version | acceptedVersion |
| best_oa_location.raw_type | posted-content |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.21203/rs.3.rs-4573702/v1 |
| primary_location.id | doi:10.21203/rs.3.rs-4573702/v1 |
| primary_location.is_oa | True |
| primary_location.source | |
| primary_location.license | cc-by |
| primary_location.pdf_url | |
| primary_location.version | acceptedVersion |
| primary_location.raw_type | posted-content |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://doi.org/10.21203/rs.3.rs-4573702/v1 |
| publication_date | 2024-07-19 |
| publication_year | 2024 |
| referenced_works | https://openalex.org/W3150212014, https://openalex.org/W3013578857, https://openalex.org/W2785645041, https://openalex.org/W4387327030, https://openalex.org/W4394877296, https://openalex.org/W2963353367, https://openalex.org/W2976398475, https://openalex.org/W3138961261, https://openalex.org/W3177044444, https://openalex.org/W2890106926, https://openalex.org/W3013681994, https://openalex.org/W6778883912, https://openalex.org/W6739901393, https://openalex.org/W4229036772, https://openalex.org/W2783687327, https://openalex.org/W3012043879, https://openalex.org/W2794518994, https://openalex.org/W3011149445, https://openalex.org/W3014301887, https://openalex.org/W4376640706, https://openalex.org/W4376640806, https://openalex.org/W4392665596, https://openalex.org/W3138769367, https://openalex.org/W4292779060, https://openalex.org/W4385245566 |
| referenced_works_count | 25 |
| abstract_inverted_index.A | 26 |
| abstract_inverted_index.by | 50 |
| abstract_inverted_index.in | 21, 46, 87, 109, 128 |
| abstract_inverted_index.of | 7, 29 |
| abstract_inverted_index.on | 44 |
| abstract_inverted_index.to | 123 |
| abstract_inverted_index.150 | 30 |
| abstract_inverted_index.The | 114 |
| abstract_inverted_index.and | 18, 53, 59, 82, 90, 95, 104, 112 |
| abstract_inverted_index.for | 119 |
| abstract_inverted_index.the | 5, 64, 117 |
| abstract_inverted_index.was | 39 |
| abstract_inverted_index.Bard | 96 |
| abstract_inverted_index.Pro, | 20 |
| abstract_inverted_index.This | 2 |
| abstract_inverted_index.also | 73 |
| abstract_inverted_index.both | 88 |
| abstract_inverted_index.from | 33 |
| abstract_inverted_index.like | 131 |
| abstract_inverted_index.need | 118 |
| abstract_inverted_index.show | 107 |
| abstract_inverted_index.well | 75 |
| abstract_inverted_index.were | 42 |
| abstract_inverted_index.with | 84, 98 |
| abstract_inverted_index.Bard, | 15 |
| abstract_inverted_index.GPT-4 | 62, 85, 103 |
| abstract_inverted_index.Large | 9 |
| abstract_inverted_index.board | 23, 35 |
| abstract_inverted_index.exams | 36 |
| abstract_inverted_index.large | 125 |
| abstract_inverted_index.study | 3, 115 |
| abstract_inverted_index.tests | 58 |
| abstract_inverted_index.types | 81 |
| abstract_inverted_index.using | 56 |
| abstract_inverted_index.while | 93 |
| abstract_inverted_index.ANOVA. | 60 |
| abstract_inverted_index.Claude | 94 |
| abstract_inverted_index.GPT-4, | 13 |
| abstract_inverted_index.Gemini | 19 |
| abstract_inverted_index.Models | 41 |
| abstract_inverted_index.Tongyi | 16, 71, 105 |
| abstract_inverted_index.across | 79 |
| abstract_inverted_index.exams. | 24 |
| abstract_inverted_index.fields | 130 |
| abstract_inverted_index.images | 38 |
| abstract_inverted_index.levels | 52 |
| abstract_inverted_index.varied | 78 |
| abstract_inverted_index.Claude, | 14 |
| abstract_inverted_index.Methods | 25 |
| abstract_inverted_index.Models, | 11 |
| abstract_inverted_index.Qianwen | 72, 106 |
| abstract_inverted_index.Results | 61 |
| abstract_inverted_index.complex | 99 |
| abstract_inverted_index.enhance | 124 |
| abstract_inverted_index.highest | 65 |
| abstract_inverted_index.medical | 54, 110 |
| abstract_inverted_index.models' | 126 |
| abstract_inverted_index.others. | 70 |
| abstract_inverted_index.promise | 108 |
| abstract_inverted_index.without | 37 |
| abstract_inverted_index.(70.7%). | 76 |
| abstract_inverted_index.(83.3%), | 67 |
| abstract_inverted_index.Language | 10 |
| abstract_inverted_index.Qianwen, | 17 |
| abstract_inverted_index.accuracy | 45, 66 |
| abstract_inverted_index.achieved | 63 |
| abstract_inverted_index.analysis | 28 |
| abstract_inverted_index.assessed | 43 |
| abstract_inverted_index.datasets | 122 |
| abstract_inverted_index.question | 80 |
| abstract_inverted_index.training | 121 |
| abstract_inverted_index.Objective | 1 |
| abstract_inverted_index.cognitive | 51 |
| abstract_inverted_index.education | 111 |
| abstract_inverted_index.evaluates | 4 |
| abstract_inverted_index.excelling | 86 |
| abstract_inverted_index.including | 12 |
| abstract_inverted_index.performed | 74 |
| abstract_inverted_index.questions | 32, 48 |
| abstract_inverted_index.radiology | 22, 34 |
| abstract_inverted_index.struggled | 97 |
| abstract_inverted_index.training. | 113 |
| abstract_inverted_index.Conclusion | 102 |
| abstract_inverted_index.chi-square | 57 |
| abstract_inverted_index.conducted. | 40 |
| abstract_inverted_index.diagnostic | 100 |
| abstract_inverted_index.emphasizes | 116 |
| abstract_inverted_index.mainstream | 8 |
| abstract_inverted_index.questions, | 92 |
| abstract_inverted_index.questions. | 101 |
| abstract_inverted_index.radiology. | 132 |
| abstract_inverted_index.text-based | 47 |
| abstract_inverted_index.Performance | 77 |
| abstract_inverted_index.categorized | 49 |
| abstract_inverted_index.comparative | 27 |
| abstract_inverted_index.lower-order | 89 |
| abstract_inverted_index.performance | 6 |
| abstract_inverted_index.specialized | 129 |
| abstract_inverted_index.specialties | 55 |
| abstract_inverted_index.higher-order | 91 |
| abstract_inverted_index.specialties, | 83 |
| abstract_inverted_index.effectiveness | 127 |
| abstract_inverted_index.outperforming | 69 |
| abstract_inverted_index.significantly | 68 |
| abstract_inverted_index.domain-specific | 120 |
| abstract_inverted_index.multiple-choice | 31 |
| abstract_inverted_index.<title>Abstract</title> | 0 |
| cited_by_percentile_year | |
| countries_distinct_count | 1 |
| institutions_distinct_count | 5 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.8600000143051147 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile.value | 0.18384383 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |