Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada Article Swipe
Mehdi Mousavi
,
Shabnam Shafiee
,
Jason M. Harley
,
Jackie Chi Kit Cheung
,
Samira Abbasgholizadeh Rahimi
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.1136/fmch-2023-002626
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.1136/fmch-2023-002626
Introduction The application of large language models such as generative pre-trained transformers (GPTs) has been promising in medical education, and its performance has been tested for different medical exams. This study aims to assess the performance of GPTs in responding to a set of sample questions of short-answer management problems (SAMPs) from the certification exam of the College of Family Physicians of Canada (CFPC). Method Between August 8th and 25th, 2023, we used GPT-3.5 and GPT-4 in five rounds to answer a sample of 77 SAMPs questions from the CFPC website. Two independent certified family physician reviewers scored AI-generated responses twice: first, according to the CFPC answer key (ie, CFPC score), and second, based on their knowledge and other references (ie, Reviews’ score). An ordinal logistic generalised estimating equations (GEE) model was applied to analyse repeated measures across the five rounds. Result According to the CFPC answer key, 607 (73.6%) lines of answers by GPT-3.5 and 691 (81%) by GPT-4 were deemed accurate. Reviewer’s scoring suggested that about 84% of the lines of answers provided by GPT-3.5 and 93% of GPT-4 were correct. The GEE analysis confirmed that over five rounds, the likelihood of achieving a higher CFPC Score Percentage for GPT-4 was 2.31 times more than GPT-3.5 (OR: 2.31; 95% CI: 1.53 to 3.47; p<0.001). Similarly, the Reviewers’ Score percentage for responses provided by GPT-4 over 5 rounds were 2.23 times more likely to exceed those of GPT-3.5 (OR: 2.23; 95% CI: 1.22 to 4.06; p=0.009). Running the GPTs after a one week interval, regeneration of the prompt or using or not using the prompt did not significantly change the CFPC score percentage. Conclusion In our study, we used GPT-3.5 and GPT-4 to answer complex, open-ended sample questions of the CFPC exam and showed that more than 70% of the answers were accurate, and GPT-4 outperformed GPT-3.5 in responding to the questions. Large language models such as GPTs seem promising for assisting candidates of the CFPC exam by providing potential answers. However, their use for family medicine education and exam preparation needs further studies.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.1136/fmch-2023-002626
- https://fmch.bmj.com/content/fmch/12/Suppl_1/e002626.full.pdf
- OA Status
- diamond
- Cited By
- 5
- References
- 22
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4399135677
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4399135677Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.1136/fmch-2023-002626Digital Object Identifier
- Title
-
Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of CanadaWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-05-01Full publication date if available
- Authors
-
Mehdi Mousavi, Shabnam Shafiee, Jason M. Harley, Jackie Chi Kit Cheung, Samira Abbasgholizadeh RahimiList of authors in order
- Landing page
-
https://doi.org/10.1136/fmch-2023-002626Publisher landing page
- PDF URL
-
https://fmch.bmj.com/content/fmch/12/Suppl_1/e002626.full.pdfDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
diamondOpen access status per OpenAlex
- OA URL
-
https://fmch.bmj.com/content/fmch/12/Suppl_1/e002626.full.pdfDirect OA link when available
- Concepts
-
Certification, Logistic regression, Medical school, Medicine, Medical education, Family medicine, Mathematics, Internal medicine, Political science, LawTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
5Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 3, 2024: 2Per-year citation counts (last 5 years)
- References (count)
-
22Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4399135677 |
|---|---|
| doi | https://doi.org/10.1136/fmch-2023-002626 |
| ids.doi | https://doi.org/10.1136/fmch-2023-002626 |
| ids.pmid | https://pubmed.ncbi.nlm.nih.gov/38806403 |
| ids.openalex | https://openalex.org/W4399135677 |
| fwci | 1.21699855 |
| mesh[0].qualifier_ui | |
| mesh[0].descriptor_ui | D002170 |
| mesh[0].is_major_topic | False |
| mesh[0].qualifier_name | |
| mesh[0].descriptor_name | Canada |
| mesh[1].qualifier_ui | |
| mesh[1].descriptor_ui | D006801 |
| mesh[1].is_major_topic | False |
| mesh[1].qualifier_name | |
| mesh[1].descriptor_name | Humans |
| mesh[2].qualifier_ui | |
| mesh[2].descriptor_ui | D002568 |
| mesh[2].is_major_topic | True |
| mesh[2].qualifier_name | |
| mesh[2].descriptor_name | Certification |
| mesh[3].qualifier_ui | Q000379 |
| mesh[3].descriptor_ui | D004521 |
| mesh[3].is_major_topic | False |
| mesh[3].qualifier_name | methods |
| mesh[3].descriptor_name | Educational Measurement |
| mesh[4].qualifier_ui | Q000193 |
| mesh[4].descriptor_ui | D010821 |
| mesh[4].is_major_topic | False |
| mesh[4].qualifier_name | education |
| mesh[4].descriptor_name | Physicians, Family |
| mesh[5].qualifier_ui | |
| mesh[5].descriptor_ui | D002983 |
| mesh[5].is_major_topic | False |
| mesh[5].qualifier_name | |
| mesh[5].descriptor_name | Clinical Competence |
| mesh[6].qualifier_ui | Q000193 |
| mesh[6].descriptor_ui | D005194 |
| mesh[6].is_major_topic | False |
| mesh[6].qualifier_name | education |
| mesh[6].descriptor_name | Family Practice |
| mesh[7].qualifier_ui | |
| mesh[7].descriptor_ui | D002170 |
| mesh[7].is_major_topic | False |
| mesh[7].qualifier_name | |
| mesh[7].descriptor_name | Canada |
| mesh[8].qualifier_ui | |
| mesh[8].descriptor_ui | D006801 |
| mesh[8].is_major_topic | False |
| mesh[8].qualifier_name | |
| mesh[8].descriptor_name | Humans |
| mesh[9].qualifier_ui | |
| mesh[9].descriptor_ui | D002568 |
| mesh[9].is_major_topic | True |
| mesh[9].qualifier_name | |
| mesh[9].descriptor_name | Certification |
| mesh[10].qualifier_ui | Q000379 |
| mesh[10].descriptor_ui | D004521 |
| mesh[10].is_major_topic | False |
| mesh[10].qualifier_name | methods |
| mesh[10].descriptor_name | Educational Measurement |
| mesh[11].qualifier_ui | Q000193 |
| mesh[11].descriptor_ui | D010821 |
| mesh[11].is_major_topic | False |
| mesh[11].qualifier_name | education |
| mesh[11].descriptor_name | Physicians, Family |
| mesh[12].qualifier_ui | |
| mesh[12].descriptor_ui | D002983 |
| mesh[12].is_major_topic | False |
| mesh[12].qualifier_name | |
| mesh[12].descriptor_name | Clinical Competence |
| mesh[13].qualifier_ui | Q000193 |
| mesh[13].descriptor_ui | D005194 |
| mesh[13].is_major_topic | False |
| mesh[13].qualifier_name | education |
| mesh[13].descriptor_name | Family Practice |
| mesh[14].qualifier_ui | |
| mesh[14].descriptor_ui | D002170 |
| mesh[14].is_major_topic | False |
| mesh[14].qualifier_name | |
| mesh[14].descriptor_name | Canada |
| mesh[15].qualifier_ui | |
| mesh[15].descriptor_ui | D006801 |
| mesh[15].is_major_topic | False |
| mesh[15].qualifier_name | |
| mesh[15].descriptor_name | Humans |
| mesh[16].qualifier_ui | |
| mesh[16].descriptor_ui | D002568 |
| mesh[16].is_major_topic | True |
| mesh[16].qualifier_name | |
| mesh[16].descriptor_name | Certification |
| mesh[17].qualifier_ui | Q000379 |
| mesh[17].descriptor_ui | D004521 |
| mesh[17].is_major_topic | False |
| mesh[17].qualifier_name | methods |
| mesh[17].descriptor_name | Educational Measurement |
| mesh[18].qualifier_ui | Q000193 |
| mesh[18].descriptor_ui | D010821 |
| mesh[18].is_major_topic | False |
| mesh[18].qualifier_name | education |
| mesh[18].descriptor_name | Physicians, Family |
| mesh[19].qualifier_ui | |
| mesh[19].descriptor_ui | D002983 |
| mesh[19].is_major_topic | False |
| mesh[19].qualifier_name | |
| mesh[19].descriptor_name | Clinical Competence |
| mesh[20].qualifier_ui | Q000193 |
| mesh[20].descriptor_ui | D005194 |
| mesh[20].is_major_topic | False |
| mesh[20].qualifier_name | education |
| mesh[20].descriptor_name | Family Practice |
| mesh[21].qualifier_ui | |
| mesh[21].descriptor_ui | D002170 |
| mesh[21].is_major_topic | False |
| mesh[21].qualifier_name | |
| mesh[21].descriptor_name | Canada |
| mesh[22].qualifier_ui | |
| mesh[22].descriptor_ui | D006801 |
| mesh[22].is_major_topic | False |
| mesh[22].qualifier_name | |
| mesh[22].descriptor_name | Humans |
| mesh[23].qualifier_ui | |
| mesh[23].descriptor_ui | D002568 |
| mesh[23].is_major_topic | True |
| mesh[23].qualifier_name | |
| mesh[23].descriptor_name | Certification |
| mesh[24].qualifier_ui | Q000379 |
| mesh[24].descriptor_ui | D004521 |
| mesh[24].is_major_topic | False |
| mesh[24].qualifier_name | methods |
| mesh[24].descriptor_name | Educational Measurement |
| mesh[25].qualifier_ui | Q000193 |
| mesh[25].descriptor_ui | D010821 |
| mesh[25].is_major_topic | False |
| mesh[25].qualifier_name | education |
| mesh[25].descriptor_name | Physicians, Family |
| mesh[26].qualifier_ui | |
| mesh[26].descriptor_ui | D002983 |
| mesh[26].is_major_topic | False |
| mesh[26].qualifier_name | |
| mesh[26].descriptor_name | Clinical Competence |
| mesh[27].qualifier_ui | Q000193 |
| mesh[27].descriptor_ui | D005194 |
| mesh[27].is_major_topic | False |
| mesh[27].qualifier_name | education |
| mesh[27].descriptor_name | Family Practice |
| type | article |
| title | Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada |
| biblio.issue | Suppl 1 |
| biblio.volume | 12 |
| biblio.last_page | e002626 |
| biblio.first_page | e002626 |
| topics[0].id | https://openalex.org/T11636 |
| topics[0].field.id | https://openalex.org/fields/27 |
| topics[0].field.display_name | Medicine |
| topics[0].score | 0.9997000098228455 |
| topics[0].domain.id | https://openalex.org/domains/4 |
| topics[0].domain.display_name | Health Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/2718 |
| topics[0].subfield.display_name | Health Informatics |
| topics[0].display_name | Artificial Intelligence in Healthcare and Education |
| topics[1].id | https://openalex.org/T12422 |
| topics[1].field.id | https://openalex.org/fields/27 |
| topics[1].field.display_name | Medicine |
| topics[1].score | 0.9154999852180481 |
| topics[1].domain.id | https://openalex.org/domains/4 |
| topics[1].domain.display_name | Health Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2741 |
| topics[1].subfield.display_name | Radiology, Nuclear Medicine and Imaging |
| topics[1].display_name | Radiomics and Machine Learning in Medical Imaging |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C46304622 |
| concepts[0].level | 2 |
| concepts[0].score | 0.6730194091796875 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q374814 |
| concepts[0].display_name | Certification |
| concepts[1].id | https://openalex.org/C151956035 |
| concepts[1].level | 2 |
| concepts[1].score | 0.4641507863998413 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q1132755 |
| concepts[1].display_name | Logistic regression |
| concepts[2].id | https://openalex.org/C2993838110 |
| concepts[2].level | 2 |
| concepts[2].score | 0.41878998279571533 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q494230 |
| concepts[2].display_name | Medical school |
| concepts[3].id | https://openalex.org/C71924100 |
| concepts[3].level | 0 |
| concepts[3].score | 0.417908638715744 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11190 |
| concepts[3].display_name | Medicine |
| concepts[4].id | https://openalex.org/C509550671 |
| concepts[4].level | 1 |
| concepts[4].score | 0.34672144055366516 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q126945 |
| concepts[4].display_name | Medical education |
| concepts[5].id | https://openalex.org/C512399662 |
| concepts[5].level | 1 |
| concepts[5].score | 0.34612762928009033 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q3505712 |
| concepts[5].display_name | Family medicine |
| concepts[6].id | https://openalex.org/C33923547 |
| concepts[6].level | 0 |
| concepts[6].score | 0.3411613702774048 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[6].display_name | Mathematics |
| concepts[7].id | https://openalex.org/C126322002 |
| concepts[7].level | 1 |
| concepts[7].score | 0.331138551235199 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11180 |
| concepts[7].display_name | Internal medicine |
| concepts[8].id | https://openalex.org/C17744445 |
| concepts[8].level | 0 |
| concepts[8].score | 0.10474440455436707 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q36442 |
| concepts[8].display_name | Political science |
| concepts[9].id | https://openalex.org/C199539241 |
| concepts[9].level | 1 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q7748 |
| concepts[9].display_name | Law |
| keywords[0].id | https://openalex.org/keywords/certification |
| keywords[0].score | 0.6730194091796875 |
| keywords[0].display_name | Certification |
| keywords[1].id | https://openalex.org/keywords/logistic-regression |
| keywords[1].score | 0.4641507863998413 |
| keywords[1].display_name | Logistic regression |
| keywords[2].id | https://openalex.org/keywords/medical-school |
| keywords[2].score | 0.41878998279571533 |
| keywords[2].display_name | Medical school |
| keywords[3].id | https://openalex.org/keywords/medicine |
| keywords[3].score | 0.417908638715744 |
| keywords[3].display_name | Medicine |
| keywords[4].id | https://openalex.org/keywords/medical-education |
| keywords[4].score | 0.34672144055366516 |
| keywords[4].display_name | Medical education |
| keywords[5].id | https://openalex.org/keywords/family-medicine |
| keywords[5].score | 0.34612762928009033 |
| keywords[5].display_name | Family medicine |
| keywords[6].id | https://openalex.org/keywords/mathematics |
| keywords[6].score | 0.3411613702774048 |
| keywords[6].display_name | Mathematics |
| keywords[7].id | https://openalex.org/keywords/internal-medicine |
| keywords[7].score | 0.331138551235199 |
| keywords[7].display_name | Internal medicine |
| keywords[8].id | https://openalex.org/keywords/political-science |
| keywords[8].score | 0.10474440455436707 |
| keywords[8].display_name | Political science |
| language | en |
| locations[0].id | doi:10.1136/fmch-2023-002626 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S2764498025 |
| locations[0].source.issn | 2009-8774, 2305-6983 |
| locations[0].source.type | journal |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | 2009-8774 |
| locations[0].source.is_core | True |
| locations[0].source.is_in_doaj | True |
| locations[0].source.display_name | Family Medicine and Community Health |
| locations[0].source.host_organization | https://openalex.org/P4310319945 |
| locations[0].source.host_organization_name | BMJ |
| locations[0].source.host_organization_lineage | https://openalex.org/P4310319945 |
| locations[0].license | cc-by-nc |
| locations[0].pdf_url | https://fmch.bmj.com/content/fmch/12/Suppl_1/e002626.full.pdf |
| locations[0].version | publishedVersion |
| locations[0].raw_type | journal-article |
| locations[0].license_id | https://openalex.org/licenses/cc-by-nc |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | Family Medicine and Community Health |
| locations[0].landing_page_url | https://doi.org/10.1136/fmch-2023-002626 |
| locations[1].id | pmid:38806403 |
| locations[1].is_oa | False |
| locations[1].source.id | https://openalex.org/S4306525036 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | False |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | PubMed |
| locations[1].source.host_organization | https://openalex.org/I1299303238 |
| locations[1].source.host_organization_name | National Institutes of Health |
| locations[1].source.host_organization_lineage | https://openalex.org/I1299303238 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | publishedVersion |
| locations[1].raw_type | |
| locations[1].license_id | |
| locations[1].is_accepted | True |
| locations[1].is_published | True |
| locations[1].raw_source_name | Family medicine and community health |
| locations[1].landing_page_url | https://pubmed.ncbi.nlm.nih.gov/38806403 |
| locations[2].id | pmh:oai:doaj.org/article:b6c4e32a4c1848cd83543a9c812d4a3a |
| locations[2].is_oa | False |
| locations[2].source.id | https://openalex.org/S4306401280 |
| locations[2].source.issn | |
| locations[2].source.type | repository |
| locations[2].source.is_oa | False |
| locations[2].source.issn_l | |
| locations[2].source.is_core | False |
| locations[2].source.is_in_doaj | False |
| locations[2].source.display_name | DOAJ (DOAJ: Directory of Open Access Journals) |
| locations[2].source.host_organization | |
| locations[2].source.host_organization_name | |
| locations[2].source.host_organization_lineage | |
| locations[2].license | |
| locations[2].pdf_url | |
| locations[2].version | submittedVersion |
| locations[2].raw_type | article |
| locations[2].license_id | |
| locations[2].is_accepted | False |
| locations[2].is_published | False |
| locations[2].raw_source_name | Family Medicine and Community Health, Vol 12, Iss Suppl 1 (2024) |
| locations[2].landing_page_url | https://doaj.org/article/b6c4e32a4c1848cd83543a9c812d4a3a |
| locations[3].id | pmh:oai:pubmedcentral.nih.gov:11138270 |
| locations[3].is_oa | True |
| locations[3].source.id | https://openalex.org/S2764455111 |
| locations[3].source.issn | |
| locations[3].source.type | repository |
| locations[3].source.is_oa | False |
| locations[3].source.issn_l | |
| locations[3].source.is_core | False |
| locations[3].source.is_in_doaj | False |
| locations[3].source.display_name | PubMed Central |
| locations[3].source.host_organization | https://openalex.org/I1299303238 |
| locations[3].source.host_organization_name | National Institutes of Health |
| locations[3].source.host_organization_lineage | https://openalex.org/I1299303238 |
| locations[3].license | other-oa |
| locations[3].pdf_url | |
| locations[3].version | submittedVersion |
| locations[3].raw_type | Text |
| locations[3].license_id | https://openalex.org/licenses/other-oa |
| locations[3].is_accepted | False |
| locations[3].is_published | False |
| locations[3].raw_source_name | Fam Med Community Health |
| locations[3].landing_page_url | https://www.ncbi.nlm.nih.gov/pmc/articles/11138270 |
| indexed_in | crossref, doaj, pubmed |
| authorships[0].author.id | https://openalex.org/A5015943915 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-3644-5741 |
| authorships[0].author.display_name | Mehdi Mousavi |
| authorships[0].countries | CA |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I32625721 |
| authorships[0].affiliations[0].raw_affiliation_string | Department of Family Medicine, Faculty of Medicine, University of Saskatchewan, Nipawin, Saskatchewan, Canada |
| authorships[0].institutions[0].id | https://openalex.org/I32625721 |
| authorships[0].institutions[0].ror | https://ror.org/010x8gc63 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I32625721 |
| authorships[0].institutions[0].country_code | CA |
| authorships[0].institutions[0].display_name | University of Saskatchewan |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Mehdi Mousavi |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | Department of Family Medicine, Faculty of Medicine, University of Saskatchewan, Nipawin, Saskatchewan, Canada |
| authorships[1].author.id | https://openalex.org/A5063696866 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Shabnam Shafiee |
| authorships[1].countries | CA |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I23625684, https://openalex.org/I2799377091 |
| authorships[1].affiliations[0].raw_affiliation_string | Department of Family Medicine, Saskatchewan Health Authority, Riverside Health Complex, Turtleford, Saskatchewan, Canada |
| authorships[1].institutions[0].id | https://openalex.org/I2799377091 |
| authorships[1].institutions[0].ror | https://ror.org/05m2a2x14 |
| authorships[1].institutions[0].type | government |
| authorships[1].institutions[0].lineage | https://openalex.org/I2799377091, https://openalex.org/I4210143365 |
| authorships[1].institutions[0].country_code | CA |
| authorships[1].institutions[0].display_name | Saskatchewan Health |
| authorships[1].institutions[1].id | https://openalex.org/I23625684 |
| authorships[1].institutions[1].ror | https://ror.org/02wtdvm35 |
| authorships[1].institutions[1].type | healthcare |
| authorships[1].institutions[1].lineage | https://openalex.org/I23625684 |
| authorships[1].institutions[1].country_code | CA |
| authorships[1].institutions[1].display_name | Saskatchewan Health Authority |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Shabnam Shafiee |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | Department of Family Medicine, Saskatchewan Health Authority, Riverside Health Complex, Turtleford, Saskatchewan, Canada |
| authorships[2].author.id | https://openalex.org/A5046516907 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-2061-9519 |
| authorships[2].author.display_name | Jason M. Harley |
| authorships[2].countries | CA |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I5023651 |
| authorships[2].affiliations[0].raw_affiliation_string | Department of Surgery, Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada |
| authorships[2].affiliations[1].institution_ids | https://openalex.org/I2801766832 |
| authorships[2].affiliations[1].raw_affiliation_string | Research Institute of the McGill University Health Centre, Montreal, Quebec, Canada |
| authorships[2].affiliations[2].institution_ids | https://openalex.org/I2801766832 |
| authorships[2].affiliations[2].raw_affiliation_string | Institute for Health Sciences Education, Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada |
| authorships[2].institutions[0].id | https://openalex.org/I5023651 |
| authorships[2].institutions[0].ror | https://ror.org/01pxwe438 |
| authorships[2].institutions[0].type | education |
| authorships[2].institutions[0].lineage | https://openalex.org/I5023651 |
| authorships[2].institutions[0].country_code | CA |
| authorships[2].institutions[0].display_name | McGill University |
| authorships[2].institutions[1].id | https://openalex.org/I2801766832 |
| authorships[2].institutions[1].ror | https://ror.org/04cpxjv19 |
| authorships[2].institutions[1].type | healthcare |
| authorships[2].institutions[1].lineage | https://openalex.org/I2801766832 |
| authorships[2].institutions[1].country_code | CA |
| authorships[2].institutions[1].display_name | McGill University Health Centre |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Jason M Harley |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Department of Surgery, Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada, Institute for Health Sciences Education, Faculty of Medicine and Health Sciences, McGill University, Montreal, Quebec, Canada, Research Institute of the McGill University Health Centre, Montreal, Quebec, Canada |
| authorships[3].author.id | https://openalex.org/A5050801868 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Jackie Chi Kit Cheung |
| authorships[3].countries | CA |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I5023651 |
| authorships[3].affiliations[0].raw_affiliation_string | McGill University School of Computer Science, Montreal, Quebec, Canada |
| authorships[3].affiliations[1].institution_ids | https://openalex.org/I109736498, https://openalex.org/I4210164802 |
| authorships[3].affiliations[1].raw_affiliation_string | CIFAR AI Chair, Mila-Quebec AI Institute, Montreal, Quebec, Canada |
| authorships[3].institutions[0].id | https://openalex.org/I109736498 |
| authorships[3].institutions[0].ror | https://ror.org/01sdtdd95 |
| authorships[3].institutions[0].type | facility |
| authorships[3].institutions[0].lineage | https://openalex.org/I109736498 |
| authorships[3].institutions[0].country_code | CA |
| authorships[3].institutions[0].display_name | Canadian Institute for Advanced Research |
| authorships[3].institutions[1].id | https://openalex.org/I5023651 |
| authorships[3].institutions[1].ror | https://ror.org/01pxwe438 |
| authorships[3].institutions[1].type | education |
| authorships[3].institutions[1].lineage | https://openalex.org/I5023651 |
| authorships[3].institutions[1].country_code | CA |
| authorships[3].institutions[1].display_name | McGill University |
| authorships[3].institutions[2].id | https://openalex.org/I4210164802 |
| authorships[3].institutions[2].ror | https://ror.org/05c22rx21 |
| authorships[3].institutions[2].type | facility |
| authorships[3].institutions[2].lineage | https://openalex.org/I4210164802 |
| authorships[3].institutions[2].country_code | CA |
| authorships[3].institutions[2].display_name | Mila - Quebec Artificial Intelligence Institute |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Jackie Chi Kit Cheung |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | CIFAR AI Chair, Mila-Quebec AI Institute, Montreal, Quebec, Canada, McGill University School of Computer Science, Montreal, Quebec, Canada |
| authorships[4].author.id | https://openalex.org/A5024597650 |
| authorships[4].author.orcid | https://orcid.org/0000-0003-3781-1360 |
| authorships[4].author.display_name | Samira Abbasgholizadeh Rahimi |
| authorships[4].countries | CA |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I5023651 |
| authorships[4].affiliations[0].raw_affiliation_string | Faculty of Dentistry Medicine and Oral Health Sciences, McGill University, Montreal, Quebec, Canada |
| authorships[4].affiliations[1].institution_ids | https://openalex.org/I5023651 |
| authorships[4].affiliations[1].raw_affiliation_string | Department of Family Medicine, McGill University, Montreal, Quebec, Canada |
| authorships[4].affiliations[2].institution_ids | https://openalex.org/I2801824472 |
| authorships[4].affiliations[2].raw_affiliation_string | Lady Davis Institute for Medical Research, Jewish General Hospital, Montreal, Quebec, Canada |
| authorships[4].affiliations[3].institution_ids | https://openalex.org/I4210164802 |
| authorships[4].affiliations[3].raw_affiliation_string | Mila Quebec AI-Institute, Montreal, Quebec, Canada |
| authorships[4].institutions[0].id | https://openalex.org/I2801824472 |
| authorships[4].institutions[0].ror | https://ror.org/056jjra10 |
| authorships[4].institutions[0].type | healthcare |
| authorships[4].institutions[0].lineage | https://openalex.org/I2801824472 |
| authorships[4].institutions[0].country_code | CA |
| authorships[4].institutions[0].display_name | Jewish General Hospital |
| authorships[4].institutions[1].id | https://openalex.org/I5023651 |
| authorships[4].institutions[1].ror | https://ror.org/01pxwe438 |
| authorships[4].institutions[1].type | education |
| authorships[4].institutions[1].lineage | https://openalex.org/I5023651 |
| authorships[4].institutions[1].country_code | CA |
| authorships[4].institutions[1].display_name | McGill University |
| authorships[4].institutions[2].id | https://openalex.org/I4210164802 |
| authorships[4].institutions[2].ror | https://ror.org/05c22rx21 |
| authorships[4].institutions[2].type | facility |
| authorships[4].institutions[2].lineage | https://openalex.org/I4210164802 |
| authorships[4].institutions[2].country_code | CA |
| authorships[4].institutions[2].display_name | Mila - Quebec Artificial Intelligence Institute |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Samira Abbasgholizadeh Rahimi |
| authorships[4].is_corresponding | True |
| authorships[4].raw_affiliation_strings | Department of Family Medicine, McGill University, Montreal, Quebec, Canada, Faculty of Dentistry Medicine and Oral Health Sciences, McGill University, Montreal, Quebec, Canada, Lady Davis Institute for Medical Research, Jewish General Hospital, Montreal, Quebec, Canada, Mila Quebec AI-Institute, Montreal, Quebec, Canada |
| has_content.pdf | True |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://fmch.bmj.com/content/fmch/12/Suppl_1/e002626.full.pdf |
| open_access.oa_status | diamond |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Performance of generative pre-trained transformers (GPTs) in Certification Examination of the College of Family Physicians of Canada |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T11636 |
| primary_topic.field.id | https://openalex.org/fields/27 |
| primary_topic.field.display_name | Medicine |
| primary_topic.score | 0.9997000098228455 |
| primary_topic.domain.id | https://openalex.org/domains/4 |
| primary_topic.domain.display_name | Health Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/2718 |
| primary_topic.subfield.display_name | Health Informatics |
| primary_topic.display_name | Artificial Intelligence in Healthcare and Education |
| related_works | https://openalex.org/W2066052364, https://openalex.org/W2023743128, https://openalex.org/W4243365217, https://openalex.org/W2224296908, https://openalex.org/W3109981693, https://openalex.org/W2470050756, https://openalex.org/W1998275520, https://openalex.org/W2151193840, https://openalex.org/W2462672465, https://openalex.org/W1976464718 |
| cited_by_count | 5 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 3 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 2 |
| locations_count | 4 |
| best_oa_location.id | doi:10.1136/fmch-2023-002626 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S2764498025 |
| best_oa_location.source.issn | 2009-8774, 2305-6983 |
| best_oa_location.source.type | journal |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | 2009-8774 |
| best_oa_location.source.is_core | True |
| best_oa_location.source.is_in_doaj | True |
| best_oa_location.source.display_name | Family Medicine and Community Health |
| best_oa_location.source.host_organization | https://openalex.org/P4310319945 |
| best_oa_location.source.host_organization_name | BMJ |
| best_oa_location.source.host_organization_lineage | https://openalex.org/P4310319945 |
| best_oa_location.license | cc-by-nc |
| best_oa_location.pdf_url | https://fmch.bmj.com/content/fmch/12/Suppl_1/e002626.full.pdf |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | journal-article |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by-nc |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | Family Medicine and Community Health |
| best_oa_location.landing_page_url | https://doi.org/10.1136/fmch-2023-002626 |
| primary_location.id | doi:10.1136/fmch-2023-002626 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S2764498025 |
| primary_location.source.issn | 2009-8774, 2305-6983 |
| primary_location.source.type | journal |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | 2009-8774 |
| primary_location.source.is_core | True |
| primary_location.source.is_in_doaj | True |
| primary_location.source.display_name | Family Medicine and Community Health |
| primary_location.source.host_organization | https://openalex.org/P4310319945 |
| primary_location.source.host_organization_name | BMJ |
| primary_location.source.host_organization_lineage | https://openalex.org/P4310319945 |
| primary_location.license | cc-by-nc |
| primary_location.pdf_url | https://fmch.bmj.com/content/fmch/12/Suppl_1/e002626.full.pdf |
| primary_location.version | publishedVersion |
| primary_location.raw_type | journal-article |
| primary_location.license_id | https://openalex.org/licenses/cc-by-nc |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | Family Medicine and Community Health |
| primary_location.landing_page_url | https://doi.org/10.1136/fmch-2023-002626 |
| publication_date | 2024-05-01 |
| publication_year | 2024 |
| referenced_works | https://openalex.org/W4319454867, https://openalex.org/W4319350602, https://openalex.org/W4320920036, https://openalex.org/W4363645798, https://openalex.org/W4327946446, https://openalex.org/W4391069573, https://openalex.org/W4386457251, https://openalex.org/W4319662928, https://openalex.org/W4319460874, https://openalex.org/W4364378939, https://openalex.org/W6854376192, https://openalex.org/W4380786006, https://openalex.org/W4376640725, https://openalex.org/W4380685958, https://openalex.org/W4321018785, https://openalex.org/W4385430748, https://openalex.org/W4368367885, https://openalex.org/W4366420437, https://openalex.org/W3196769534, https://openalex.org/W4387327030, https://openalex.org/W4385827193, https://openalex.org/W4381743186 |
| referenced_works_count | 22 |
| abstract_inverted_index.5 | 227 |
| abstract_inverted_index.a | 41, 81, 195, 251 |
| abstract_inverted_index.77 | 84 |
| abstract_inverted_index.An | 123 |
| abstract_inverted_index.In | 275 |
| abstract_inverted_index.as | 8, 317 |
| abstract_inverted_index.by | 153, 158, 175, 224, 328 |
| abstract_inverted_index.in | 16, 38, 76, 308 |
| abstract_inverted_index.of | 3, 36, 43, 46, 55, 58, 61, 83, 151, 169, 172, 179, 193, 237, 256, 289, 299, 324 |
| abstract_inverted_index.on | 114 |
| abstract_inverted_index.or | 259, 261 |
| abstract_inverted_index.to | 32, 40, 79, 103, 133, 143, 213, 234, 244, 283, 310 |
| abstract_inverted_index.we | 71, 278 |
| abstract_inverted_index.607 | 148 |
| abstract_inverted_index.691 | 156 |
| abstract_inverted_index.70% | 298 |
| abstract_inverted_index.84% | 168 |
| abstract_inverted_index.8th | 67 |
| abstract_inverted_index.93% | 178 |
| abstract_inverted_index.95% | 210, 241 |
| abstract_inverted_index.CI: | 211, 242 |
| abstract_inverted_index.GEE | 184 |
| abstract_inverted_index.The | 1, 183 |
| abstract_inverted_index.Two | 91 |
| abstract_inverted_index.and | 19, 68, 74, 111, 117, 155, 177, 281, 293, 304, 339 |
| abstract_inverted_index.did | 266 |
| abstract_inverted_index.for | 25, 200, 221, 321, 335 |
| abstract_inverted_index.has | 13, 22 |
| abstract_inverted_index.its | 20 |
| abstract_inverted_index.key | 107 |
| abstract_inverted_index.not | 262, 267 |
| abstract_inverted_index.one | 252 |
| abstract_inverted_index.our | 276 |
| abstract_inverted_index.set | 42 |
| abstract_inverted_index.the | 34, 52, 56, 88, 104, 138, 144, 170, 191, 217, 248, 257, 264, 270, 290, 300, 311, 325 |
| abstract_inverted_index.use | 334 |
| abstract_inverted_index.was | 131, 202 |
| abstract_inverted_index.(OR: | 208, 239 |
| abstract_inverted_index.(ie, | 108, 120 |
| abstract_inverted_index.1.22 | 243 |
| abstract_inverted_index.1.53 | 212 |
| abstract_inverted_index.2.23 | 230 |
| abstract_inverted_index.2.31 | 203 |
| abstract_inverted_index.CFPC | 89, 105, 109, 145, 197, 271, 291, 326 |
| abstract_inverted_index.GPTs | 37, 249, 318 |
| abstract_inverted_index.This | 29 |
| abstract_inverted_index.aims | 31 |
| abstract_inverted_index.been | 14, 23 |
| abstract_inverted_index.exam | 54, 292, 327, 340 |
| abstract_inverted_index.five | 77, 139, 189 |
| abstract_inverted_index.from | 51, 87 |
| abstract_inverted_index.key, | 147 |
| abstract_inverted_index.more | 205, 232, 296 |
| abstract_inverted_index.over | 188, 226 |
| abstract_inverted_index.seem | 319 |
| abstract_inverted_index.such | 7, 316 |
| abstract_inverted_index.than | 206, 297 |
| abstract_inverted_index.that | 166, 187, 295 |
| abstract_inverted_index.used | 72, 279 |
| abstract_inverted_index.week | 253 |
| abstract_inverted_index.were | 160, 181, 229, 302 |
| abstract_inverted_index.(81%) | 157 |
| abstract_inverted_index.(GEE) | 129 |
| abstract_inverted_index.2.23; | 240 |
| abstract_inverted_index.2.31; | 209 |
| abstract_inverted_index.2023, | 70 |
| abstract_inverted_index.25th, | 69 |
| abstract_inverted_index.3.47; | 214 |
| abstract_inverted_index.4.06; | 245 |
| abstract_inverted_index.GPT-4 | 75, 159, 180, 201, 225, 282, 305 |
| abstract_inverted_index.Large | 313 |
| abstract_inverted_index.SAMPs | 85 |
| abstract_inverted_index.Score | 198, 219 |
| abstract_inverted_index.about | 167 |
| abstract_inverted_index.after | 250 |
| abstract_inverted_index.based | 113 |
| abstract_inverted_index.large | 4 |
| abstract_inverted_index.lines | 150, 171 |
| abstract_inverted_index.model | 130 |
| abstract_inverted_index.needs | 342 |
| abstract_inverted_index.other | 118 |
| abstract_inverted_index.score | 272 |
| abstract_inverted_index.study | 30 |
| abstract_inverted_index.their | 115, 333 |
| abstract_inverted_index.those | 236 |
| abstract_inverted_index.times | 204, 231 |
| abstract_inverted_index.using | 260, 263 |
| abstract_inverted_index.(GPTs) | 12 |
| abstract_inverted_index.August | 66 |
| abstract_inverted_index.Canada | 62 |
| abstract_inverted_index.Family | 59 |
| abstract_inverted_index.Method | 64 |
| abstract_inverted_index.Result | 141 |
| abstract_inverted_index.across | 137 |
| abstract_inverted_index.answer | 80, 106, 146, 284 |
| abstract_inverted_index.assess | 33 |
| abstract_inverted_index.change | 269 |
| abstract_inverted_index.deemed | 161 |
| abstract_inverted_index.exams. | 28 |
| abstract_inverted_index.exceed | 235 |
| abstract_inverted_index.family | 94, 336 |
| abstract_inverted_index.first, | 101 |
| abstract_inverted_index.higher | 196 |
| abstract_inverted_index.likely | 233 |
| abstract_inverted_index.models | 6, 315 |
| abstract_inverted_index.prompt | 258, 265 |
| abstract_inverted_index.rounds | 78, 228 |
| abstract_inverted_index.sample | 44, 82, 287 |
| abstract_inverted_index.scored | 97 |
| abstract_inverted_index.showed | 294 |
| abstract_inverted_index.study, | 277 |
| abstract_inverted_index.tested | 24 |
| abstract_inverted_index.twice: | 100 |
| abstract_inverted_index.(73.6%) | 149 |
| abstract_inverted_index.(CFPC). | 63 |
| abstract_inverted_index.(SAMPs) | 50 |
| abstract_inverted_index.Between | 65 |
| abstract_inverted_index.College | 57 |
| abstract_inverted_index.GPT-3.5 | 73, 154, 176, 207, 238, 280, 307 |
| abstract_inverted_index.Running | 247 |
| abstract_inverted_index.analyse | 134 |
| abstract_inverted_index.answers | 152, 173, 301 |
| abstract_inverted_index.applied | 132 |
| abstract_inverted_index.further | 343 |
| abstract_inverted_index.medical | 17, 27 |
| abstract_inverted_index.ordinal | 124 |
| abstract_inverted_index.rounds, | 190 |
| abstract_inverted_index.rounds. | 140 |
| abstract_inverted_index.score), | 110 |
| abstract_inverted_index.score). | 122 |
| abstract_inverted_index.scoring | 164 |
| abstract_inverted_index.second, | 112 |
| abstract_inverted_index.However, | 332 |
| abstract_inverted_index.analysis | 185 |
| abstract_inverted_index.answers. | 331 |
| abstract_inverted_index.complex, | 285 |
| abstract_inverted_index.correct. | 182 |
| abstract_inverted_index.language | 5, 314 |
| abstract_inverted_index.logistic | 125 |
| abstract_inverted_index.measures | 136 |
| abstract_inverted_index.medicine | 337 |
| abstract_inverted_index.problems | 49 |
| abstract_inverted_index.provided | 174, 223 |
| abstract_inverted_index.repeated | 135 |
| abstract_inverted_index.studies. | 344 |
| abstract_inverted_index.website. | 90 |
| abstract_inverted_index.According | 142 |
| abstract_inverted_index.according | 102 |
| abstract_inverted_index.accurate, | 303 |
| abstract_inverted_index.accurate. | 162 |
| abstract_inverted_index.achieving | 194 |
| abstract_inverted_index.assisting | 322 |
| abstract_inverted_index.certified | 93 |
| abstract_inverted_index.confirmed | 186 |
| abstract_inverted_index.different | 26 |
| abstract_inverted_index.education | 338 |
| abstract_inverted_index.equations | 128 |
| abstract_inverted_index.interval, | 254 |
| abstract_inverted_index.knowledge | 116 |
| abstract_inverted_index.p=0.009). | 246 |
| abstract_inverted_index.physician | 95 |
| abstract_inverted_index.potential | 330 |
| abstract_inverted_index.promising | 15, 320 |
| abstract_inverted_index.providing | 329 |
| abstract_inverted_index.questions | 45, 86, 288 |
| abstract_inverted_index.responses | 99, 222 |
| abstract_inverted_index.reviewers | 96 |
| abstract_inverted_index.suggested | 165 |
| abstract_inverted_index.Conclusion | 274 |
| abstract_inverted_index.Percentage | 199 |
| abstract_inverted_index.Physicians | 60 |
| abstract_inverted_index.Reviews’ | 121 |
| abstract_inverted_index.Similarly, | 216 |
| abstract_inverted_index.candidates | 323 |
| abstract_inverted_index.education, | 18 |
| abstract_inverted_index.estimating | 127 |
| abstract_inverted_index.generative | 9 |
| abstract_inverted_index.likelihood | 192 |
| abstract_inverted_index.management | 48 |
| abstract_inverted_index.open-ended | 286 |
| abstract_inverted_index.percentage | 220 |
| abstract_inverted_index.questions. | 312 |
| abstract_inverted_index.references | 119 |
| abstract_inverted_index.responding | 39, 309 |
| abstract_inverted_index.application | 2 |
| abstract_inverted_index.generalised | 126 |
| abstract_inverted_index.independent | 92 |
| abstract_inverted_index.percentage. | 273 |
| abstract_inverted_index.performance | 21, 35 |
| abstract_inverted_index.pre-trained | 10 |
| abstract_inverted_index.preparation | 341 |
| abstract_inverted_index.AI-generated | 98 |
| abstract_inverted_index.Introduction | 0 |
| abstract_inverted_index.Reviewers’ | 218 |
| abstract_inverted_index.Reviewer’s | 163 |
| abstract_inverted_index.outperformed | 306 |
| abstract_inverted_index.p<0.001). | 215 |
| abstract_inverted_index.regeneration | 255 |
| abstract_inverted_index.short-answer | 47 |
| abstract_inverted_index.transformers | 11 |
| abstract_inverted_index.certification | 53 |
| abstract_inverted_index.significantly | 268 |
| cited_by_percentile_year.max | 97 |
| cited_by_percentile_year.min | 94 |
| corresponding_author_ids | https://openalex.org/A5024597650 |
| countries_distinct_count | 1 |
| institutions_distinct_count | 5 |
| corresponding_institution_ids | https://openalex.org/I2801824472, https://openalex.org/I4210164802, https://openalex.org/I5023651 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.5699999928474426 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile.value | 0.73329874 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |