Considerations for Quality Control Monitoring of Machine Learning Models in Clinical Practice Article Swipe
Louis Faust
,
Patrick M. Wilson
,
Shusaku Asai
,
Sunyang Fu
,
Hongfang Liu
,
Xiaoyang Ruan
,
Curt Storlie
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.2196/50437
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.2196/50437
Integrating machine learning (ML) models into clinical practice presents a challenge of maintaining their efficacy over time. While existing literature offers valuable strategies for detecting declining model performance, there is a need to document the broader challenges and solutions associated with the real-world development and integration of model monitoring solutions. This work details the development and use of a platform for monitoring the performance of a production-level ML model operating in Mayo Clinic. In this paper, we aimed to provide a series of considerations and guidelines necessary for integrating such a platform into a team’s technical infrastructure and workflow. We have documented our experiences with this integration process, discussed the broader challenges encountered with real-world implementation and maintenance, and included the source code for the platform. Our monitoring platform was built as an R shiny application, developed and implemented over the course of 6 months. The platform has been used and maintained for 2 years and is still in use as of July 2023. The considerations necessary for the implementation of the monitoring platform center around 4 pillars: feasibility (what resources can be used for platform development?); design (through what statistics or models will the model be monitored, and how will these results be efficiently displayed to the end user?); implementation (how will this platform be built, and where will it exist within the IT ecosystem?); and policy (based on monitoring feedback, when and what actions will be taken to fix problems, and how will these problems be translated to clinical staff?). While much of the literature surrounding ML performance monitoring emphasizes methodological approaches for capturing changes in performance, there remains a battery of other challenges and considerations that must be addressed for successful real-world implementation.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.2196/50437
- https://s3.ca-central-1.amazonaws.com/assets.jmir.org/assets/preprints/preprint-50437-accepted.pdf
- OA Status
- gold
- Cited By
- 5
- References
- 26
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4396636615
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4396636615Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.2196/50437Digital Object Identifier
- Title
-
Considerations for Quality Control Monitoring of Machine Learning Models in Clinical PracticeWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-05-04Full publication date if available
- Authors
-
Louis Faust, Patrick M. Wilson, Shusaku Asai, Sunyang Fu, Hongfang Liu, Xiaoyang Ruan, Curt StorlieList of authors in order
- Landing page
-
https://doi.org/10.2196/50437Publisher landing page
- PDF URL
-
https://s3.ca-central-1.amazonaws.com/assets.jmir.org/assets/preprints/preprint-50437-accepted.pdfDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://s3.ca-central-1.amazonaws.com/assets.jmir.org/assets/preprints/preprint-50437-accepted.pdfDirect OA link when available
- Concepts
-
Quality (philosophy), Computer science, Control (management), Medicine, Machine learning, Artificial intelligence, Epistemology, PhilosophyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
5Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 5Per-year citation counts (last 5 years)
- References (count)
-
26Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4396636615 |
|---|---|
| doi | https://doi.org/10.2196/50437 |
| ids.doi | https://doi.org/10.2196/50437 |
| ids.pmid | https://pubmed.ncbi.nlm.nih.gov/38941140 |
| ids.openalex | https://openalex.org/W4396636615 |
| fwci | 3.19389275 |
| type | article |
| title | Considerations for Quality Control Monitoring of Machine Learning Models in Clinical Practice |
| biblio.issue | |
| biblio.volume | 12 |
| biblio.last_page | e50437 |
| biblio.first_page | e50437 |
| topics[0].id | https://openalex.org/T13702 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9814000129699707 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Machine Learning in Healthcare |
| topics[1].id | https://openalex.org/T11235 |
| topics[1].field.id | https://openalex.org/fields/26 |
| topics[1].field.display_name | Mathematics |
| topics[1].score | 0.9750999808311462 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/2613 |
| topics[1].subfield.display_name | Statistics and Probability |
| topics[1].display_name | Statistical Methods in Clinical Trials |
| topics[2].id | https://openalex.org/T11443 |
| topics[2].field.id | https://openalex.org/fields/18 |
| topics[2].field.display_name | Decision Sciences |
| topics[2].score | 0.9653000235557556 |
| topics[2].domain.id | https://openalex.org/domains/2 |
| topics[2].domain.display_name | Social Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1804 |
| topics[2].subfield.display_name | Statistics, Probability and Uncertainty |
| topics[2].display_name | Advanced Statistical Process Monitoring |
| is_xpac | False |
| apc_list.value | 2300 |
| apc_list.currency | USD |
| apc_list.value_usd | 2300 |
| apc_paid.value | 2300 |
| apc_paid.currency | USD |
| apc_paid.value_usd | 2300 |
| concepts[0].id | https://openalex.org/C2779530757 |
| concepts[0].level | 2 |
| concepts[0].score | 0.5746715068817139 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q1207505 |
| concepts[0].display_name | Quality (philosophy) |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.5660897493362427 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C2775924081 |
| concepts[2].level | 2 |
| concepts[2].score | 0.4206981658935547 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q55608371 |
| concepts[2].display_name | Control (management) |
| concepts[3].id | https://openalex.org/C71924100 |
| concepts[3].level | 0 |
| concepts[3].score | 0.32319962978363037 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11190 |
| concepts[3].display_name | Medicine |
| concepts[4].id | https://openalex.org/C119857082 |
| concepts[4].level | 1 |
| concepts[4].score | 0.3219219744205475 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[4].display_name | Machine learning |
| concepts[5].id | https://openalex.org/C154945302 |
| concepts[5].level | 1 |
| concepts[5].score | 0.3115796744823456 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[5].display_name | Artificial intelligence |
| concepts[6].id | https://openalex.org/C111472728 |
| concepts[6].level | 1 |
| concepts[6].score | 0.0 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q9471 |
| concepts[6].display_name | Epistemology |
| concepts[7].id | https://openalex.org/C138885662 |
| concepts[7].level | 0 |
| concepts[7].score | 0.0 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q5891 |
| concepts[7].display_name | Philosophy |
| keywords[0].id | https://openalex.org/keywords/quality |
| keywords[0].score | 0.5746715068817139 |
| keywords[0].display_name | Quality (philosophy) |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.5660897493362427 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/control |
| keywords[2].score | 0.4206981658935547 |
| keywords[2].display_name | Control (management) |
| keywords[3].id | https://openalex.org/keywords/medicine |
| keywords[3].score | 0.32319962978363037 |
| keywords[3].display_name | Medicine |
| keywords[4].id | https://openalex.org/keywords/machine-learning |
| keywords[4].score | 0.3219219744205475 |
| keywords[4].display_name | Machine learning |
| keywords[5].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[5].score | 0.3115796744823456 |
| keywords[5].display_name | Artificial intelligence |
| language | en |
| locations[0].id | doi:10.2196/50437 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S2764650051 |
| locations[0].source.issn | 2291-9694 |
| locations[0].source.type | journal |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | 2291-9694 |
| locations[0].source.is_core | True |
| locations[0].source.is_in_doaj | True |
| locations[0].source.display_name | JMIR Medical Informatics |
| locations[0].source.host_organization | https://openalex.org/P4310320608 |
| locations[0].source.host_organization_name | JMIR Publications |
| locations[0].source.host_organization_lineage | https://openalex.org/P4310320608 |
| locations[0].source.host_organization_lineage_names | JMIR Publications |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://s3.ca-central-1.amazonaws.com/assets.jmir.org/assets/preprints/preprint-50437-accepted.pdf |
| locations[0].version | publishedVersion |
| locations[0].raw_type | journal-article |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | JMIR Medical Informatics |
| locations[0].landing_page_url | https://doi.org/10.2196/50437 |
| locations[1].id | pmid:38941140 |
| locations[1].is_oa | False |
| locations[1].source.id | https://openalex.org/S4306525036 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | False |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | PubMed |
| locations[1].source.host_organization | https://openalex.org/I1299303238 |
| locations[1].source.host_organization_name | National Institutes of Health |
| locations[1].source.host_organization_lineage | https://openalex.org/I1299303238 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | publishedVersion |
| locations[1].raw_type | |
| locations[1].license_id | |
| locations[1].is_accepted | True |
| locations[1].is_published | True |
| locations[1].raw_source_name | JMIR medical informatics |
| locations[1].landing_page_url | https://pubmed.ncbi.nlm.nih.gov/38941140 |
| locations[2].id | pmh:oai:digitalcommons.library.tmc.edu:uthshis_docs-1157 |
| locations[2].is_oa | False |
| locations[2].source.id | https://openalex.org/S4306401994 |
| locations[2].source.issn | |
| locations[2].source.type | repository |
| locations[2].source.is_oa | False |
| locations[2].source.issn_l | |
| locations[2].source.is_core | False |
| locations[2].source.is_in_doaj | False |
| locations[2].source.display_name | DigtalCommons @ Texas Medical Center Library (Texas Medical Center) |
| locations[2].source.host_organization | https://openalex.org/I867280407 |
| locations[2].source.host_organization_name | The University of Texas Southwestern Medical Center |
| locations[2].source.host_organization_lineage | https://openalex.org/I867280407 |
| locations[2].license | |
| locations[2].pdf_url | |
| locations[2].version | submittedVersion |
| locations[2].raw_type | |
| locations[2].license_id | |
| locations[2].is_accepted | False |
| locations[2].is_published | False |
| locations[2].raw_source_name | Faculty, Staff and Student Publications |
| locations[2].landing_page_url | https://digitalcommons.library.tmc.edu/uthshis_docs/162 |
| locations[3].id | pmh:oai:doaj.org/article:dd2317fbbb8942ce9331604efcdb8252 |
| locations[3].is_oa | False |
| locations[3].source.id | https://openalex.org/S4306401280 |
| locations[3].source.issn | |
| locations[3].source.type | repository |
| locations[3].source.is_oa | False |
| locations[3].source.issn_l | |
| locations[3].source.is_core | False |
| locations[3].source.is_in_doaj | False |
| locations[3].source.display_name | DOAJ (DOAJ: Directory of Open Access Journals) |
| locations[3].source.host_organization | |
| locations[3].source.host_organization_name | |
| locations[3].license | |
| locations[3].pdf_url | |
| locations[3].version | submittedVersion |
| locations[3].raw_type | article |
| locations[3].license_id | |
| locations[3].is_accepted | False |
| locations[3].is_published | False |
| locations[3].raw_source_name | JMIR Medical Informatics, Vol 12, p e50437 (2024) |
| locations[3].landing_page_url | https://doaj.org/article/dd2317fbbb8942ce9331604efcdb8252 |
| locations[4].id | pmh:oai:pubmedcentral.nih.gov:11245651 |
| locations[4].is_oa | True |
| locations[4].source.id | https://openalex.org/S2764455111 |
| locations[4].source.issn | |
| locations[4].source.type | repository |
| locations[4].source.is_oa | False |
| locations[4].source.issn_l | |
| locations[4].source.is_core | False |
| locations[4].source.is_in_doaj | False |
| locations[4].source.display_name | PubMed Central |
| locations[4].source.host_organization | https://openalex.org/I1299303238 |
| locations[4].source.host_organization_name | National Institutes of Health |
| locations[4].source.host_organization_lineage | https://openalex.org/I1299303238 |
| locations[4].license | other-oa |
| locations[4].pdf_url | |
| locations[4].version | submittedVersion |
| locations[4].raw_type | Text |
| locations[4].license_id | https://openalex.org/licenses/other-oa |
| locations[4].is_accepted | False |
| locations[4].is_published | False |
| locations[4].raw_source_name | JMIR Med Inform |
| locations[4].landing_page_url | https://www.ncbi.nlm.nih.gov/pmc/articles/11245651 |
| indexed_in | crossref, doaj, pubmed |
| authorships[0].author.id | https://openalex.org/A5076011510 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-9741-7894 |
| authorships[0].author.display_name | Louis Faust |
| authorships[0].countries | US |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I1330342723 |
| authorships[0].affiliations[0].raw_affiliation_string | Mayo Clinic, 200 First St. SW, Rochester, MN 55905, United States; |
| authorships[0].affiliations[1].institution_ids | https://openalex.org/I1330342723 |
| authorships[0].affiliations[1].raw_affiliation_string | Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[0].affiliations[2].institution_ids | https://openalex.org/I1330342723 |
| authorships[0].affiliations[2].raw_affiliation_string | Center for the Science of Health Care Delivery Mayo Clinic Mayo Clinic, 200 First St. SW Rochester US |
| authorships[0].affiliations[3].institution_ids | https://openalex.org/I4210125099 |
| authorships[0].affiliations[3].raw_affiliation_string | Center for the Science of Health Care Delivery Mayo Clinic Rochester US |
| authorships[0].institutions[0].id | https://openalex.org/I1330342723 |
| authorships[0].institutions[0].ror | https://ror.org/02qp3tb03 |
| authorships[0].institutions[0].type | healthcare |
| authorships[0].institutions[0].lineage | https://openalex.org/I1330342723 |
| authorships[0].institutions[0].country_code | US |
| authorships[0].institutions[0].display_name | Mayo Clinic |
| authorships[0].institutions[1].id | https://openalex.org/I4210125099 |
| authorships[0].institutions[1].ror | https://ror.org/03jp40720 |
| authorships[0].institutions[1].type | healthcare |
| authorships[0].institutions[1].lineage | https://openalex.org/I1330342723, https://openalex.org/I4210125099 |
| authorships[0].institutions[1].country_code | US |
| authorships[0].institutions[1].display_name | Mayo Clinic in Arizona |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Louis Faust |
| authorships[0].is_corresponding | True |
| authorships[0].raw_affiliation_strings | Center for the Science of Health Care Delivery Mayo Clinic Mayo Clinic, 200 First St. SW Rochester US, Center for the Science of Health Care Delivery Mayo Clinic Rochester US, Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA, Mayo Clinic, 200 First St. SW, Rochester, MN 55905, United States; |
| authorships[1].author.id | https://openalex.org/A5102970277 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-6604-444X |
| authorships[1].author.display_name | Patrick M. Wilson |
| authorships[1].countries | US |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I1330342723 |
| authorships[1].affiliations[0].raw_affiliation_string | Center for the Science of Health Care Delivery Mayo Clinic Mayo Clinic, 200 First St. SW Rochester US |
| authorships[1].affiliations[1].institution_ids | https://openalex.org/I1330342723 |
| authorships[1].affiliations[1].raw_affiliation_string | Mayo Clinic, 200 First St. SW, Rochester, MN 55905, United States; |
| authorships[1].affiliations[2].institution_ids | https://openalex.org/I4210125099 |
| authorships[1].affiliations[2].raw_affiliation_string | Center for the Science of Health Care Delivery Mayo Clinic Rochester US |
| authorships[1].affiliations[3].institution_ids | https://openalex.org/I1330342723 |
| authorships[1].affiliations[3].raw_affiliation_string | Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[1].institutions[0].id | https://openalex.org/I1330342723 |
| authorships[1].institutions[0].ror | https://ror.org/02qp3tb03 |
| authorships[1].institutions[0].type | healthcare |
| authorships[1].institutions[0].lineage | https://openalex.org/I1330342723 |
| authorships[1].institutions[0].country_code | US |
| authorships[1].institutions[0].display_name | Mayo Clinic |
| authorships[1].institutions[1].id | https://openalex.org/I4210125099 |
| authorships[1].institutions[1].ror | https://ror.org/03jp40720 |
| authorships[1].institutions[1].type | healthcare |
| authorships[1].institutions[1].lineage | https://openalex.org/I1330342723, https://openalex.org/I4210125099 |
| authorships[1].institutions[1].country_code | US |
| authorships[1].institutions[1].display_name | Mayo Clinic in Arizona |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Patrick Wilson |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | Center for the Science of Health Care Delivery Mayo Clinic Mayo Clinic, 200 First St. SW Rochester US, Center for the Science of Health Care Delivery Mayo Clinic Rochester US, Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA, Mayo Clinic, 200 First St. SW, Rochester, MN 55905, United States; |
| authorships[2].author.id | https://openalex.org/A5084348251 |
| authorships[2].author.orcid | https://orcid.org/0009-0007-9230-3103 |
| authorships[2].author.display_name | Shusaku Asai |
| authorships[2].countries | US |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I1330342723 |
| authorships[2].affiliations[0].raw_affiliation_string | Mayo Clinic, 200 First St. SW, Rochester, MN 55905, United States; |
| authorships[2].affiliations[1].institution_ids | https://openalex.org/I1330342723 |
| authorships[2].affiliations[1].raw_affiliation_string | Center for the Science of Health Care Delivery Mayo Clinic Mayo Clinic, 200 First St. SW Rochester US |
| authorships[2].affiliations[2].institution_ids | https://openalex.org/I4210125099 |
| authorships[2].affiliations[2].raw_affiliation_string | Center for the Science of Health Care Delivery Mayo Clinic Rochester US |
| authorships[2].affiliations[3].institution_ids | https://openalex.org/I1330342723 |
| authorships[2].affiliations[3].raw_affiliation_string | Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[2].institutions[0].id | https://openalex.org/I1330342723 |
| authorships[2].institutions[0].ror | https://ror.org/02qp3tb03 |
| authorships[2].institutions[0].type | healthcare |
| authorships[2].institutions[0].lineage | https://openalex.org/I1330342723 |
| authorships[2].institutions[0].country_code | US |
| authorships[2].institutions[0].display_name | Mayo Clinic |
| authorships[2].institutions[1].id | https://openalex.org/I4210125099 |
| authorships[2].institutions[1].ror | https://ror.org/03jp40720 |
| authorships[2].institutions[1].type | healthcare |
| authorships[2].institutions[1].lineage | https://openalex.org/I1330342723, https://openalex.org/I4210125099 |
| authorships[2].institutions[1].country_code | US |
| authorships[2].institutions[1].display_name | Mayo Clinic in Arizona |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Shusaku Asai |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Center for the Science of Health Care Delivery Mayo Clinic Mayo Clinic, 200 First St. SW Rochester US, Center for the Science of Health Care Delivery Mayo Clinic Rochester US, Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA, Mayo Clinic, 200 First St. SW, Rochester, MN 55905, United States; |
| authorships[3].author.id | https://openalex.org/A5032226587 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-1691-5179 |
| authorships[3].author.display_name | Sunyang Fu |
| authorships[3].countries | US |
| authorships[3].affiliations[0].raw_affiliation_string | Department of Artificial Intelligence and Informatics Mayo Clinic Rochester US |
| authorships[3].affiliations[1].institution_ids | https://openalex.org/I4210146710 |
| authorships[3].affiliations[1].raw_affiliation_string | Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[3].institutions[0].id | https://openalex.org/I4210146710 |
| authorships[3].institutions[0].ror | https://ror.org/03zzw1w08 |
| authorships[3].institutions[0].type | healthcare |
| authorships[3].institutions[0].lineage | https://openalex.org/I1330342723, https://openalex.org/I4210146710 |
| authorships[3].institutions[0].country_code | US |
| authorships[3].institutions[0].display_name | Mayo Clinic in Florida |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Sunyang Fu |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | Department of Artificial Intelligence and Informatics Mayo Clinic Rochester US, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[4].author.id | https://openalex.org/A5101585391 |
| authorships[4].author.orcid | https://orcid.org/0000-0003-2570-3741 |
| authorships[4].author.display_name | Hongfang Liu |
| authorships[4].countries | US |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I4210146710 |
| authorships[4].affiliations[0].raw_affiliation_string | Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[4].affiliations[1].raw_affiliation_string | Department of Artificial Intelligence and Informatics Mayo Clinic Rochester US |
| authorships[4].institutions[0].id | https://openalex.org/I4210146710 |
| authorships[4].institutions[0].ror | https://ror.org/03zzw1w08 |
| authorships[4].institutions[0].type | healthcare |
| authorships[4].institutions[0].lineage | https://openalex.org/I1330342723, https://openalex.org/I4210146710 |
| authorships[4].institutions[0].country_code | US |
| authorships[4].institutions[0].display_name | Mayo Clinic in Florida |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Hongfang Liu |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | Department of Artificial Intelligence and Informatics Mayo Clinic Rochester US, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[5].author.id | https://openalex.org/A5069990109 |
| authorships[5].author.orcid | https://orcid.org/0009-0006-7085-744X |
| authorships[5].author.display_name | Xiaoyang Ruan |
| authorships[5].countries | US |
| authorships[5].affiliations[0].institution_ids | https://openalex.org/I1330342723 |
| authorships[5].affiliations[0].raw_affiliation_string | Mayo Clinic, 200 First St. SW, Rochester, MN 55905, United States; |
| authorships[5].affiliations[1].institution_ids | https://openalex.org/I4210125099 |
| authorships[5].affiliations[1].raw_affiliation_string | Center for the Science of Health Care Delivery Mayo Clinic Rochester US |
| authorships[5].affiliations[2].institution_ids | https://openalex.org/I1330342723 |
| authorships[5].affiliations[2].raw_affiliation_string | Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[5].affiliations[3].institution_ids | https://openalex.org/I1330342723 |
| authorships[5].affiliations[3].raw_affiliation_string | Center for the Science of Health Care Delivery Mayo Clinic Mayo Clinic, 200 First St. SW Rochester US |
| authorships[5].institutions[0].id | https://openalex.org/I1330342723 |
| authorships[5].institutions[0].ror | https://ror.org/02qp3tb03 |
| authorships[5].institutions[0].type | healthcare |
| authorships[5].institutions[0].lineage | https://openalex.org/I1330342723 |
| authorships[5].institutions[0].country_code | US |
| authorships[5].institutions[0].display_name | Mayo Clinic |
| authorships[5].institutions[1].id | https://openalex.org/I4210125099 |
| authorships[5].institutions[1].ror | https://ror.org/03jp40720 |
| authorships[5].institutions[1].type | healthcare |
| authorships[5].institutions[1].lineage | https://openalex.org/I1330342723, https://openalex.org/I4210125099 |
| authorships[5].institutions[1].country_code | US |
| authorships[5].institutions[1].display_name | Mayo Clinic in Arizona |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Xiaoyang Ruan |
| authorships[5].is_corresponding | False |
| authorships[5].raw_affiliation_strings | Center for the Science of Health Care Delivery Mayo Clinic Mayo Clinic, 200 First St. SW Rochester US, Center for the Science of Health Care Delivery Mayo Clinic Rochester US, Center for the Science of Health Care Delivery, Mayo Clinic, Rochester, Minnesota, USA, Mayo Clinic, 200 First St. SW, Rochester, MN 55905, United States; |
| authorships[6].author.id | https://openalex.org/A5113239139 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Curt Storlie |
| authorships[6].countries | US |
| authorships[6].affiliations[0].raw_affiliation_string | Department of Artificial Intelligence and Informatics Mayo Clinic Rochester US |
| authorships[6].affiliations[1].institution_ids | https://openalex.org/I4210146710 |
| authorships[6].affiliations[1].raw_affiliation_string | Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, USA |
| authorships[6].institutions[0].id | https://openalex.org/I4210146710 |
| authorships[6].institutions[0].ror | https://ror.org/03zzw1w08 |
| authorships[6].institutions[0].type | healthcare |
| authorships[6].institutions[0].lineage | https://openalex.org/I1330342723, https://openalex.org/I4210146710 |
| authorships[6].institutions[0].country_code | US |
| authorships[6].institutions[0].display_name | Mayo Clinic in Florida |
| authorships[6].author_position | last |
| authorships[6].raw_author_name | Curt Storlie |
| authorships[6].is_corresponding | False |
| authorships[6].raw_affiliation_strings | Department of Artificial Intelligence and Informatics Mayo Clinic Rochester US, Department of Artificial Intelligence and Informatics, Mayo Clinic, Rochester, Minnesota, USA |
| has_content.pdf | True |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://s3.ca-central-1.amazonaws.com/assets.jmir.org/assets/preprints/preprint-50437-accepted.pdf |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Considerations for Quality Control Monitoring of Machine Learning Models in Clinical Practice |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T13702 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9814000129699707 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Machine Learning in Healthcare |
| related_works | https://openalex.org/W2961085424, https://openalex.org/W4306674287, https://openalex.org/W3046775127, https://openalex.org/W3107602296, https://openalex.org/W4394896187, https://openalex.org/W3170094116, https://openalex.org/W4386462264, https://openalex.org/W4364306694, https://openalex.org/W4312192474, https://openalex.org/W4283697347 |
| cited_by_count | 5 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 5 |
| locations_count | 5 |
| best_oa_location.id | doi:10.2196/50437 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S2764650051 |
| best_oa_location.source.issn | 2291-9694 |
| best_oa_location.source.type | journal |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | 2291-9694 |
| best_oa_location.source.is_core | True |
| best_oa_location.source.is_in_doaj | True |
| best_oa_location.source.display_name | JMIR Medical Informatics |
| best_oa_location.source.host_organization | https://openalex.org/P4310320608 |
| best_oa_location.source.host_organization_name | JMIR Publications |
| best_oa_location.source.host_organization_lineage | https://openalex.org/P4310320608 |
| best_oa_location.source.host_organization_lineage_names | JMIR Publications |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://s3.ca-central-1.amazonaws.com/assets.jmir.org/assets/preprints/preprint-50437-accepted.pdf |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | journal-article |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | JMIR Medical Informatics |
| best_oa_location.landing_page_url | https://doi.org/10.2196/50437 |
| primary_location.id | doi:10.2196/50437 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S2764650051 |
| primary_location.source.issn | 2291-9694 |
| primary_location.source.type | journal |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | 2291-9694 |
| primary_location.source.is_core | True |
| primary_location.source.is_in_doaj | True |
| primary_location.source.display_name | JMIR Medical Informatics |
| primary_location.source.host_organization | https://openalex.org/P4310320608 |
| primary_location.source.host_organization_name | JMIR Publications |
| primary_location.source.host_organization_lineage | https://openalex.org/P4310320608 |
| primary_location.source.host_organization_lineage_names | JMIR Publications |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://s3.ca-central-1.amazonaws.com/assets.jmir.org/assets/preprints/preprint-50437-accepted.pdf |
| primary_location.version | publishedVersion |
| primary_location.raw_type | journal-article |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | JMIR Medical Informatics |
| primary_location.landing_page_url | https://doi.org/10.2196/50437 |
| publication_date | 2024-05-04 |
| publication_year | 2024 |
| referenced_works | https://openalex.org/W3202394389, https://openalex.org/W4286377178, https://openalex.org/W3174786846, https://openalex.org/W3177617320, https://openalex.org/W1485123848, https://openalex.org/W4237486011, https://openalex.org/W4295008384, https://openalex.org/W3133150084, https://openalex.org/W3199013878, https://openalex.org/W4321793084, https://openalex.org/W3026401238, https://openalex.org/W4230184601, https://openalex.org/W4213435515, https://openalex.org/W3183267169, https://openalex.org/W3217489051, https://openalex.org/W3136312839, https://openalex.org/W1580673759, https://openalex.org/W3122748600, https://openalex.org/W2898017895, https://openalex.org/W2099419573, https://openalex.org/W1998188520, https://openalex.org/W1913116935, https://openalex.org/W3031989616, https://openalex.org/W1593532658, https://openalex.org/W3083909202, https://openalex.org/W2287755163 |
| referenced_works_count | 26 |
| abstract_inverted_index.2 | 153 |
| abstract_inverted_index.4 | 176 |
| abstract_inverted_index.6 | 143 |
| abstract_inverted_index.R | 133 |
| abstract_inverted_index.a | 9, 30, 58, 65, 80, 90, 93, 271 |
| abstract_inverted_index.IT | 224 |
| abstract_inverted_index.In | 73 |
| abstract_inverted_index.ML | 67, 258 |
| abstract_inverted_index.We | 99 |
| abstract_inverted_index.an | 132 |
| abstract_inverted_index.as | 131, 160 |
| abstract_inverted_index.be | 182, 196, 203, 215, 237, 247, 280 |
| abstract_inverted_index.in | 70, 158, 267 |
| abstract_inverted_index.is | 29, 156 |
| abstract_inverted_index.it | 220 |
| abstract_inverted_index.of | 11, 46, 57, 64, 82, 142, 161, 170, 254, 273 |
| abstract_inverted_index.on | 229 |
| abstract_inverted_index.or | 191 |
| abstract_inverted_index.to | 32, 78, 206, 239, 249 |
| abstract_inverted_index.we | 76 |
| abstract_inverted_index.Our | 126 |
| abstract_inverted_index.The | 145, 164 |
| abstract_inverted_index.and | 37, 44, 55, 84, 97, 116, 118, 137, 150, 155, 198, 217, 226, 233, 242, 276 |
| abstract_inverted_index.can | 181 |
| abstract_inverted_index.end | 208 |
| abstract_inverted_index.fix | 240 |
| abstract_inverted_index.for | 23, 60, 87, 123, 152, 167, 184, 264, 282 |
| abstract_inverted_index.has | 147 |
| abstract_inverted_index.how | 199, 243 |
| abstract_inverted_index.our | 102 |
| abstract_inverted_index.the | 34, 41, 53, 62, 109, 120, 124, 140, 168, 171, 194, 207, 223, 255 |
| abstract_inverted_index.use | 56, 159 |
| abstract_inverted_index.was | 129 |
| abstract_inverted_index.(ML) | 3 |
| abstract_inverted_index.(how | 211 |
| abstract_inverted_index.July | 162 |
| abstract_inverted_index.Mayo | 71 |
| abstract_inverted_index.This | 50 |
| abstract_inverted_index.been | 148 |
| abstract_inverted_index.code | 122 |
| abstract_inverted_index.have | 100 |
| abstract_inverted_index.into | 5, 92 |
| abstract_inverted_index.much | 253 |
| abstract_inverted_index.must | 279 |
| abstract_inverted_index.need | 31 |
| abstract_inverted_index.over | 15, 139 |
| abstract_inverted_index.such | 89 |
| abstract_inverted_index.that | 278 |
| abstract_inverted_index.this | 74, 105, 213 |
| abstract_inverted_index.used | 149, 183 |
| abstract_inverted_index.what | 189, 234 |
| abstract_inverted_index.when | 232 |
| abstract_inverted_index.will | 193, 200, 212, 219, 236, 244 |
| abstract_inverted_index.with | 40, 104, 113 |
| abstract_inverted_index.work | 51 |
| abstract_inverted_index.(what | 179 |
| abstract_inverted_index.2023. | 163 |
| abstract_inverted_index.While | 17, 252 |
| abstract_inverted_index.aimed | 77 |
| abstract_inverted_index.built | 130 |
| abstract_inverted_index.exist | 221 |
| abstract_inverted_index.model | 26, 47, 68, 195 |
| abstract_inverted_index.other | 274 |
| abstract_inverted_index.shiny | 134 |
| abstract_inverted_index.still | 157 |
| abstract_inverted_index.taken | 238 |
| abstract_inverted_index.their | 13 |
| abstract_inverted_index.there | 28, 269 |
| abstract_inverted_index.these | 201, 245 |
| abstract_inverted_index.time. | 16 |
| abstract_inverted_index.where | 218 |
| abstract_inverted_index.years | 154 |
| abstract_inverted_index.(based | 228 |
| abstract_inverted_index.around | 175 |
| abstract_inverted_index.built, | 216 |
| abstract_inverted_index.center | 174 |
| abstract_inverted_index.course | 141 |
| abstract_inverted_index.design | 187 |
| abstract_inverted_index.models | 4, 192 |
| abstract_inverted_index.offers | 20 |
| abstract_inverted_index.paper, | 75 |
| abstract_inverted_index.policy | 227 |
| abstract_inverted_index.series | 81 |
| abstract_inverted_index.source | 121 |
| abstract_inverted_index.within | 222 |
| abstract_inverted_index.Clinic. | 72 |
| abstract_inverted_index.actions | 235 |
| abstract_inverted_index.battery | 272 |
| abstract_inverted_index.broader | 35, 110 |
| abstract_inverted_index.changes | 266 |
| abstract_inverted_index.details | 52 |
| abstract_inverted_index.machine | 1 |
| abstract_inverted_index.months. | 144 |
| abstract_inverted_index.provide | 79 |
| abstract_inverted_index.remains | 270 |
| abstract_inverted_index.results | 202 |
| abstract_inverted_index.user?); | 209 |
| abstract_inverted_index.(through | 188 |
| abstract_inverted_index.clinical | 6, 250 |
| abstract_inverted_index.document | 33 |
| abstract_inverted_index.efficacy | 14 |
| abstract_inverted_index.existing | 18 |
| abstract_inverted_index.included | 119 |
| abstract_inverted_index.learning | 2 |
| abstract_inverted_index.pillars: | 177 |
| abstract_inverted_index.platform | 59, 91, 128, 146, 173, 185, 214 |
| abstract_inverted_index.practice | 7 |
| abstract_inverted_index.presents | 8 |
| abstract_inverted_index.problems | 246 |
| abstract_inverted_index.process, | 107 |
| abstract_inverted_index.staff?). | 251 |
| abstract_inverted_index.team’s | 94 |
| abstract_inverted_index.valuable | 21 |
| abstract_inverted_index.addressed | 281 |
| abstract_inverted_index.capturing | 265 |
| abstract_inverted_index.challenge | 10 |
| abstract_inverted_index.declining | 25 |
| abstract_inverted_index.detecting | 24 |
| abstract_inverted_index.developed | 136 |
| abstract_inverted_index.discussed | 108 |
| abstract_inverted_index.displayed | 205 |
| abstract_inverted_index.feedback, | 231 |
| abstract_inverted_index.necessary | 86, 166 |
| abstract_inverted_index.operating | 69 |
| abstract_inverted_index.platform. | 125 |
| abstract_inverted_index.problems, | 241 |
| abstract_inverted_index.resources | 180 |
| abstract_inverted_index.solutions | 38 |
| abstract_inverted_index.technical | 95 |
| abstract_inverted_index.workflow. | 98 |
| abstract_inverted_index.approaches | 263 |
| abstract_inverted_index.associated | 39 |
| abstract_inverted_index.challenges | 36, 111, 275 |
| abstract_inverted_index.documented | 101 |
| abstract_inverted_index.emphasizes | 261 |
| abstract_inverted_index.guidelines | 85 |
| abstract_inverted_index.literature | 19, 256 |
| abstract_inverted_index.maintained | 151 |
| abstract_inverted_index.monitored, | 197 |
| abstract_inverted_index.monitoring | 48, 61, 127, 172, 230, 260 |
| abstract_inverted_index.real-world | 42, 114, 284 |
| abstract_inverted_index.solutions. | 49 |
| abstract_inverted_index.statistics | 190 |
| abstract_inverted_index.strategies | 22 |
| abstract_inverted_index.successful | 283 |
| abstract_inverted_index.translated | 248 |
| abstract_inverted_index.Integrating | 0 |
| abstract_inverted_index.development | 43, 54 |
| abstract_inverted_index.efficiently | 204 |
| abstract_inverted_index.encountered | 112 |
| abstract_inverted_index.experiences | 103 |
| abstract_inverted_index.feasibility | 178 |
| abstract_inverted_index.implemented | 138 |
| abstract_inverted_index.integrating | 88 |
| abstract_inverted_index.integration | 45, 106 |
| abstract_inverted_index.maintaining | 12 |
| abstract_inverted_index.performance | 63, 259 |
| abstract_inverted_index.surrounding | 257 |
| abstract_inverted_index.application, | 135 |
| abstract_inverted_index.ecosystem?); | 225 |
| abstract_inverted_index.maintenance, | 117 |
| abstract_inverted_index.performance, | 27, 268 |
| abstract_inverted_index.considerations | 83, 165, 277 |
| abstract_inverted_index.development?); | 186 |
| abstract_inverted_index.implementation | 115, 169, 210 |
| abstract_inverted_index.infrastructure | 96 |
| abstract_inverted_index.methodological | 262 |
| abstract_inverted_index.implementation. | 285 |
| abstract_inverted_index.production-level | 66 |
| cited_by_percentile_year.max | 98 |
| cited_by_percentile_year.min | 97 |
| corresponding_author_ids | https://openalex.org/A5076011510 |
| countries_distinct_count | 1 |
| institutions_distinct_count | 7 |
| corresponding_institution_ids | https://openalex.org/I1330342723, https://openalex.org/I4210125099 |
| citation_normalized_percentile.value | 0.89510437 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | True |