Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models Article Swipe
Paul P. Calle
,
A.W. Bates
,
Justin C. Reynolds
,
Yunlong Liu
,
Haoyang Cui
,
S. Ly
,
Chen Wang
,
Qinghao Zhang
,
Alberto J. de Armendi
,
Shashank S. Shettar
,
Kar Ming Fung
,
Qinggong Tang
,
Chongle Pan
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2503.08589
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2503.08589
Background and Objectives: The variability and biases in the real-world performance benchmarking of deep learning models for medical imaging compromise their trustworthiness for real-world deployment. The common approach of holding out a single fixed test set fails to quantify the variance in the estimation of test performance metrics. This study introduces NACHOS (Nested and Automated Cross-validation and Hyperparameter Optimization using Supercomputing) to reduce and quantify the variance of test performance metrics of deep learning models. Methods: NACHOS integrates Nested Cross-Validation (NCV) and Automated Hyperparameter Optimization (AHPO) within a parallelized high-performance computing (HPC) framework. NACHOS was demonstrated on a chest X-ray repository and an Optical Coherence Tomography (OCT) dataset under multiple data partitioning schemes. Beyond performance estimation, DACHOS (Deployment with Automated Cross-validation and Hyperparameter Optimization using Supercomputing) is introduced to leverage AHPO and cross-validation to build the final model on the full dataset, improving expected deployment performance. Results: The findings underscore the importance of NCV in quantifying and reducing estimation variance, AHPO in optimizing hyperparameters consistently across test folds, and HPC in ensuring computational feasibility. Conclusions: By integrating these methodologies, NACHOS and DACHOS provide a scalable, reproducible, and trustworthy framework for DL model evaluation and deployment in medical imaging. To maximize public availability, the full open-source codebase is provided at https://github.com/thepanlab/NACHOS
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- https://doi.org/10.48550/arxiv.2503.08589
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4414578491
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4414578491Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2503.08589Digital Object Identifier
- Title
-
Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning modelsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-03-11Full publication date if available
- Authors
-
Paul P. Calle, A.W. Bates, Justin C. Reynolds, Yunlong Liu, Haoyang Cui, S. Ly, Chen Wang, Qinghao Zhang, Alberto J. de Armendi, Shashank S. Shettar, Kar Ming Fung, Qinggong Tang, Chongle PanList of authors in order
- Landing page
-
https://doi.org/10.48550/arxiv.2503.08589Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://doi.org/10.48550/arxiv.2503.08589Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4414578491 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2503.08589 |
| ids.doi | https://doi.org/10.48550/arxiv.2503.08589 |
| ids.openalex | https://openalex.org/W4414578491 |
| fwci | |
| type | preprint |
| title | Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11512 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.7511000037193298 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Anomaly Detection Techniques and Applications |
| topics[1].id | https://openalex.org/T12535 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.7078999876976013 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Machine Learning and Data Classification |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | doi:10.48550/arxiv.2503.08589 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | |
| locations[0].version | |
| locations[0].raw_type | article |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://doi.org/10.48550/arxiv.2503.08589 |
| indexed_in | datacite |
| authorships[0].author.id | https://openalex.org/A5056674230 |
| authorships[0].author.orcid | https://orcid.org/0009-0000-1849-4481 |
| authorships[0].author.display_name | Paul P. Calle |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Calle, Paul |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5076591143 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | A.W. Bates |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Bates, Averi |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5073689896 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Justin C. Reynolds |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Reynolds, Justin C. |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5051623896 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-8448-7845 |
| authorships[3].author.display_name | Yunlong Liu |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Liu, Yunlong |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5036664597 |
| authorships[4].author.orcid | https://orcid.org/0000-0003-2551-0624 |
| authorships[4].author.display_name | Haoyang Cui |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Cui, Haoyang |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5108146294 |
| authorships[5].author.orcid | |
| authorships[5].author.display_name | S. Ly |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Ly, Sinaro |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5100337697 |
| authorships[6].author.orcid | https://orcid.org/0000-0003-4645-3227 |
| authorships[6].author.display_name | Chen Wang |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Wang, Chen |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5003052424 |
| authorships[7].author.orcid | https://orcid.org/0000-0001-8175-4266 |
| authorships[7].author.display_name | Qinghao Zhang |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Zhang, Qinghao |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5053222440 |
| authorships[8].author.orcid | https://orcid.org/0000-0003-0685-2821 |
| authorships[8].author.display_name | Alberto J. de Armendi |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | de Armendi, Alberto J. |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5023963607 |
| authorships[9].author.orcid | https://orcid.org/0000-0002-3484-205X |
| authorships[9].author.display_name | Shashank S. Shettar |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Shettar, Shashank S. |
| authorships[9].is_corresponding | False |
| authorships[10].author.id | https://openalex.org/A5114029684 |
| authorships[10].author.orcid | |
| authorships[10].author.display_name | Kar Ming Fung |
| authorships[10].author_position | middle |
| authorships[10].raw_author_name | Fung, Kar Ming |
| authorships[10].is_corresponding | False |
| authorships[11].author.id | https://openalex.org/A5077567483 |
| authorships[11].author.orcid | https://orcid.org/0000-0001-9499-5384 |
| authorships[11].author.display_name | Qinggong Tang |
| authorships[11].author_position | middle |
| authorships[11].raw_author_name | Tang, Qinggong |
| authorships[11].is_corresponding | False |
| authorships[12].author.id | https://openalex.org/A5035236332 |
| authorships[12].author.orcid | https://orcid.org/0000-0003-2860-0334 |
| authorships[12].author.display_name | Chongle Pan |
| authorships[12].author_position | last |
| authorships[12].raw_author_name | Pan, Chongle |
| authorships[12].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://doi.org/10.48550/arxiv.2503.08589 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Integration of nested cross-validation, automated hyperparameter optimization, high-performance computing to reduce and quantify the variance of test performance estimation of deep learning models |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-23T05:10:03.516525 |
| primary_topic.id | https://openalex.org/T11512 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.7511000037193298 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Anomaly Detection Techniques and Applications |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.48550/arxiv.2503.08589 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | |
| best_oa_location.version | |
| best_oa_location.raw_type | article |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.48550/arxiv.2503.08589 |
| primary_location.id | doi:10.48550/arxiv.2503.08589 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | |
| primary_location.version | |
| primary_location.raw_type | article |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://doi.org/10.48550/arxiv.2503.08589 |
| publication_date | 2025-03-11 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 31, 87, 97, 183 |
| abstract_inverted_index.By | 175 |
| abstract_inverted_index.DL | 190 |
| abstract_inverted_index.To | 198 |
| abstract_inverted_index.an | 102 |
| abstract_inverted_index.at | 208 |
| abstract_inverted_index.in | 7, 41, 154, 161, 170, 195 |
| abstract_inverted_index.is | 126, 206 |
| abstract_inverted_index.of | 12, 28, 44, 67, 71, 152 |
| abstract_inverted_index.on | 96, 138 |
| abstract_inverted_index.to | 37, 61, 128, 133 |
| abstract_inverted_index.HPC | 169 |
| abstract_inverted_index.NCV | 153 |
| abstract_inverted_index.The | 3, 25, 147 |
| abstract_inverted_index.and | 1, 5, 53, 56, 63, 81, 101, 121, 131, 156, 168, 180, 186, 193 |
| abstract_inverted_index.for | 16, 22, 189 |
| abstract_inverted_index.out | 30 |
| abstract_inverted_index.set | 35 |
| abstract_inverted_index.the | 8, 39, 42, 65, 135, 139, 150, 202 |
| abstract_inverted_index.was | 94 |
| abstract_inverted_index.AHPO | 130, 160 |
| abstract_inverted_index.This | 48 |
| abstract_inverted_index.data | 110 |
| abstract_inverted_index.deep | 13, 72 |
| abstract_inverted_index.full | 140, 203 |
| abstract_inverted_index.test | 34, 45, 68, 166 |
| abstract_inverted_index.with | 118 |
| abstract_inverted_index.(HPC) | 91 |
| abstract_inverted_index.(NCV) | 80 |
| abstract_inverted_index.(OCT) | 106 |
| abstract_inverted_index.X-ray | 99 |
| abstract_inverted_index.build | 134 |
| abstract_inverted_index.chest | 98 |
| abstract_inverted_index.fails | 36 |
| abstract_inverted_index.final | 136 |
| abstract_inverted_index.fixed | 33 |
| abstract_inverted_index.model | 137, 191 |
| abstract_inverted_index.study | 49 |
| abstract_inverted_index.their | 20 |
| abstract_inverted_index.these | 177 |
| abstract_inverted_index.under | 108 |
| abstract_inverted_index.using | 59, 124 |
| abstract_inverted_index.(AHPO) | 85 |
| abstract_inverted_index.Beyond | 113 |
| abstract_inverted_index.DACHOS | 116, 181 |
| abstract_inverted_index.NACHOS | 51, 76, 93, 179 |
| abstract_inverted_index.Nested | 78 |
| abstract_inverted_index.across | 165 |
| abstract_inverted_index.biases | 6 |
| abstract_inverted_index.common | 26 |
| abstract_inverted_index.folds, | 167 |
| abstract_inverted_index.models | 15 |
| abstract_inverted_index.public | 200 |
| abstract_inverted_index.reduce | 62 |
| abstract_inverted_index.single | 32 |
| abstract_inverted_index.within | 86 |
| abstract_inverted_index.(Nested | 52 |
| abstract_inverted_index.Optical | 103 |
| abstract_inverted_index.dataset | 107 |
| abstract_inverted_index.holding | 29 |
| abstract_inverted_index.imaging | 18 |
| abstract_inverted_index.medical | 17, 196 |
| abstract_inverted_index.metrics | 70 |
| abstract_inverted_index.models. | 74 |
| abstract_inverted_index.provide | 182 |
| abstract_inverted_index.Methods: | 75 |
| abstract_inverted_index.Results: | 146 |
| abstract_inverted_index.approach | 27 |
| abstract_inverted_index.codebase | 205 |
| abstract_inverted_index.dataset, | 141 |
| abstract_inverted_index.ensuring | 171 |
| abstract_inverted_index.expected | 143 |
| abstract_inverted_index.findings | 148 |
| abstract_inverted_index.imaging. | 197 |
| abstract_inverted_index.learning | 14, 73 |
| abstract_inverted_index.leverage | 129 |
| abstract_inverted_index.maximize | 199 |
| abstract_inverted_index.metrics. | 47 |
| abstract_inverted_index.multiple | 109 |
| abstract_inverted_index.provided | 207 |
| abstract_inverted_index.quantify | 38, 64 |
| abstract_inverted_index.reducing | 157 |
| abstract_inverted_index.schemes. | 112 |
| abstract_inverted_index.variance | 40, 66 |
| abstract_inverted_index.Automated | 54, 82, 119 |
| abstract_inverted_index.Coherence | 104 |
| abstract_inverted_index.computing | 90 |
| abstract_inverted_index.framework | 188 |
| abstract_inverted_index.improving | 142 |
| abstract_inverted_index.scalable, | 184 |
| abstract_inverted_index.variance, | 159 |
| abstract_inverted_index.Background | 0 |
| abstract_inverted_index.Tomography | 105 |
| abstract_inverted_index.compromise | 19 |
| abstract_inverted_index.deployment | 144, 194 |
| abstract_inverted_index.estimation | 43, 158 |
| abstract_inverted_index.evaluation | 192 |
| abstract_inverted_index.framework. | 92 |
| abstract_inverted_index.importance | 151 |
| abstract_inverted_index.integrates | 77 |
| abstract_inverted_index.introduced | 127 |
| abstract_inverted_index.introduces | 50 |
| abstract_inverted_index.optimizing | 162 |
| abstract_inverted_index.real-world | 9, 23 |
| abstract_inverted_index.repository | 100 |
| abstract_inverted_index.underscore | 149 |
| abstract_inverted_index.(Deployment | 117 |
| abstract_inverted_index.Objectives: | 2 |
| abstract_inverted_index.deployment. | 24 |
| abstract_inverted_index.estimation, | 115 |
| abstract_inverted_index.integrating | 176 |
| abstract_inverted_index.open-source | 204 |
| abstract_inverted_index.performance | 10, 46, 69, 114 |
| abstract_inverted_index.quantifying | 155 |
| abstract_inverted_index.trustworthy | 187 |
| abstract_inverted_index.variability | 4 |
| abstract_inverted_index.Conclusions: | 174 |
| abstract_inverted_index.Optimization | 58, 84, 123 |
| abstract_inverted_index.benchmarking | 11 |
| abstract_inverted_index.consistently | 164 |
| abstract_inverted_index.demonstrated | 95 |
| abstract_inverted_index.feasibility. | 173 |
| abstract_inverted_index.parallelized | 88 |
| abstract_inverted_index.partitioning | 111 |
| abstract_inverted_index.performance. | 145 |
| abstract_inverted_index.availability, | 201 |
| abstract_inverted_index.computational | 172 |
| abstract_inverted_index.reproducible, | 185 |
| abstract_inverted_index.Hyperparameter | 57, 83, 122 |
| abstract_inverted_index.methodologies, | 178 |
| abstract_inverted_index.Supercomputing) | 60, 125 |
| abstract_inverted_index.hyperparameters | 163 |
| abstract_inverted_index.trustworthiness | 21 |
| abstract_inverted_index.Cross-Validation | 79 |
| abstract_inverted_index.Cross-validation | 55, 120 |
| abstract_inverted_index.cross-validation | 132 |
| abstract_inverted_index.high-performance | 89 |
| abstract_inverted_index.https://github.com/thepanlab/NACHOS | 209 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 13 |
| citation_normalized_percentile |