Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural Networks Article Swipe
Sinjini Banerjee
,
Reilly Cannon
,
Tim Marrinan
,
Tony Chiang
,
Anand D. Sarwate
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2310.00541
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2310.00541
Training a deep neural network (DNN) often involves stochastic optimization, which means each run will produce a different model. Several works suggest this variability is negligible when models have the same performance, which in the case of classification is test accuracy. However, models with similar test accuracy may not be computing the same function. We propose a new measure of closeness between classification models based on the output of the network before thresholding. Our measure is based on a robust hypothesis-testing framework and can be adapted to other quantities derived from trained models.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2310.00541
- https://arxiv.org/pdf/2310.00541
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4387323596
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4387323596Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2310.00541Digital Object Identifier
- Title
-
Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural NetworksWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-10-01Full publication date if available
- Authors
-
Sinjini Banerjee, Reilly Cannon, Tim Marrinan, Tony Chiang, Anand D. SarwateList of authors in order
- Landing page
-
https://arxiv.org/abs/2310.00541Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2310.00541Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2310.00541Direct OA link when available
- Concepts
-
Closeness, Computer science, Measure (data warehouse), Artificial neural network, Nonparametric statistics, Machine learning, Artificial intelligence, Thresholding, Statistical hypothesis testing, Deep neural networks, Data mining, Econometrics, Mathematics, Statistics, Mathematical analysis, Image (mathematics)Top concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4387323596 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2310.00541 |
| ids.doi | https://doi.org/10.48550/arxiv.2310.00541 |
| ids.openalex | https://openalex.org/W4387323596 |
| fwci | |
| type | preprint |
| title | Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural Networks |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12535 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9962000250816345 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Machine Learning and Data Classification |
| topics[1].id | https://openalex.org/T12814 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.995199978351593 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Gaussian Processes and Bayesian Inference |
| topics[2].id | https://openalex.org/T12072 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9932000041007996 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Machine Learning and Algorithms |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C2779545769 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7822823524475098 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q5135364 |
| concepts[0].display_name | Closeness |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.6876369714736938 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C2780009758 |
| concepts[2].level | 2 |
| concepts[2].score | 0.6787822842597961 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q6804172 |
| concepts[2].display_name | Measure (data warehouse) |
| concepts[3].id | https://openalex.org/C50644808 |
| concepts[3].level | 2 |
| concepts[3].score | 0.6624850034713745 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q192776 |
| concepts[3].display_name | Artificial neural network |
| concepts[4].id | https://openalex.org/C102366305 |
| concepts[4].level | 2 |
| concepts[4].score | 0.6517010927200317 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q1097688 |
| concepts[4].display_name | Nonparametric statistics |
| concepts[5].id | https://openalex.org/C119857082 |
| concepts[5].level | 1 |
| concepts[5].score | 0.5995892882347107 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[5].display_name | Machine learning |
| concepts[6].id | https://openalex.org/C154945302 |
| concepts[6].level | 1 |
| concepts[6].score | 0.5982033014297485 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[6].display_name | Artificial intelligence |
| concepts[7].id | https://openalex.org/C191178318 |
| concepts[7].level | 3 |
| concepts[7].score | 0.4917495548725128 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q2256906 |
| concepts[7].display_name | Thresholding |
| concepts[8].id | https://openalex.org/C87007009 |
| concepts[8].level | 2 |
| concepts[8].score | 0.4488334357738495 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q210832 |
| concepts[8].display_name | Statistical hypothesis testing |
| concepts[9].id | https://openalex.org/C2984842247 |
| concepts[9].level | 3 |
| concepts[9].score | 0.41111549735069275 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q197536 |
| concepts[9].display_name | Deep neural networks |
| concepts[10].id | https://openalex.org/C124101348 |
| concepts[10].level | 1 |
| concepts[10].score | 0.3236420452594757 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q172491 |
| concepts[10].display_name | Data mining |
| concepts[11].id | https://openalex.org/C149782125 |
| concepts[11].level | 1 |
| concepts[11].score | 0.23677173256874084 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q160039 |
| concepts[11].display_name | Econometrics |
| concepts[12].id | https://openalex.org/C33923547 |
| concepts[12].level | 0 |
| concepts[12].score | 0.17691084742546082 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[12].display_name | Mathematics |
| concepts[13].id | https://openalex.org/C105795698 |
| concepts[13].level | 1 |
| concepts[13].score | 0.1654817759990692 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q12483 |
| concepts[13].display_name | Statistics |
| concepts[14].id | https://openalex.org/C134306372 |
| concepts[14].level | 1 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q7754 |
| concepts[14].display_name | Mathematical analysis |
| concepts[15].id | https://openalex.org/C115961682 |
| concepts[15].level | 2 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q860623 |
| concepts[15].display_name | Image (mathematics) |
| keywords[0].id | https://openalex.org/keywords/closeness |
| keywords[0].score | 0.7822823524475098 |
| keywords[0].display_name | Closeness |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.6876369714736938 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/measure |
| keywords[2].score | 0.6787822842597961 |
| keywords[2].display_name | Measure (data warehouse) |
| keywords[3].id | https://openalex.org/keywords/artificial-neural-network |
| keywords[3].score | 0.6624850034713745 |
| keywords[3].display_name | Artificial neural network |
| keywords[4].id | https://openalex.org/keywords/nonparametric-statistics |
| keywords[4].score | 0.6517010927200317 |
| keywords[4].display_name | Nonparametric statistics |
| keywords[5].id | https://openalex.org/keywords/machine-learning |
| keywords[5].score | 0.5995892882347107 |
| keywords[5].display_name | Machine learning |
| keywords[6].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[6].score | 0.5982033014297485 |
| keywords[6].display_name | Artificial intelligence |
| keywords[7].id | https://openalex.org/keywords/thresholding |
| keywords[7].score | 0.4917495548725128 |
| keywords[7].display_name | Thresholding |
| keywords[8].id | https://openalex.org/keywords/statistical-hypothesis-testing |
| keywords[8].score | 0.4488334357738495 |
| keywords[8].display_name | Statistical hypothesis testing |
| keywords[9].id | https://openalex.org/keywords/deep-neural-networks |
| keywords[9].score | 0.41111549735069275 |
| keywords[9].display_name | Deep neural networks |
| keywords[10].id | https://openalex.org/keywords/data-mining |
| keywords[10].score | 0.3236420452594757 |
| keywords[10].display_name | Data mining |
| keywords[11].id | https://openalex.org/keywords/econometrics |
| keywords[11].score | 0.23677173256874084 |
| keywords[11].display_name | Econometrics |
| keywords[12].id | https://openalex.org/keywords/mathematics |
| keywords[12].score | 0.17691084742546082 |
| keywords[12].display_name | Mathematics |
| keywords[13].id | https://openalex.org/keywords/statistics |
| keywords[13].score | 0.1654817759990692 |
| keywords[13].display_name | Statistics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2310.00541 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2310.00541 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2310.00541 |
| locations[1].id | doi:10.48550/arxiv.2310.00541 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2310.00541 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5049760570 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-1028-4863 |
| authorships[0].author.display_name | Sinjini Banerjee |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Banerjee, Sinjini |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5104327293 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Reilly Cannon |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Cannon, Reilly |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5024418285 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-5240-3053 |
| authorships[2].author.display_name | Tim Marrinan |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Marrinan, Tim |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5088679899 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-9817-2526 |
| authorships[3].author.display_name | Tony Chiang |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Chiang, Tony |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5087162046 |
| authorships[4].author.orcid | https://orcid.org/0000-0001-6123-5282 |
| authorships[4].author.display_name | Anand D. Sarwate |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Sarwate, Anand D. |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2310.00541 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Robust Nonparametric Hypothesis Testing to Understand Variability in Training Neural Networks |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T12535 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9962000250816345 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Machine Learning and Data Classification |
| related_works | https://openalex.org/W2156910174, https://openalex.org/W1995054232, https://openalex.org/W1997747147, https://openalex.org/W4296826658, https://openalex.org/W4251996830, https://openalex.org/W2924932104, https://openalex.org/W2135894974, https://openalex.org/W4297915994, https://openalex.org/W2985158452, https://openalex.org/W2131297085 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2310.00541 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2310.00541 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2310.00541 |
| primary_location.id | pmh:oai:arXiv.org:2310.00541 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2310.00541 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2310.00541 |
| publication_date | 2023-10-01 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 1, 16, 56, 78 |
| abstract_inverted_index.We | 54 |
| abstract_inverted_index.be | 49, 84 |
| abstract_inverted_index.in | 33 |
| abstract_inverted_index.is | 24, 38, 75 |
| abstract_inverted_index.of | 36, 59, 68 |
| abstract_inverted_index.on | 65, 77 |
| abstract_inverted_index.to | 86 |
| abstract_inverted_index.Our | 73 |
| abstract_inverted_index.and | 82 |
| abstract_inverted_index.can | 83 |
| abstract_inverted_index.may | 47 |
| abstract_inverted_index.new | 57 |
| abstract_inverted_index.not | 48 |
| abstract_inverted_index.run | 13 |
| abstract_inverted_index.the | 29, 34, 51, 66, 69 |
| abstract_inverted_index.case | 35 |
| abstract_inverted_index.deep | 2 |
| abstract_inverted_index.each | 12 |
| abstract_inverted_index.from | 90 |
| abstract_inverted_index.have | 28 |
| abstract_inverted_index.same | 30, 52 |
| abstract_inverted_index.test | 39, 45 |
| abstract_inverted_index.this | 22 |
| abstract_inverted_index.when | 26 |
| abstract_inverted_index.will | 14 |
| abstract_inverted_index.with | 43 |
| abstract_inverted_index.(DNN) | 5 |
| abstract_inverted_index.based | 64, 76 |
| abstract_inverted_index.means | 11 |
| abstract_inverted_index.often | 6 |
| abstract_inverted_index.other | 87 |
| abstract_inverted_index.which | 10, 32 |
| abstract_inverted_index.works | 20 |
| abstract_inverted_index.before | 71 |
| abstract_inverted_index.model. | 18 |
| abstract_inverted_index.models | 27, 42, 63 |
| abstract_inverted_index.neural | 3 |
| abstract_inverted_index.output | 67 |
| abstract_inverted_index.robust | 79 |
| abstract_inverted_index.Several | 19 |
| abstract_inverted_index.adapted | 85 |
| abstract_inverted_index.between | 61 |
| abstract_inverted_index.derived | 89 |
| abstract_inverted_index.measure | 58, 74 |
| abstract_inverted_index.models. | 92 |
| abstract_inverted_index.network | 4, 70 |
| abstract_inverted_index.produce | 15 |
| abstract_inverted_index.propose | 55 |
| abstract_inverted_index.similar | 44 |
| abstract_inverted_index.suggest | 21 |
| abstract_inverted_index.trained | 91 |
| abstract_inverted_index.However, | 41 |
| abstract_inverted_index.Training | 0 |
| abstract_inverted_index.accuracy | 46 |
| abstract_inverted_index.involves | 7 |
| abstract_inverted_index.accuracy. | 40 |
| abstract_inverted_index.closeness | 60 |
| abstract_inverted_index.computing | 50 |
| abstract_inverted_index.different | 17 |
| abstract_inverted_index.framework | 81 |
| abstract_inverted_index.function. | 53 |
| abstract_inverted_index.negligible | 25 |
| abstract_inverted_index.quantities | 88 |
| abstract_inverted_index.stochastic | 8 |
| abstract_inverted_index.variability | 23 |
| abstract_inverted_index.performance, | 31 |
| abstract_inverted_index.optimization, | 9 |
| abstract_inverted_index.thresholding. | 72 |
| abstract_inverted_index.classification | 37, 62 |
| abstract_inverted_index.hypothesis-testing | 80 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |