Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models Article Swipe
Batch Normalization is quite effective at accelerating and improving the training of deep models. However, its effectiveness diminishes when the training minibatches are small, or do not consist of independent samples. We hypothesize that this is due to the dependence of model layer inputs on all the examples in the minibatch, and different activations being produced between training and inference. We propose Batch Renormalization, a simple and effective extension to ensure that the training and inference models generate the same outputs that depend on individual examples rather than the entire minibatch. Models trained with Batch Renormalization perform substantially better than batchnorm when training with small or non-i.i.d. minibatches. At the same time, Batch Renormalization retains the benefits of batchnorm such as insensitivity to initialization and training efficiency.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/1702.03275
- https://arxiv.org/pdf/1702.03275
- OA Status
- green
- Cited By
- 243
- References
- 10
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W2588610957
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W2588610957Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.1702.03275Digital Object Identifier
- Title
-
Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized ModelsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2017Year of publication
- Publication date
-
2017-02-10Full publication date if available
- Authors
-
Sergey IoffeList of authors in order
- Landing page
-
https://arxiv.org/abs/1702.03275Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/1702.03275Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/1702.03275Direct OA link when available
- Concepts
-
Initialization, Normalization (sociology), Renormalization, Inference, Computer science, Training (meteorology), Mathematics, Artificial intelligence, Physics, Mathematical physics, Meteorology, Anthropology, Sociology, Programming languageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
243Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 15, 2024: 21, 2023: 37, 2022: 53, 2021: 32Per-year citation counts (last 5 years)
- References (count)
-
10Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W2588610957 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.1702.03275 |
| ids.doi | https://doi.org/10.48550/arxiv.1702.03275 |
| ids.mag | 2588610957 |
| ids.openalex | https://openalex.org/W2588610957 |
| fwci | |
| type | preprint |
| title | Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12535 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9994999766349792 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Machine Learning and Data Classification |
| topics[1].id | https://openalex.org/T10036 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9983000159263611 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1707 |
| topics[1].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[1].display_name | Advanced Neural Network Applications |
| topics[2].id | https://openalex.org/T10775 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9976000189781189 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1707 |
| topics[2].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[2].display_name | Generative Adversarial Networks and Image Synthesis |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C114466953 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8567613363265991 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q6034165 |
| concepts[0].display_name | Initialization |
| concepts[1].id | https://openalex.org/C136886441 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7904412746429443 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q926129 |
| concepts[1].display_name | Normalization (sociology) |
| concepts[2].id | https://openalex.org/C166124518 |
| concepts[2].level | 2 |
| concepts[2].score | 0.7028104066848755 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q1047702 |
| concepts[2].display_name | Renormalization |
| concepts[3].id | https://openalex.org/C2776214188 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5971558094024658 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q408386 |
| concepts[3].display_name | Inference |
| concepts[4].id | https://openalex.org/C41008148 |
| concepts[4].level | 0 |
| concepts[4].score | 0.5234915018081665 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[4].display_name | Computer science |
| concepts[5].id | https://openalex.org/C2777211547 |
| concepts[5].level | 2 |
| concepts[5].score | 0.4678754508495331 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q17141490 |
| concepts[5].display_name | Training (meteorology) |
| concepts[6].id | https://openalex.org/C33923547 |
| concepts[6].level | 0 |
| concepts[6].score | 0.2745354175567627 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[6].display_name | Mathematics |
| concepts[7].id | https://openalex.org/C154945302 |
| concepts[7].level | 1 |
| concepts[7].score | 0.24622365832328796 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[7].display_name | Artificial intelligence |
| concepts[8].id | https://openalex.org/C121332964 |
| concepts[8].level | 0 |
| concepts[8].score | 0.16876167058944702 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[8].display_name | Physics |
| concepts[9].id | https://openalex.org/C37914503 |
| concepts[9].level | 1 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q156495 |
| concepts[9].display_name | Mathematical physics |
| concepts[10].id | https://openalex.org/C153294291 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q25261 |
| concepts[10].display_name | Meteorology |
| concepts[11].id | https://openalex.org/C19165224 |
| concepts[11].level | 1 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q23404 |
| concepts[11].display_name | Anthropology |
| concepts[12].id | https://openalex.org/C144024400 |
| concepts[12].level | 0 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q21201 |
| concepts[12].display_name | Sociology |
| concepts[13].id | https://openalex.org/C199360897 |
| concepts[13].level | 1 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[13].display_name | Programming language |
| keywords[0].id | https://openalex.org/keywords/initialization |
| keywords[0].score | 0.8567613363265991 |
| keywords[0].display_name | Initialization |
| keywords[1].id | https://openalex.org/keywords/normalization |
| keywords[1].score | 0.7904412746429443 |
| keywords[1].display_name | Normalization (sociology) |
| keywords[2].id | https://openalex.org/keywords/renormalization |
| keywords[2].score | 0.7028104066848755 |
| keywords[2].display_name | Renormalization |
| keywords[3].id | https://openalex.org/keywords/inference |
| keywords[3].score | 0.5971558094024658 |
| keywords[3].display_name | Inference |
| keywords[4].id | https://openalex.org/keywords/computer-science |
| keywords[4].score | 0.5234915018081665 |
| keywords[4].display_name | Computer science |
| keywords[5].id | https://openalex.org/keywords/training |
| keywords[5].score | 0.4678754508495331 |
| keywords[5].display_name | Training (meteorology) |
| keywords[6].id | https://openalex.org/keywords/mathematics |
| keywords[6].score | 0.2745354175567627 |
| keywords[6].display_name | Mathematics |
| keywords[7].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[7].score | 0.24622365832328796 |
| keywords[7].display_name | Artificial intelligence |
| keywords[8].id | https://openalex.org/keywords/physics |
| keywords[8].score | 0.16876167058944702 |
| keywords[8].display_name | Physics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:1702.03275 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/1702.03275 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/1702.03275 |
| locations[1].id | doi:10.48550/arxiv.1702.03275 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.1702.03275 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5032800189 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Sergey Ioffe |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Sergey Ioffe |
| authorships[0].is_corresponding | True |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/1702.03275 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2017-02-24T00:00:00 |
| display_name | Batch Renormalization: Towards Reducing Minibatch Dependence in Batch-Normalized Models |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T12535 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9994999766349792 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Machine Learning and Data Classification |
| related_works | https://openalex.org/W3204184292, https://openalex.org/W3176564347, https://openalex.org/W1985458517, https://openalex.org/W2355833770, https://openalex.org/W3031039437, https://openalex.org/W183202219, https://openalex.org/W4226466875, https://openalex.org/W2977257638, https://openalex.org/W4288095186, https://openalex.org/W3210541621 |
| cited_by_count | 243 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 15 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 21 |
| counts_by_year[2].year | 2023 |
| counts_by_year[2].cited_by_count | 37 |
| counts_by_year[3].year | 2022 |
| counts_by_year[3].cited_by_count | 53 |
| counts_by_year[4].year | 2021 |
| counts_by_year[4].cited_by_count | 32 |
| counts_by_year[5].year | 2020 |
| counts_by_year[5].cited_by_count | 33 |
| counts_by_year[6].year | 2019 |
| counts_by_year[6].cited_by_count | 27 |
| counts_by_year[7].year | 2018 |
| counts_by_year[7].cited_by_count | 18 |
| counts_by_year[8].year | 2017 |
| counts_by_year[8].cited_by_count | 7 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:1702.03275 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/1702.03275 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/1702.03275 |
| primary_location.id | pmh:oai:arXiv.org:1702.03275 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/1702.03275 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/1702.03275 |
| publication_date | 2017-02-10 |
| publication_year | 2017 |
| referenced_works | https://openalex.org/W2144935315, https://openalex.org/W2432004435, https://openalex.org/W2194775991, https://openalex.org/W2336650964, https://openalex.org/W2183341477, https://openalex.org/W3099206234, https://openalex.org/W2952020226, https://openalex.org/W2949117887, https://openalex.org/W2292729293, https://openalex.org/W2284050935 |
| referenced_works_count | 10 |
| abstract_inverted_index.a | 64 |
| abstract_inverted_index.At | 108 |
| abstract_inverted_index.We | 31, 60 |
| abstract_inverted_index.as | 120 |
| abstract_inverted_index.at | 5 |
| abstract_inverted_index.do | 25 |
| abstract_inverted_index.in | 48 |
| abstract_inverted_index.is | 2, 35 |
| abstract_inverted_index.of | 11, 28, 40, 117 |
| abstract_inverted_index.on | 44, 83 |
| abstract_inverted_index.or | 24, 105 |
| abstract_inverted_index.to | 37, 69, 122 |
| abstract_inverted_index.all | 45 |
| abstract_inverted_index.and | 7, 51, 58, 66, 74, 124 |
| abstract_inverted_index.are | 22 |
| abstract_inverted_index.due | 36 |
| abstract_inverted_index.its | 15 |
| abstract_inverted_index.not | 26 |
| abstract_inverted_index.the | 9, 19, 38, 46, 49, 72, 78, 88, 109, 115 |
| abstract_inverted_index.deep | 12 |
| abstract_inverted_index.same | 79, 110 |
| abstract_inverted_index.such | 119 |
| abstract_inverted_index.than | 87, 99 |
| abstract_inverted_index.that | 33, 71, 81 |
| abstract_inverted_index.this | 34 |
| abstract_inverted_index.when | 18, 101 |
| abstract_inverted_index.with | 93, 103 |
| abstract_inverted_index.Batch | 0, 62, 94, 112 |
| abstract_inverted_index.being | 54 |
| abstract_inverted_index.layer | 42 |
| abstract_inverted_index.model | 41 |
| abstract_inverted_index.quite | 3 |
| abstract_inverted_index.small | 104 |
| abstract_inverted_index.time, | 111 |
| abstract_inverted_index.Models | 91 |
| abstract_inverted_index.better | 98 |
| abstract_inverted_index.depend | 82 |
| abstract_inverted_index.ensure | 70 |
| abstract_inverted_index.entire | 89 |
| abstract_inverted_index.inputs | 43 |
| abstract_inverted_index.models | 76 |
| abstract_inverted_index.rather | 86 |
| abstract_inverted_index.simple | 65 |
| abstract_inverted_index.small, | 23 |
| abstract_inverted_index.between | 56 |
| abstract_inverted_index.consist | 27 |
| abstract_inverted_index.models. | 13 |
| abstract_inverted_index.outputs | 80 |
| abstract_inverted_index.perform | 96 |
| abstract_inverted_index.propose | 61 |
| abstract_inverted_index.retains | 114 |
| abstract_inverted_index.trained | 92 |
| abstract_inverted_index.However, | 14 |
| abstract_inverted_index.benefits | 116 |
| abstract_inverted_index.examples | 47, 85 |
| abstract_inverted_index.generate | 77 |
| abstract_inverted_index.produced | 55 |
| abstract_inverted_index.samples. | 30 |
| abstract_inverted_index.training | 10, 20, 57, 73, 102, 125 |
| abstract_inverted_index.batchnorm | 100, 118 |
| abstract_inverted_index.different | 52 |
| abstract_inverted_index.effective | 4, 67 |
| abstract_inverted_index.extension | 68 |
| abstract_inverted_index.improving | 8 |
| abstract_inverted_index.inference | 75 |
| abstract_inverted_index.dependence | 39 |
| abstract_inverted_index.diminishes | 17 |
| abstract_inverted_index.individual | 84 |
| abstract_inverted_index.inference. | 59 |
| abstract_inverted_index.minibatch, | 50 |
| abstract_inverted_index.minibatch. | 90 |
| abstract_inverted_index.non-i.i.d. | 106 |
| abstract_inverted_index.activations | 53 |
| abstract_inverted_index.efficiency. | 126 |
| abstract_inverted_index.hypothesize | 32 |
| abstract_inverted_index.independent | 29 |
| abstract_inverted_index.minibatches | 21 |
| abstract_inverted_index.accelerating | 6 |
| abstract_inverted_index.minibatches. | 107 |
| abstract_inverted_index.Normalization | 1 |
| abstract_inverted_index.effectiveness | 16 |
| abstract_inverted_index.insensitivity | 121 |
| abstract_inverted_index.substantially | 97 |
| abstract_inverted_index.initialization | 123 |
| abstract_inverted_index.Renormalization | 95, 113 |
| abstract_inverted_index.Renormalization, | 63 |
| cited_by_percentile_year | |
| corresponding_author_ids | https://openalex.org/A5032800189 |
| countries_distinct_count | 0 |
| institutions_distinct_count | 1 |
| citation_normalized_percentile |