Decorrelated Batch Normalization Article Swipe
YOU?
·
· 2018
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.1804.08450
Batch Normalization (BN) is capable of accelerating the training of deep models by centering and scaling activations within mini-batches. In this work, we propose Decorrelated Batch Normalization (DBN), which not just centers and scales activations but whitens them. We explore multiple whitening techniques, and find that PCA whitening causes a problem we call stochastic axis swapping, which is detrimental to learning. We show that ZCA whitening does not suffer from this problem, permitting successful learning. DBN retains the desirable qualities of BN and further improves BN's optimization efficiency and generalization ability. We design comprehensive experiments to show that DBN can improve the performance of BN on multilayer perceptrons and convolutional neural networks. Furthermore, we consistently improve the accuracy of residual networks on CIFAR-10, CIFAR-100, and ImageNet.
Related Topics
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/1804.08450
- https://arxiv.org/pdf/1804.08450
- OA Status
- green
- Cited By
- 46
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W2798535712
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W2798535712Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.1804.08450Digital Object Identifier
- Title
-
Decorrelated Batch NormalizationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2018Year of publication
- Publication date
-
2018-04-23Full publication date if available
- Authors
-
Lei Huang, Dawei Yang, Bo Lang, Jia DengList of authors in order
- Landing page
-
https://arxiv.org/abs/1804.08450Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/1804.08450Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/1804.08450Direct OA link when available
- Concepts
-
Normalization (sociology), Computer science, Residual, Artificial intelligence, Convolutional neural network, Deep learning, Scaling, Deep neural networks, Perceptron, Artificial neural network, Generalization, Machine learning, Pattern recognition (psychology), Algorithm, Mathematics, Anthropology, Sociology, Geometry, Mathematical analysisTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
46Total citation count in OpenAlex
- Citations by year (recent)
-
2022: 2, 2021: 14, 2020: 19, 2019: 10, 2018: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W2798535712 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.1804.08450 |
| ids.doi | https://doi.org/10.48550/arxiv.1804.08450 |
| ids.mag | 2798535712 |
| ids.openalex | https://openalex.org/W2798535712 |
| fwci | |
| type | preprint |
| title | Decorrelated Batch Normalization |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10036 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9995999932289124 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1707 |
| topics[0].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[0].display_name | Advanced Neural Network Applications |
| topics[1].id | https://openalex.org/T11307 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9976999759674072 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Domain Adaptation and Few-Shot Learning |
| topics[2].id | https://openalex.org/T12702 |
| topics[2].field.id | https://openalex.org/fields/28 |
| topics[2].field.display_name | Neuroscience |
| topics[2].score | 0.9901000261306763 |
| topics[2].domain.id | https://openalex.org/domains/1 |
| topics[2].domain.display_name | Life Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/2808 |
| topics[2].subfield.display_name | Neurology |
| topics[2].display_name | Brain Tumor Detection and Classification |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C136886441 |
| concepts[0].level | 2 |
| concepts[0].score | 0.9024016857147217 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q926129 |
| concepts[0].display_name | Normalization (sociology) |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.7148475646972656 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C155512373 |
| concepts[2].level | 2 |
| concepts[2].score | 0.6962536573410034 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q287450 |
| concepts[2].display_name | Residual |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.6885254383087158 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C81363708 |
| concepts[4].level | 2 |
| concepts[4].score | 0.6114064455032349 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q17084460 |
| concepts[4].display_name | Convolutional neural network |
| concepts[5].id | https://openalex.org/C108583219 |
| concepts[5].level | 2 |
| concepts[5].score | 0.5477019548416138 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q197536 |
| concepts[5].display_name | Deep learning |
| concepts[6].id | https://openalex.org/C99844830 |
| concepts[6].level | 2 |
| concepts[6].score | 0.497743159532547 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q102441924 |
| concepts[6].display_name | Scaling |
| concepts[7].id | https://openalex.org/C2984842247 |
| concepts[7].level | 3 |
| concepts[7].score | 0.47133827209472656 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q197536 |
| concepts[7].display_name | Deep neural networks |
| concepts[8].id | https://openalex.org/C60908668 |
| concepts[8].level | 3 |
| concepts[8].score | 0.4550451338291168 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q690207 |
| concepts[8].display_name | Perceptron |
| concepts[9].id | https://openalex.org/C50644808 |
| concepts[9].level | 2 |
| concepts[9].score | 0.4274987280368805 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q192776 |
| concepts[9].display_name | Artificial neural network |
| concepts[10].id | https://openalex.org/C177148314 |
| concepts[10].level | 2 |
| concepts[10].score | 0.42125535011291504 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q170084 |
| concepts[10].display_name | Generalization |
| concepts[11].id | https://openalex.org/C119857082 |
| concepts[11].level | 1 |
| concepts[11].score | 0.41830748319625854 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[11].display_name | Machine learning |
| concepts[12].id | https://openalex.org/C153180895 |
| concepts[12].level | 2 |
| concepts[12].score | 0.3897552192211151 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q7148389 |
| concepts[12].display_name | Pattern recognition (psychology) |
| concepts[13].id | https://openalex.org/C11413529 |
| concepts[13].level | 1 |
| concepts[13].score | 0.23031499981880188 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q8366 |
| concepts[13].display_name | Algorithm |
| concepts[14].id | https://openalex.org/C33923547 |
| concepts[14].level | 0 |
| concepts[14].score | 0.13162612915039062 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[14].display_name | Mathematics |
| concepts[15].id | https://openalex.org/C19165224 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q23404 |
| concepts[15].display_name | Anthropology |
| concepts[16].id | https://openalex.org/C144024400 |
| concepts[16].level | 0 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q21201 |
| concepts[16].display_name | Sociology |
| concepts[17].id | https://openalex.org/C2524010 |
| concepts[17].level | 1 |
| concepts[17].score | 0.0 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q8087 |
| concepts[17].display_name | Geometry |
| concepts[18].id | https://openalex.org/C134306372 |
| concepts[18].level | 1 |
| concepts[18].score | 0.0 |
| concepts[18].wikidata | https://www.wikidata.org/wiki/Q7754 |
| concepts[18].display_name | Mathematical analysis |
| keywords[0].id | https://openalex.org/keywords/normalization |
| keywords[0].score | 0.9024016857147217 |
| keywords[0].display_name | Normalization (sociology) |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.7148475646972656 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/residual |
| keywords[2].score | 0.6962536573410034 |
| keywords[2].display_name | Residual |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.6885254383087158 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/convolutional-neural-network |
| keywords[4].score | 0.6114064455032349 |
| keywords[4].display_name | Convolutional neural network |
| keywords[5].id | https://openalex.org/keywords/deep-learning |
| keywords[5].score | 0.5477019548416138 |
| keywords[5].display_name | Deep learning |
| keywords[6].id | https://openalex.org/keywords/scaling |
| keywords[6].score | 0.497743159532547 |
| keywords[6].display_name | Scaling |
| keywords[7].id | https://openalex.org/keywords/deep-neural-networks |
| keywords[7].score | 0.47133827209472656 |
| keywords[7].display_name | Deep neural networks |
| keywords[8].id | https://openalex.org/keywords/perceptron |
| keywords[8].score | 0.4550451338291168 |
| keywords[8].display_name | Perceptron |
| keywords[9].id | https://openalex.org/keywords/artificial-neural-network |
| keywords[9].score | 0.4274987280368805 |
| keywords[9].display_name | Artificial neural network |
| keywords[10].id | https://openalex.org/keywords/generalization |
| keywords[10].score | 0.42125535011291504 |
| keywords[10].display_name | Generalization |
| keywords[11].id | https://openalex.org/keywords/machine-learning |
| keywords[11].score | 0.41830748319625854 |
| keywords[11].display_name | Machine learning |
| keywords[12].id | https://openalex.org/keywords/pattern-recognition |
| keywords[12].score | 0.3897552192211151 |
| keywords[12].display_name | Pattern recognition (psychology) |
| keywords[13].id | https://openalex.org/keywords/algorithm |
| keywords[13].score | 0.23031499981880188 |
| keywords[13].display_name | Algorithm |
| keywords[14].id | https://openalex.org/keywords/mathematics |
| keywords[14].score | 0.13162612915039062 |
| keywords[14].display_name | Mathematics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:1804.08450 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/1804.08450 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/1804.08450 |
| locations[1].id | doi:10.48550/arxiv.1804.08450 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.1804.08450 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101670806 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-0502-168X |
| authorships[0].author.display_name | Lei Huang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Lei Huang |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5079747523 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-7646-3891 |
| authorships[1].author.display_name | Dawei Yang |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Dawei Yang |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5100310123 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Bo Lang |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Bo Lang |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5101542158 |
| authorships[3].author.orcid | https://orcid.org/0000-0001-9594-4554 |
| authorships[3].author.display_name | Jia Deng |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Jia Deng |
| authorships[3].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/1804.08450 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Decorrelated Batch Normalization |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10036 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9995999932289124 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1707 |
| primary_topic.subfield.display_name | Computer Vision and Pattern Recognition |
| primary_topic.display_name | Advanced Neural Network Applications |
| related_works | https://openalex.org/W2591697403, https://openalex.org/W2953716828, https://openalex.org/W2904857019, https://openalex.org/W2944728705, https://openalex.org/W3011538607, https://openalex.org/W2904022177, https://openalex.org/W2359348847, https://openalex.org/W4297807321, https://openalex.org/W4287755480, https://openalex.org/W3113607506 |
| cited_by_count | 46 |
| counts_by_year[0].year | 2022 |
| counts_by_year[0].cited_by_count | 2 |
| counts_by_year[1].year | 2021 |
| counts_by_year[1].cited_by_count | 14 |
| counts_by_year[2].year | 2020 |
| counts_by_year[2].cited_by_count | 19 |
| counts_by_year[3].year | 2019 |
| counts_by_year[3].cited_by_count | 10 |
| counts_by_year[4].year | 2018 |
| counts_by_year[4].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:1804.08450 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/1804.08450 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/1804.08450 |
| primary_location.id | pmh:oai:arXiv.org:1804.08450 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/1804.08450 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/1804.08450 |
| publication_date | 2018-04-23 |
| publication_year | 2018 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 49 |
| abstract_inverted_index.BN | 81, 104 |
| abstract_inverted_index.In | 19 |
| abstract_inverted_index.We | 38, 61, 91 |
| abstract_inverted_index.by | 12 |
| abstract_inverted_index.is | 3, 57 |
| abstract_inverted_index.of | 5, 9, 80, 103, 118 |
| abstract_inverted_index.on | 105, 121 |
| abstract_inverted_index.to | 59, 95 |
| abstract_inverted_index.we | 22, 51, 113 |
| abstract_inverted_index.DBN | 75, 98 |
| abstract_inverted_index.PCA | 46 |
| abstract_inverted_index.ZCA | 64 |
| abstract_inverted_index.and | 14, 32, 43, 82, 88, 108, 124 |
| abstract_inverted_index.but | 35 |
| abstract_inverted_index.can | 99 |
| abstract_inverted_index.not | 29, 67 |
| abstract_inverted_index.the | 7, 77, 101, 116 |
| abstract_inverted_index.(BN) | 2 |
| abstract_inverted_index.BN's | 85 |
| abstract_inverted_index.axis | 54 |
| abstract_inverted_index.call | 52 |
| abstract_inverted_index.deep | 10 |
| abstract_inverted_index.does | 66 |
| abstract_inverted_index.find | 44 |
| abstract_inverted_index.from | 69 |
| abstract_inverted_index.just | 30 |
| abstract_inverted_index.show | 62, 96 |
| abstract_inverted_index.that | 45, 63, 97 |
| abstract_inverted_index.this | 20, 70 |
| abstract_inverted_index.Batch | 0, 25 |
| abstract_inverted_index.them. | 37 |
| abstract_inverted_index.which | 28, 56 |
| abstract_inverted_index.work, | 21 |
| abstract_inverted_index.(DBN), | 27 |
| abstract_inverted_index.causes | 48 |
| abstract_inverted_index.design | 92 |
| abstract_inverted_index.models | 11 |
| abstract_inverted_index.neural | 110 |
| abstract_inverted_index.scales | 33 |
| abstract_inverted_index.suffer | 68 |
| abstract_inverted_index.within | 17 |
| abstract_inverted_index.capable | 4 |
| abstract_inverted_index.centers | 31 |
| abstract_inverted_index.explore | 39 |
| abstract_inverted_index.further | 83 |
| abstract_inverted_index.improve | 100, 115 |
| abstract_inverted_index.problem | 50 |
| abstract_inverted_index.propose | 23 |
| abstract_inverted_index.retains | 76 |
| abstract_inverted_index.scaling | 15 |
| abstract_inverted_index.whitens | 36 |
| abstract_inverted_index.ability. | 90 |
| abstract_inverted_index.accuracy | 117 |
| abstract_inverted_index.improves | 84 |
| abstract_inverted_index.multiple | 40 |
| abstract_inverted_index.networks | 120 |
| abstract_inverted_index.problem, | 71 |
| abstract_inverted_index.residual | 119 |
| abstract_inverted_index.training | 8 |
| abstract_inverted_index.CIFAR-10, | 122 |
| abstract_inverted_index.ImageNet. | 125 |
| abstract_inverted_index.centering | 13 |
| abstract_inverted_index.desirable | 78 |
| abstract_inverted_index.learning. | 60, 74 |
| abstract_inverted_index.networks. | 111 |
| abstract_inverted_index.qualities | 79 |
| abstract_inverted_index.swapping, | 55 |
| abstract_inverted_index.whitening | 41, 47, 65 |
| abstract_inverted_index.CIFAR-100, | 123 |
| abstract_inverted_index.efficiency | 87 |
| abstract_inverted_index.multilayer | 106 |
| abstract_inverted_index.permitting | 72 |
| abstract_inverted_index.stochastic | 53 |
| abstract_inverted_index.successful | 73 |
| abstract_inverted_index.activations | 16, 34 |
| abstract_inverted_index.detrimental | 58 |
| abstract_inverted_index.experiments | 94 |
| abstract_inverted_index.perceptrons | 107 |
| abstract_inverted_index.performance | 102 |
| abstract_inverted_index.techniques, | 42 |
| abstract_inverted_index.Decorrelated | 24 |
| abstract_inverted_index.Furthermore, | 112 |
| abstract_inverted_index.accelerating | 6 |
| abstract_inverted_index.consistently | 114 |
| abstract_inverted_index.optimization | 86 |
| abstract_inverted_index.Normalization | 1, 26 |
| abstract_inverted_index.comprehensive | 93 |
| abstract_inverted_index.convolutional | 109 |
| abstract_inverted_index.mini-batches. | 18 |
| abstract_inverted_index.generalization | 89 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |