Neural Collapse Inspired Knowledge Distillation Article Swipe
Shuoxi Zhang
,
Z. Song
,
Kun He
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.1609/aaai.v39i21.34412
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.1609/aaai.v39i21.34412
Existing knowledge distillation (KD) methods have demonstrated their ability to achieve student network performance on par with their teachers. However, the knowledge gap between the teacher and student remains significant and may hinder the effectiveness of the distillation process. In this work, we introduce the structure of Neural Collapse (NC) into the KD framework. NC typically occurs in the final phase of training, resulting in a graceful geometric structure where the last-layer features form a simplex equiangular tight frame. We hypothesize that NC can alleviate the knowledge gap in distillation, thereby enhancing student performance. This paper begins with an empirical analysis to bridge the connection between KD and NC. Through this analysis, we establish that transferring the teacher's NC structure to the student benefits the distillation process. Therefore, instead of merely transferring instance-level logits or features, as done by existing distillation methods, we encourage students to learn the teacher's NC structure. We propose the new distillation paradigm termed Neural Collapse-inspired Knowledge Distillation (NCKD). Comprehensive experiments demonstrate that NCKD is simple yet effective, improving the generalization of all distilled student models and achieving state-of-the-art accuracy performance.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.1609/aaai.v39i21.34412
- https://ojs.aaai.org/index.php/AAAI/article/download/34412/36567
- OA Status
- diamond
- References
- 23
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4409362958
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4409362958Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.1609/aaai.v39i21.34412Digital Object Identifier
- Title
-
Neural Collapse Inspired Knowledge DistillationWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-04-11Full publication date if available
- Authors
-
Shuoxi Zhang, Z. Song, Kun HeList of authors in order
- Landing page
-
https://doi.org/10.1609/aaai.v39i21.34412Publisher landing page
- PDF URL
-
https://ojs.aaai.org/index.php/AAAI/article/download/34412/36567Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
diamondOpen access status per OpenAlex
- OA URL
-
https://ojs.aaai.org/index.php/AAAI/article/download/34412/36567Direct OA link when available
- Concepts
-
Artificial neural network, Distillation, Computer science, Artificial intelligence, Biochemical engineering, Chemistry, Engineering, ChromatographyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- References (count)
-
23Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4409362958 |
|---|---|
| doi | https://doi.org/10.1609/aaai.v39i21.34412 |
| ids.doi | https://doi.org/10.1609/aaai.v39i21.34412 |
| ids.openalex | https://openalex.org/W4409362958 |
| fwci | 0.0 |
| type | article |
| title | Neural Collapse Inspired Knowledge Distillation |
| biblio.issue | 21 |
| biblio.volume | 39 |
| biblio.last_page | 22550 |
| biblio.first_page | 22542 |
| topics[0].id | https://openalex.org/T10320 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.42170000076293945 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Neural Networks and Applications |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C50644808 |
| concepts[0].level | 2 |
| concepts[0].score | 0.5413479804992676 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q192776 |
| concepts[0].display_name | Artificial neural network |
| concepts[1].id | https://openalex.org/C204030448 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5102691650390625 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q101017 |
| concepts[1].display_name | Distillation |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.4726475775241852 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.43245261907577515 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C183696295 |
| concepts[4].level | 1 |
| concepts[4].score | 0.3330298364162445 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q2487696 |
| concepts[4].display_name | Biochemical engineering |
| concepts[5].id | https://openalex.org/C185592680 |
| concepts[5].level | 0 |
| concepts[5].score | 0.26000726222991943 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q2329 |
| concepts[5].display_name | Chemistry |
| concepts[6].id | https://openalex.org/C127413603 |
| concepts[6].level | 0 |
| concepts[6].score | 0.22485396265983582 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[6].display_name | Engineering |
| concepts[7].id | https://openalex.org/C43617362 |
| concepts[7].level | 1 |
| concepts[7].score | 0.17087167501449585 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q170050 |
| concepts[7].display_name | Chromatography |
| keywords[0].id | https://openalex.org/keywords/artificial-neural-network |
| keywords[0].score | 0.5413479804992676 |
| keywords[0].display_name | Artificial neural network |
| keywords[1].id | https://openalex.org/keywords/distillation |
| keywords[1].score | 0.5102691650390625 |
| keywords[1].display_name | Distillation |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.4726475775241852 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.43245261907577515 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/biochemical-engineering |
| keywords[4].score | 0.3330298364162445 |
| keywords[4].display_name | Biochemical engineering |
| keywords[5].id | https://openalex.org/keywords/chemistry |
| keywords[5].score | 0.26000726222991943 |
| keywords[5].display_name | Chemistry |
| keywords[6].id | https://openalex.org/keywords/engineering |
| keywords[6].score | 0.22485396265983582 |
| keywords[6].display_name | Engineering |
| keywords[7].id | https://openalex.org/keywords/chromatography |
| keywords[7].score | 0.17087167501449585 |
| keywords[7].display_name | Chromatography |
| language | en |
| locations[0].id | doi:10.1609/aaai.v39i21.34412 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4210191458 |
| locations[0].source.issn | 2159-5399, 2374-3468 |
| locations[0].source.type | conference |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | 2159-5399 |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | Proceedings of the AAAI Conference on Artificial Intelligence |
| locations[0].source.host_organization | https://openalex.org/P4310320058 |
| locations[0].source.host_organization_name | Association for the Advancement of Artificial Intelligence |
| locations[0].source.host_organization_lineage | https://openalex.org/P4310320058 |
| locations[0].source.host_organization_lineage_names | Association for the Advancement of Artificial Intelligence |
| locations[0].license | |
| locations[0].pdf_url | https://ojs.aaai.org/index.php/AAAI/article/download/34412/36567 |
| locations[0].version | publishedVersion |
| locations[0].raw_type | journal-article |
| locations[0].license_id | |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | Proceedings of the AAAI Conference on Artificial Intelligence |
| locations[0].landing_page_url | https://doi.org/10.1609/aaai.v39i21.34412 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5005741692 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Shuoxi Zhang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Shuoxi Zhang |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5107860544 |
| authorships[1].author.orcid | https://orcid.org/0009-0002-8357-2102 |
| authorships[1].author.display_name | Z. Song |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Zijian Song |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5100700363 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-8943-8671 |
| authorships[2].author.display_name | Kun He |
| authorships[2].author_position | last |
| authorships[2].raw_author_name | Kun He |
| authorships[2].is_corresponding | False |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://ojs.aaai.org/index.php/AAAI/article/download/34412/36567 |
| open_access.oa_status | diamond |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Neural Collapse Inspired Knowledge Distillation |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10320 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.42170000076293945 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Neural Networks and Applications |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052 |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.1609/aaai.v39i21.34412 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4210191458 |
| best_oa_location.source.issn | 2159-5399, 2374-3468 |
| best_oa_location.source.type | conference |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | 2159-5399 |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | Proceedings of the AAAI Conference on Artificial Intelligence |
| best_oa_location.source.host_organization | https://openalex.org/P4310320058 |
| best_oa_location.source.host_organization_name | Association for the Advancement of Artificial Intelligence |
| best_oa_location.source.host_organization_lineage | https://openalex.org/P4310320058 |
| best_oa_location.source.host_organization_lineage_names | Association for the Advancement of Artificial Intelligence |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://ojs.aaai.org/index.php/AAAI/article/download/34412/36567 |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | journal-article |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | Proceedings of the AAAI Conference on Artificial Intelligence |
| best_oa_location.landing_page_url | https://doi.org/10.1609/aaai.v39i21.34412 |
| primary_location.id | doi:10.1609/aaai.v39i21.34412 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4210191458 |
| primary_location.source.issn | 2159-5399, 2374-3468 |
| primary_location.source.type | conference |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | 2159-5399 |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | Proceedings of the AAAI Conference on Artificial Intelligence |
| primary_location.source.host_organization | https://openalex.org/P4310320058 |
| primary_location.source.host_organization_name | Association for the Advancement of Artificial Intelligence |
| primary_location.source.host_organization_lineage | https://openalex.org/P4310320058 |
| primary_location.source.host_organization_lineage_names | Association for the Advancement of Artificial Intelligence |
| primary_location.license | |
| primary_location.pdf_url | https://ojs.aaai.org/index.php/AAAI/article/download/34412/36567 |
| primary_location.version | publishedVersion |
| primary_location.raw_type | journal-article |
| primary_location.license_id | |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | Proceedings of the AAAI Conference on Artificial Intelligence |
| primary_location.landing_page_url | https://doi.org/10.1609/aaai.v39i21.34412 |
| publication_date | 2025-04-11 |
| publication_year | 2025 |
| referenced_works | https://openalex.org/W4221152651, https://openalex.org/W6787589371, https://openalex.org/W6794267180, https://openalex.org/W4367060876, https://openalex.org/W6687483927, https://openalex.org/W6743731764, https://openalex.org/W6839053282, https://openalex.org/W6855345637, https://openalex.org/W6604426834, https://openalex.org/W6776700526, https://openalex.org/W2786945063, https://openalex.org/W6730903564, https://openalex.org/W6805780310, https://openalex.org/W3065974826, https://openalex.org/W6761503440, https://openalex.org/W2613718673, https://openalex.org/W4393928674, https://openalex.org/W2948527124, https://openalex.org/W2549139847, https://openalex.org/W3015036596, https://openalex.org/W4391755841, https://openalex.org/W6811639908, https://openalex.org/W4221157420 |
| referenced_works_count | 23 |
| abstract_inverted_index.a | 65, 74 |
| abstract_inverted_index.In | 39 |
| abstract_inverted_index.KD | 52, 106 |
| abstract_inverted_index.NC | 54, 82, 118, 149 |
| abstract_inverted_index.We | 79, 151 |
| abstract_inverted_index.an | 98 |
| abstract_inverted_index.as | 136 |
| abstract_inverted_index.by | 138 |
| abstract_inverted_index.in | 57, 64, 88 |
| abstract_inverted_index.is | 168 |
| abstract_inverted_index.of | 35, 46, 61, 129, 175 |
| abstract_inverted_index.on | 14 |
| abstract_inverted_index.or | 134 |
| abstract_inverted_index.to | 9, 101, 120, 145 |
| abstract_inverted_index.we | 42, 112, 142 |
| abstract_inverted_index.NC. | 108 |
| abstract_inverted_index.all | 176 |
| abstract_inverted_index.and | 26, 30, 107, 180 |
| abstract_inverted_index.can | 83 |
| abstract_inverted_index.gap | 22, 87 |
| abstract_inverted_index.may | 31 |
| abstract_inverted_index.new | 154 |
| abstract_inverted_index.par | 15 |
| abstract_inverted_index.the | 20, 24, 33, 36, 44, 51, 58, 70, 85, 103, 116, 121, 124, 147, 153, 173 |
| abstract_inverted_index.yet | 170 |
| abstract_inverted_index.(KD) | 3 |
| abstract_inverted_index.(NC) | 49 |
| abstract_inverted_index.NCKD | 167 |
| abstract_inverted_index.This | 94 |
| abstract_inverted_index.done | 137 |
| abstract_inverted_index.form | 73 |
| abstract_inverted_index.have | 5 |
| abstract_inverted_index.into | 50 |
| abstract_inverted_index.that | 81, 114, 166 |
| abstract_inverted_index.this | 40, 110 |
| abstract_inverted_index.with | 16, 97 |
| abstract_inverted_index.final | 59 |
| abstract_inverted_index.learn | 146 |
| abstract_inverted_index.paper | 95 |
| abstract_inverted_index.phase | 60 |
| abstract_inverted_index.their | 7, 17 |
| abstract_inverted_index.tight | 77 |
| abstract_inverted_index.where | 69 |
| abstract_inverted_index.work, | 41 |
| abstract_inverted_index.Neural | 47, 158 |
| abstract_inverted_index.begins | 96 |
| abstract_inverted_index.bridge | 102 |
| abstract_inverted_index.frame. | 78 |
| abstract_inverted_index.hinder | 32 |
| abstract_inverted_index.logits | 133 |
| abstract_inverted_index.merely | 130 |
| abstract_inverted_index.models | 179 |
| abstract_inverted_index.occurs | 56 |
| abstract_inverted_index.simple | 169 |
| abstract_inverted_index.termed | 157 |
| abstract_inverted_index.(NCKD). | 162 |
| abstract_inverted_index.Through | 109 |
| abstract_inverted_index.ability | 8 |
| abstract_inverted_index.achieve | 10 |
| abstract_inverted_index.between | 23, 105 |
| abstract_inverted_index.instead | 128 |
| abstract_inverted_index.methods | 4 |
| abstract_inverted_index.network | 12 |
| abstract_inverted_index.propose | 152 |
| abstract_inverted_index.remains | 28 |
| abstract_inverted_index.simplex | 75 |
| abstract_inverted_index.student | 11, 27, 92, 122, 178 |
| abstract_inverted_index.teacher | 25 |
| abstract_inverted_index.thereby | 90 |
| abstract_inverted_index.Collapse | 48 |
| abstract_inverted_index.Existing | 0 |
| abstract_inverted_index.However, | 19 |
| abstract_inverted_index.accuracy | 183 |
| abstract_inverted_index.analysis | 100 |
| abstract_inverted_index.benefits | 123 |
| abstract_inverted_index.existing | 139 |
| abstract_inverted_index.features | 72 |
| abstract_inverted_index.graceful | 66 |
| abstract_inverted_index.methods, | 141 |
| abstract_inverted_index.paradigm | 156 |
| abstract_inverted_index.process. | 38, 126 |
| abstract_inverted_index.students | 144 |
| abstract_inverted_index.Knowledge | 160 |
| abstract_inverted_index.achieving | 181 |
| abstract_inverted_index.alleviate | 84 |
| abstract_inverted_index.analysis, | 111 |
| abstract_inverted_index.distilled | 177 |
| abstract_inverted_index.empirical | 99 |
| abstract_inverted_index.encourage | 143 |
| abstract_inverted_index.enhancing | 91 |
| abstract_inverted_index.establish | 113 |
| abstract_inverted_index.features, | 135 |
| abstract_inverted_index.geometric | 67 |
| abstract_inverted_index.improving | 172 |
| abstract_inverted_index.introduce | 43 |
| abstract_inverted_index.knowledge | 1, 21, 86 |
| abstract_inverted_index.resulting | 63 |
| abstract_inverted_index.structure | 45, 68, 119 |
| abstract_inverted_index.teacher's | 117, 148 |
| abstract_inverted_index.teachers. | 18 |
| abstract_inverted_index.training, | 62 |
| abstract_inverted_index.typically | 55 |
| abstract_inverted_index.Therefore, | 127 |
| abstract_inverted_index.connection | 104 |
| abstract_inverted_index.effective, | 171 |
| abstract_inverted_index.framework. | 53 |
| abstract_inverted_index.last-layer | 71 |
| abstract_inverted_index.structure. | 150 |
| abstract_inverted_index.demonstrate | 165 |
| abstract_inverted_index.equiangular | 76 |
| abstract_inverted_index.experiments | 164 |
| abstract_inverted_index.hypothesize | 80 |
| abstract_inverted_index.performance | 13 |
| abstract_inverted_index.significant | 29 |
| abstract_inverted_index.Distillation | 161 |
| abstract_inverted_index.demonstrated | 6 |
| abstract_inverted_index.distillation | 2, 37, 125, 140, 155 |
| abstract_inverted_index.performance. | 93, 184 |
| abstract_inverted_index.transferring | 115, 131 |
| abstract_inverted_index.Comprehensive | 163 |
| abstract_inverted_index.distillation, | 89 |
| abstract_inverted_index.effectiveness | 34 |
| abstract_inverted_index.generalization | 174 |
| abstract_inverted_index.instance-level | 132 |
| abstract_inverted_index.state-of-the-art | 182 |
| abstract_inverted_index.Collapse-inspired | 159 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 3 |
| citation_normalized_percentile.value | 0.21155779 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |