The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning Article Swipe
Borja Rodríguez-Gálvez
,
Arno Blaas
,
Pau Rodríguez
,
Adam Goliński
,
Xavier Suau
,
Jason Ramapuram
,
Dan Busbridge
,
Luca Zappella
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2307.10907
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2307.10907
The mechanisms behind the success of multi-view self-supervised learning (MVSSL) are not yet fully understood. Contrastive MVSSL methods have been studied through the lens of InfoNCE, a lower bound of the Mutual Information (MI). However, the relation between other MVSSL methods and MI remains unclear. We consider a different lower bound on the MI consisting of an entropy and a reconstruction term (ER), and analyze the main MVSSL families through its lens. Through this ER bound, we show that clustering-based methods such as DeepCluster and SwAV maximize the MI. We also re-interpret the mechanisms of distillation-based approaches such as BYOL and DINO, showing that they explicitly maximize the reconstruction term and implicitly encourage a stable entropy, and we confirm this empirically. We show that replacing the objectives of common MVSSL methods with this ER bound achieves competitive performance, while making them stable when training with smaller batch sizes or smaller exponential moving average (EMA) coefficients. Github repo: https://github.com/apple/ml-entropy-reconstruction.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2307.10907
- https://arxiv.org/pdf/2307.10907
- OA Status
- green
- Cited By
- 1
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4385005536
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4385005536Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2307.10907Digital Object Identifier
- Title
-
The Role of Entropy and Reconstruction in Multi-View Self-Supervised LearningWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-07-20Full publication date if available
- Authors
-
Borja Rodríguez-Gálvez, Arno Blaas, Pau Rodríguez, Adam Goliński, Xavier Suau, Jason Ramapuram, Dan Busbridge, Luca ZappellaList of authors in order
- Landing page
-
https://arxiv.org/abs/2307.10907Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2307.10907Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2307.10907Direct OA link when available
- Concepts
-
Upper and lower bounds, Entropy (arrow of time), Computer science, Cluster analysis, Artificial intelligence, Machine learning, Algorithm, Mathematics, Physics, Mathematical analysis, Quantum mechanicsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
1Total citation count in OpenAlex
- Citations by year (recent)
-
2024: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4385005536 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2307.10907 |
| ids.doi | https://doi.org/10.48550/arxiv.2307.10907 |
| ids.openalex | https://openalex.org/W4385005536 |
| fwci | |
| type | preprint |
| title | The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T11307 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.8959000110626221 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Domain Adaptation and Few-Shot Learning |
| topics[1].id | https://openalex.org/T13731 |
| topics[1].field.id | https://openalex.org/fields/33 |
| topics[1].field.display_name | Social Sciences |
| topics[1].score | 0.8615999817848206 |
| topics[1].domain.id | https://openalex.org/domains/2 |
| topics[1].domain.display_name | Social Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/3322 |
| topics[1].subfield.display_name | Urban Studies |
| topics[1].display_name | Advanced Computing and Algorithms |
| topics[2].id | https://openalex.org/T10331 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.8317999839782715 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1707 |
| topics[2].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[2].display_name | Video Surveillance and Tracking Methods |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C77553402 |
| concepts[0].level | 2 |
| concepts[0].score | 0.656778872013092 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q13222579 |
| concepts[0].display_name | Upper and lower bounds |
| concepts[1].id | https://openalex.org/C106301342 |
| concepts[1].level | 2 |
| concepts[1].score | 0.6071505546569824 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q4117933 |
| concepts[1].display_name | Entropy (arrow of time) |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.5910906195640564 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C73555534 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5171305537223816 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q622825 |
| concepts[3].display_name | Cluster analysis |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.4610968828201294 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C119857082 |
| concepts[5].level | 1 |
| concepts[5].score | 0.376832515001297 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[5].display_name | Machine learning |
| concepts[6].id | https://openalex.org/C11413529 |
| concepts[6].level | 1 |
| concepts[6].score | 0.3559545874595642 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q8366 |
| concepts[6].display_name | Algorithm |
| concepts[7].id | https://openalex.org/C33923547 |
| concepts[7].level | 0 |
| concepts[7].score | 0.32972845435142517 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[7].display_name | Mathematics |
| concepts[8].id | https://openalex.org/C121332964 |
| concepts[8].level | 0 |
| concepts[8].score | 0.09164905548095703 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[8].display_name | Physics |
| concepts[9].id | https://openalex.org/C134306372 |
| concepts[9].level | 1 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q7754 |
| concepts[9].display_name | Mathematical analysis |
| concepts[10].id | https://openalex.org/C62520636 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q944 |
| concepts[10].display_name | Quantum mechanics |
| keywords[0].id | https://openalex.org/keywords/upper-and-lower-bounds |
| keywords[0].score | 0.656778872013092 |
| keywords[0].display_name | Upper and lower bounds |
| keywords[1].id | https://openalex.org/keywords/entropy |
| keywords[1].score | 0.6071505546569824 |
| keywords[1].display_name | Entropy (arrow of time) |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.5910906195640564 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/cluster-analysis |
| keywords[3].score | 0.5171305537223816 |
| keywords[3].display_name | Cluster analysis |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.4610968828201294 |
| keywords[4].display_name | Artificial intelligence |
| keywords[5].id | https://openalex.org/keywords/machine-learning |
| keywords[5].score | 0.376832515001297 |
| keywords[5].display_name | Machine learning |
| keywords[6].id | https://openalex.org/keywords/algorithm |
| keywords[6].score | 0.3559545874595642 |
| keywords[6].display_name | Algorithm |
| keywords[7].id | https://openalex.org/keywords/mathematics |
| keywords[7].score | 0.32972845435142517 |
| keywords[7].display_name | Mathematics |
| keywords[8].id | https://openalex.org/keywords/physics |
| keywords[8].score | 0.09164905548095703 |
| keywords[8].display_name | Physics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2307.10907 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2307.10907 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2307.10907 |
| locations[1].id | doi:10.48550/arxiv.2307.10907 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2307.10907 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5041149980 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-0862-1333 |
| authorships[0].author.display_name | Borja Rodríguez-Gálvez |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Rodríguez-Gálvez, Borja |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5086300347 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Arno Blaas |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Blaas, Arno |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5103191934 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-1689-8084 |
| authorships[2].author.display_name | Pau Rodríguez |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Rodríguez, Pau |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5103022758 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-8873-1072 |
| authorships[3].author.display_name | Adam Goliński |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Goliński, Adam |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5046208400 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-4052-7048 |
| authorships[4].author.display_name | Xavier Suau |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Suau, Xavier |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5020604198 |
| authorships[5].author.orcid | https://orcid.org/0000-0001-9717-1188 |
| authorships[5].author.display_name | Jason Ramapuram |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Ramapuram, Jason |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5050567760 |
| authorships[6].author.orcid | https://orcid.org/0000-0002-2178-6917 |
| authorships[6].author.display_name | Dan Busbridge |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Busbridge, Dan |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5043365442 |
| authorships[7].author.orcid | |
| authorships[7].author.display_name | Luca Zappella |
| authorships[7].author_position | last |
| authorships[7].raw_author_name | Zappella, Luca |
| authorships[7].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2307.10907 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2023-07-22T00:00:00 |
| display_name | The Role of Entropy and Reconstruction in Multi-View Self-Supervised Learning |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T11307 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.8959000110626221 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Domain Adaptation and Few-Shot Learning |
| related_works | https://openalex.org/W2051487156, https://openalex.org/W2073681303, https://openalex.org/W4298130764, https://openalex.org/W2804364458, https://openalex.org/W2053286651, https://openalex.org/W2181743346, https://openalex.org/W2187401768, https://openalex.org/W2132641928, https://openalex.org/W2181413294, https://openalex.org/W4310225030 |
| cited_by_count | 1 |
| counts_by_year[0].year | 2024 |
| counts_by_year[0].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2307.10907 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2307.10907 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2307.10907 |
| primary_location.id | pmh:oai:arXiv.org:2307.10907 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2307.10907 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2307.10907 |
| publication_date | 2023-07-20 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 26, 47, 59, 113 |
| abstract_inverted_index.ER | 74, 133 |
| abstract_inverted_index.MI | 42, 53 |
| abstract_inverted_index.We | 45, 89, 121 |
| abstract_inverted_index.an | 56 |
| abstract_inverted_index.as | 82, 98 |
| abstract_inverted_index.of | 5, 24, 29, 55, 94, 127 |
| abstract_inverted_index.on | 51 |
| abstract_inverted_index.or | 148 |
| abstract_inverted_index.we | 76, 117 |
| abstract_inverted_index.MI. | 88 |
| abstract_inverted_index.The | 0 |
| abstract_inverted_index.and | 41, 58, 63, 84, 100, 110, 116 |
| abstract_inverted_index.are | 10 |
| abstract_inverted_index.its | 70 |
| abstract_inverted_index.not | 11 |
| abstract_inverted_index.the | 3, 22, 30, 35, 52, 65, 87, 92, 107, 125 |
| abstract_inverted_index.yet | 12 |
| abstract_inverted_index.BYOL | 99 |
| abstract_inverted_index.SwAV | 85 |
| abstract_inverted_index.also | 90 |
| abstract_inverted_index.been | 19 |
| abstract_inverted_index.have | 18 |
| abstract_inverted_index.lens | 23 |
| abstract_inverted_index.main | 66 |
| abstract_inverted_index.show | 77, 122 |
| abstract_inverted_index.such | 81, 97 |
| abstract_inverted_index.term | 61, 109 |
| abstract_inverted_index.that | 78, 103, 123 |
| abstract_inverted_index.them | 140 |
| abstract_inverted_index.they | 104 |
| abstract_inverted_index.this | 73, 119, 132 |
| abstract_inverted_index.when | 142 |
| abstract_inverted_index.with | 131, 144 |
| abstract_inverted_index.(EMA) | 153 |
| abstract_inverted_index.(ER), | 62 |
| abstract_inverted_index.(MI). | 33 |
| abstract_inverted_index.DINO, | 101 |
| abstract_inverted_index.MVSSL | 16, 39, 67, 129 |
| abstract_inverted_index.batch | 146 |
| abstract_inverted_index.bound | 28, 50, 134 |
| abstract_inverted_index.fully | 13 |
| abstract_inverted_index.lens. | 71 |
| abstract_inverted_index.lower | 27, 49 |
| abstract_inverted_index.other | 38 |
| abstract_inverted_index.repo: | 156 |
| abstract_inverted_index.sizes | 147 |
| abstract_inverted_index.while | 138 |
| abstract_inverted_index.Github | 155 |
| abstract_inverted_index.Mutual | 31 |
| abstract_inverted_index.behind | 2 |
| abstract_inverted_index.bound, | 75 |
| abstract_inverted_index.common | 128 |
| abstract_inverted_index.making | 139 |
| abstract_inverted_index.moving | 151 |
| abstract_inverted_index.stable | 114, 141 |
| abstract_inverted_index.(MVSSL) | 9 |
| abstract_inverted_index.Through | 72 |
| abstract_inverted_index.analyze | 64 |
| abstract_inverted_index.average | 152 |
| abstract_inverted_index.between | 37 |
| abstract_inverted_index.confirm | 118 |
| abstract_inverted_index.entropy | 57 |
| abstract_inverted_index.methods | 17, 40, 80, 130 |
| abstract_inverted_index.remains | 43 |
| abstract_inverted_index.showing | 102 |
| abstract_inverted_index.smaller | 145, 149 |
| abstract_inverted_index.studied | 20 |
| abstract_inverted_index.success | 4 |
| abstract_inverted_index.through | 21, 69 |
| abstract_inverted_index.However, | 34 |
| abstract_inverted_index.InfoNCE, | 25 |
| abstract_inverted_index.achieves | 135 |
| abstract_inverted_index.consider | 46 |
| abstract_inverted_index.entropy, | 115 |
| abstract_inverted_index.families | 68 |
| abstract_inverted_index.learning | 8 |
| abstract_inverted_index.maximize | 86, 106 |
| abstract_inverted_index.relation | 36 |
| abstract_inverted_index.training | 143 |
| abstract_inverted_index.unclear. | 44 |
| abstract_inverted_index.different | 48 |
| abstract_inverted_index.encourage | 112 |
| abstract_inverted_index.replacing | 124 |
| abstract_inverted_index.approaches | 96 |
| abstract_inverted_index.consisting | 54 |
| abstract_inverted_index.explicitly | 105 |
| abstract_inverted_index.implicitly | 111 |
| abstract_inverted_index.mechanisms | 1, 93 |
| abstract_inverted_index.multi-view | 6 |
| abstract_inverted_index.objectives | 126 |
| abstract_inverted_index.Contrastive | 15 |
| abstract_inverted_index.DeepCluster | 83 |
| abstract_inverted_index.Information | 32 |
| abstract_inverted_index.competitive | 136 |
| abstract_inverted_index.exponential | 150 |
| abstract_inverted_index.understood. | 14 |
| abstract_inverted_index.empirically. | 120 |
| abstract_inverted_index.performance, | 137 |
| abstract_inverted_index.re-interpret | 91 |
| abstract_inverted_index.coefficients. | 154 |
| abstract_inverted_index.reconstruction | 60, 108 |
| abstract_inverted_index.self-supervised | 7 |
| abstract_inverted_index.clustering-based | 79 |
| abstract_inverted_index.distillation-based | 95 |
| abstract_inverted_index.https://github.com/apple/ml-entropy-reconstruction. | 157 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 8 |
| citation_normalized_percentile |