Generalization Analysis for Contrastive Representation Learning Article Swipe
Yunwen Lei
,
Tianbao Yang
,
Yiming Ying
,
Ding‐Xuan Zhou
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2302.12383
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2302.12383
Recently, contrastive learning has found impressive success in advancing the state of the art in solving various machine learning tasks. However, the existing generalization analysis is very limited or even not meaningful. In particular, the existing generalization error bounds depend linearly on the number $k$ of negative examples while it was widely shown in practice that choosing a large $k$ is necessary to guarantee good generalization of contrastive learning in downstream tasks. In this paper, we establish novel generalization bounds for contrastive learning which do not depend on $k$, up to logarithmic terms. Our analysis uses structural results on empirical covering numbers and Rademacher complexities to exploit the Lipschitz continuity of loss functions. For self-bounding Lipschitz loss functions, we further improve our results by developing optimistic bounds which imply fast rates in a low noise condition. We apply our results to learning with both linear representation and nonlinear representation by deep neural networks, for both of which we derive Rademacher complexity bounds to get improved generalization bounds.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2302.12383
- https://arxiv.org/pdf/2302.12383
- OA Status
- green
- Cited By
- 1
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4322759322
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4322759322Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2302.12383Digital Object Identifier
- Title
-
Generalization Analysis for Contrastive Representation LearningWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-02-24Full publication date if available
- Authors
-
Yunwen Lei, Tianbao Yang, Yiming Ying, Ding‐Xuan ZhouList of authors in order
- Landing page
-
https://arxiv.org/abs/2302.12383Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2302.12383Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2302.12383Direct OA link when available
- Concepts
-
Generalization, Computer science, Representation (politics), Lipschitz continuity, Bounding overwatch, Logarithm, Empirical risk minimization, Artificial intelligence, Theoretical computer science, Algorithm, Mathematics, Pure mathematics, Law, Politics, Political science, Mathematical analysisTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
1Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4322759322 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2302.12383 |
| ids.doi | https://doi.org/10.48550/arxiv.2302.12383 |
| ids.openalex | https://openalex.org/W4322759322 |
| fwci | |
| type | preprint |
| title | Generalization Analysis for Contrastive Representation Learning |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10500 |
| topics[0].field.id | https://openalex.org/fields/22 |
| topics[0].field.display_name | Engineering |
| topics[0].score | 0.9976999759674072 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/2206 |
| topics[0].subfield.display_name | Computational Mechanics |
| topics[0].display_name | Sparse and Compressive Sensing Techniques |
| topics[1].id | https://openalex.org/T11307 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9937000274658203 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Domain Adaptation and Few-Shot Learning |
| topics[2].id | https://openalex.org/T12676 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9918000102043152 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Machine Learning and ELM |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C177148314 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8497493267059326 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q170084 |
| concepts[0].display_name | Generalization |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.6497832536697388 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C2776359362 |
| concepts[2].level | 3 |
| concepts[2].score | 0.6001851558685303 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q2145286 |
| concepts[2].display_name | Representation (politics) |
| concepts[3].id | https://openalex.org/C22324862 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5731357932090759 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q652707 |
| concepts[3].display_name | Lipschitz continuity |
| concepts[4].id | https://openalex.org/C63584917 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5370439887046814 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q333286 |
| concepts[4].display_name | Bounding overwatch |
| concepts[5].id | https://openalex.org/C39927690 |
| concepts[5].level | 2 |
| concepts[5].score | 0.49431872367858887 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11197 |
| concepts[5].display_name | Logarithm |
| concepts[6].id | https://openalex.org/C107321475 |
| concepts[6].level | 2 |
| concepts[6].score | 0.4314943552017212 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q5374254 |
| concepts[6].display_name | Empirical risk minimization |
| concepts[7].id | https://openalex.org/C154945302 |
| concepts[7].level | 1 |
| concepts[7].score | 0.42852693796157837 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[7].display_name | Artificial intelligence |
| concepts[8].id | https://openalex.org/C80444323 |
| concepts[8].level | 1 |
| concepts[8].score | 0.35974013805389404 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q2878974 |
| concepts[8].display_name | Theoretical computer science |
| concepts[9].id | https://openalex.org/C11413529 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3381906747817993 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q8366 |
| concepts[9].display_name | Algorithm |
| concepts[10].id | https://openalex.org/C33923547 |
| concepts[10].level | 0 |
| concepts[10].score | 0.28326454758644104 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[10].display_name | Mathematics |
| concepts[11].id | https://openalex.org/C202444582 |
| concepts[11].level | 1 |
| concepts[11].score | 0.07819956541061401 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q837863 |
| concepts[11].display_name | Pure mathematics |
| concepts[12].id | https://openalex.org/C199539241 |
| concepts[12].level | 1 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q7748 |
| concepts[12].display_name | Law |
| concepts[13].id | https://openalex.org/C94625758 |
| concepts[13].level | 2 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q7163 |
| concepts[13].display_name | Politics |
| concepts[14].id | https://openalex.org/C17744445 |
| concepts[14].level | 0 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q36442 |
| concepts[14].display_name | Political science |
| concepts[15].id | https://openalex.org/C134306372 |
| concepts[15].level | 1 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q7754 |
| concepts[15].display_name | Mathematical analysis |
| keywords[0].id | https://openalex.org/keywords/generalization |
| keywords[0].score | 0.8497493267059326 |
| keywords[0].display_name | Generalization |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.6497832536697388 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/representation |
| keywords[2].score | 0.6001851558685303 |
| keywords[2].display_name | Representation (politics) |
| keywords[3].id | https://openalex.org/keywords/lipschitz-continuity |
| keywords[3].score | 0.5731357932090759 |
| keywords[3].display_name | Lipschitz continuity |
| keywords[4].id | https://openalex.org/keywords/bounding-overwatch |
| keywords[4].score | 0.5370439887046814 |
| keywords[4].display_name | Bounding overwatch |
| keywords[5].id | https://openalex.org/keywords/logarithm |
| keywords[5].score | 0.49431872367858887 |
| keywords[5].display_name | Logarithm |
| keywords[6].id | https://openalex.org/keywords/empirical-risk-minimization |
| keywords[6].score | 0.4314943552017212 |
| keywords[6].display_name | Empirical risk minimization |
| keywords[7].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[7].score | 0.42852693796157837 |
| keywords[7].display_name | Artificial intelligence |
| keywords[8].id | https://openalex.org/keywords/theoretical-computer-science |
| keywords[8].score | 0.35974013805389404 |
| keywords[8].display_name | Theoretical computer science |
| keywords[9].id | https://openalex.org/keywords/algorithm |
| keywords[9].score | 0.3381906747817993 |
| keywords[9].display_name | Algorithm |
| keywords[10].id | https://openalex.org/keywords/mathematics |
| keywords[10].score | 0.28326454758644104 |
| keywords[10].display_name | Mathematics |
| keywords[11].id | https://openalex.org/keywords/pure-mathematics |
| keywords[11].score | 0.07819956541061401 |
| keywords[11].display_name | Pure mathematics |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2302.12383 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2302.12383 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2302.12383 |
| locations[1].id | doi:10.48550/arxiv.2302.12383 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2302.12383 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5046468616 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-5383-467X |
| authorships[0].author.display_name | Yunwen Lei |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Lei, Yunwen |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5023288846 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-7858-5438 |
| authorships[1].author.display_name | Tianbao Yang |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Yang, Tianbao |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5048960543 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Yiming Ying |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Ying, Yiming |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5006802589 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-0224-9216 |
| authorships[3].author.display_name | Ding‐Xuan Zhou |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Zhou, Ding-Xuan |
| authorships[3].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2302.12383 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Generalization Analysis for Contrastive Representation Learning |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10500 |
| primary_topic.field.id | https://openalex.org/fields/22 |
| primary_topic.field.display_name | Engineering |
| primary_topic.score | 0.9976999759674072 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/2206 |
| primary_topic.subfield.display_name | Computational Mechanics |
| primary_topic.display_name | Sparse and Compressive Sensing Techniques |
| related_works | https://openalex.org/W3185235544, https://openalex.org/W4297791327, https://openalex.org/W2897842840, https://openalex.org/W2911623553, https://openalex.org/W2397777611, https://openalex.org/W1502031429, https://openalex.org/W2318923278, https://openalex.org/W4299587028, https://openalex.org/W2398277178, https://openalex.org/W2177289819 |
| cited_by_count | 1 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2302.12383 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2302.12383 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2302.12383 |
| primary_location.id | pmh:oai:arXiv.org:2302.12383 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2302.12383 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2302.12383 |
| publication_date | 2023-02-24 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 57, 132 |
| abstract_inverted_index.In | 32, 72 |
| abstract_inverted_index.We | 136 |
| abstract_inverted_index.by | 123, 149 |
| abstract_inverted_index.do | 84 |
| abstract_inverted_index.in | 7, 14, 53, 69, 131 |
| abstract_inverted_index.is | 25, 60 |
| abstract_inverted_index.it | 49 |
| abstract_inverted_index.of | 11, 45, 66, 110, 155 |
| abstract_inverted_index.on | 41, 87, 98 |
| abstract_inverted_index.or | 28 |
| abstract_inverted_index.to | 62, 90, 105, 140, 162 |
| abstract_inverted_index.up | 89 |
| abstract_inverted_index.we | 75, 118, 157 |
| abstract_inverted_index.$k$ | 44, 59 |
| abstract_inverted_index.For | 113 |
| abstract_inverted_index.Our | 93 |
| abstract_inverted_index.and | 102, 146 |
| abstract_inverted_index.art | 13 |
| abstract_inverted_index.for | 80, 153 |
| abstract_inverted_index.get | 163 |
| abstract_inverted_index.has | 3 |
| abstract_inverted_index.low | 133 |
| abstract_inverted_index.not | 30, 85 |
| abstract_inverted_index.our | 121, 138 |
| abstract_inverted_index.the | 9, 12, 21, 34, 42, 107 |
| abstract_inverted_index.was | 50 |
| abstract_inverted_index.$k$, | 88 |
| abstract_inverted_index.both | 143, 154 |
| abstract_inverted_index.deep | 150 |
| abstract_inverted_index.even | 29 |
| abstract_inverted_index.fast | 129 |
| abstract_inverted_index.good | 64 |
| abstract_inverted_index.loss | 111, 116 |
| abstract_inverted_index.that | 55 |
| abstract_inverted_index.this | 73 |
| abstract_inverted_index.uses | 95 |
| abstract_inverted_index.very | 26 |
| abstract_inverted_index.with | 142 |
| abstract_inverted_index.apply | 137 |
| abstract_inverted_index.error | 37 |
| abstract_inverted_index.found | 4 |
| abstract_inverted_index.imply | 128 |
| abstract_inverted_index.large | 58 |
| abstract_inverted_index.noise | 134 |
| abstract_inverted_index.novel | 77 |
| abstract_inverted_index.rates | 130 |
| abstract_inverted_index.shown | 52 |
| abstract_inverted_index.state | 10 |
| abstract_inverted_index.which | 83, 127, 156 |
| abstract_inverted_index.while | 48 |
| abstract_inverted_index.bounds | 38, 79, 126, 161 |
| abstract_inverted_index.depend | 39, 86 |
| abstract_inverted_index.derive | 158 |
| abstract_inverted_index.linear | 144 |
| abstract_inverted_index.neural | 151 |
| abstract_inverted_index.number | 43 |
| abstract_inverted_index.paper, | 74 |
| abstract_inverted_index.tasks. | 19, 71 |
| abstract_inverted_index.terms. | 92 |
| abstract_inverted_index.widely | 51 |
| abstract_inverted_index.bounds. | 166 |
| abstract_inverted_index.exploit | 106 |
| abstract_inverted_index.further | 119 |
| abstract_inverted_index.improve | 120 |
| abstract_inverted_index.limited | 27 |
| abstract_inverted_index.machine | 17 |
| abstract_inverted_index.numbers | 101 |
| abstract_inverted_index.results | 97, 122, 139 |
| abstract_inverted_index.solving | 15 |
| abstract_inverted_index.success | 6 |
| abstract_inverted_index.various | 16 |
| abstract_inverted_index.However, | 20 |
| abstract_inverted_index.analysis | 24, 94 |
| abstract_inverted_index.choosing | 56 |
| abstract_inverted_index.covering | 100 |
| abstract_inverted_index.examples | 47 |
| abstract_inverted_index.existing | 22, 35 |
| abstract_inverted_index.improved | 164 |
| abstract_inverted_index.learning | 2, 18, 68, 82, 141 |
| abstract_inverted_index.linearly | 40 |
| abstract_inverted_index.negative | 46 |
| abstract_inverted_index.practice | 54 |
| abstract_inverted_index.Lipschitz | 108, 115 |
| abstract_inverted_index.Recently, | 0 |
| abstract_inverted_index.advancing | 8 |
| abstract_inverted_index.empirical | 99 |
| abstract_inverted_index.establish | 76 |
| abstract_inverted_index.guarantee | 63 |
| abstract_inverted_index.necessary | 61 |
| abstract_inverted_index.networks, | 152 |
| abstract_inverted_index.nonlinear | 147 |
| abstract_inverted_index.Rademacher | 103, 159 |
| abstract_inverted_index.complexity | 160 |
| abstract_inverted_index.condition. | 135 |
| abstract_inverted_index.continuity | 109 |
| abstract_inverted_index.developing | 124 |
| abstract_inverted_index.downstream | 70 |
| abstract_inverted_index.functions, | 117 |
| abstract_inverted_index.functions. | 112 |
| abstract_inverted_index.impressive | 5 |
| abstract_inverted_index.optimistic | 125 |
| abstract_inverted_index.structural | 96 |
| abstract_inverted_index.contrastive | 1, 67, 81 |
| abstract_inverted_index.logarithmic | 91 |
| abstract_inverted_index.meaningful. | 31 |
| abstract_inverted_index.particular, | 33 |
| abstract_inverted_index.complexities | 104 |
| abstract_inverted_index.self-bounding | 114 |
| abstract_inverted_index.generalization | 23, 36, 65, 78, 165 |
| abstract_inverted_index.representation | 145, 148 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |