Exploring Robust Overfitting for Pre-trained Language Models Article Swipe
Bin Zhu
,
Yanghui Rao
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.18653/v1/2023.findings-acl.340
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.18653/v1/2023.findings-acl.340
We identify the robust overfitting issue for pre-trained language models by showing that the robust test loss increases as the epoch grows. Through comprehensive exploration of the robust loss on the training set, we attribute robust overfitting to the model’s memorization of the adversarial training data. We attempt to mitigate robust overfitting by combining regularization methods with adversarial training. Following the philosophy that prevents the model from memorizing the adversarial data, we find that flooding, a regularization method with loss scaling, can mitigate robust overfitting for pre-trained language models. Eventually, we investigate the effect of flooding levels and evaluate the models’ adversarial robustness under textual attacks. Extensive experiments demonstrate that our methods can mitigate robust overfitting upon three top adversarial training methods and further promote adversarial robustness.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.18653/v1/2023.findings-acl.340
- https://aclanthology.org/2023.findings-acl.340.pdf
- OA Status
- gold
- Cited By
- 3
- References
- 54
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4385571681
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4385571681Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.18653/v1/2023.findings-acl.340Digital Object Identifier
- Title
-
Exploring Robust Overfitting for Pre-trained Language ModelsWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-01-01Full publication date if available
- Authors
-
Bin Zhu, Yanghui RaoList of authors in order
- Landing page
-
https://doi.org/10.18653/v1/2023.findings-acl.340Publisher landing page
- PDF URL
-
https://aclanthology.org/2023.findings-acl.340.pdfDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://aclanthology.org/2023.findings-acl.340.pdfDirect OA link when available
- Concepts
-
Overfitting, Adversarial system, Computer science, Artificial intelligence, Machine learning, Robustness (evolution), Regularization (linguistics), Training set, Artificial neural network, Chemistry, Biochemistry, GeneTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
3Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 2, 2024: 1Per-year citation counts (last 5 years)
- References (count)
-
54Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4385571681 |
|---|---|
| doi | https://doi.org/10.18653/v1/2023.findings-acl.340 |
| ids.doi | https://doi.org/10.18653/v1/2023.findings-acl.340 |
| ids.openalex | https://openalex.org/W4385571681 |
| fwci | 0.76632866 |
| type | article |
| title | Exploring Robust Overfitting for Pre-trained Language Models |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | 5522 |
| biblio.first_page | 5506 |
| topics[0].id | https://openalex.org/T10028 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9908999800682068 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Topic Modeling |
| topics[1].id | https://openalex.org/T11689 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9828000068664551 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Adversarial Robustness in Machine Learning |
| topics[2].id | https://openalex.org/T10181 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9811000227928162 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Natural Language Processing Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C22019652 |
| concepts[0].level | 3 |
| concepts[0].score | 0.9609353542327881 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q331309 |
| concepts[0].display_name | Overfitting |
| concepts[1].id | https://openalex.org/C37736160 |
| concepts[1].level | 2 |
| concepts[1].score | 0.7528773546218872 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q1801315 |
| concepts[1].display_name | Adversarial system |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.7073885202407837 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.6768550276756287 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C119857082 |
| concepts[4].level | 1 |
| concepts[4].score | 0.6408437490463257 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[4].display_name | Machine learning |
| concepts[5].id | https://openalex.org/C63479239 |
| concepts[5].level | 3 |
| concepts[5].score | 0.6032399535179138 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q7353546 |
| concepts[5].display_name | Robustness (evolution) |
| concepts[6].id | https://openalex.org/C2776135515 |
| concepts[6].level | 2 |
| concepts[6].score | 0.5085707902908325 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q17143721 |
| concepts[6].display_name | Regularization (linguistics) |
| concepts[7].id | https://openalex.org/C51632099 |
| concepts[7].level | 2 |
| concepts[7].score | 0.42490875720977783 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q3985153 |
| concepts[7].display_name | Training set |
| concepts[8].id | https://openalex.org/C50644808 |
| concepts[8].level | 2 |
| concepts[8].score | 0.12638738751411438 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q192776 |
| concepts[8].display_name | Artificial neural network |
| concepts[9].id | https://openalex.org/C185592680 |
| concepts[9].level | 0 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q2329 |
| concepts[9].display_name | Chemistry |
| concepts[10].id | https://openalex.org/C55493867 |
| concepts[10].level | 1 |
| concepts[10].score | 0.0 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q7094 |
| concepts[10].display_name | Biochemistry |
| concepts[11].id | https://openalex.org/C104317684 |
| concepts[11].level | 2 |
| concepts[11].score | 0.0 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q7187 |
| concepts[11].display_name | Gene |
| keywords[0].id | https://openalex.org/keywords/overfitting |
| keywords[0].score | 0.9609353542327881 |
| keywords[0].display_name | Overfitting |
| keywords[1].id | https://openalex.org/keywords/adversarial-system |
| keywords[1].score | 0.7528773546218872 |
| keywords[1].display_name | Adversarial system |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.7073885202407837 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.6768550276756287 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/machine-learning |
| keywords[4].score | 0.6408437490463257 |
| keywords[4].display_name | Machine learning |
| keywords[5].id | https://openalex.org/keywords/robustness |
| keywords[5].score | 0.6032399535179138 |
| keywords[5].display_name | Robustness (evolution) |
| keywords[6].id | https://openalex.org/keywords/regularization |
| keywords[6].score | 0.5085707902908325 |
| keywords[6].display_name | Regularization (linguistics) |
| keywords[7].id | https://openalex.org/keywords/training-set |
| keywords[7].score | 0.42490875720977783 |
| keywords[7].display_name | Training set |
| keywords[8].id | https://openalex.org/keywords/artificial-neural-network |
| keywords[8].score | 0.12638738751411438 |
| keywords[8].display_name | Artificial neural network |
| language | en |
| locations[0].id | doi:10.18653/v1/2023.findings-acl.340 |
| locations[0].is_oa | True |
| locations[0].source | |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://aclanthology.org/2023.findings-acl.340.pdf |
| locations[0].version | publishedVersion |
| locations[0].raw_type | proceedings-article |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | Findings of the Association for Computational Linguistics: ACL 2023 |
| locations[0].landing_page_url | https://doi.org/10.18653/v1/2023.findings-acl.340 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5100769745 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-6841-8062 |
| authorships[0].author.display_name | Bin Zhu |
| authorships[0].countries | CN |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I157773358 |
| authorships[0].affiliations[0].raw_affiliation_string | School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China |
| authorships[0].institutions[0].id | https://openalex.org/I157773358 |
| authorships[0].institutions[0].ror | https://ror.org/0064kty71 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I157773358 |
| authorships[0].institutions[0].country_code | CN |
| authorships[0].institutions[0].display_name | Sun Yat-sen University |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Bin Zhu |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China |
| authorships[1].author.id | https://openalex.org/A5058291454 |
| authorships[1].author.orcid | https://orcid.org/0000-0003-1610-9599 |
| authorships[1].author.display_name | Yanghui Rao |
| authorships[1].countries | CN |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I157773358 |
| authorships[1].affiliations[0].raw_affiliation_string | School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China |
| authorships[1].institutions[0].id | https://openalex.org/I157773358 |
| authorships[1].institutions[0].ror | https://ror.org/0064kty71 |
| authorships[1].institutions[0].type | education |
| authorships[1].institutions[0].lineage | https://openalex.org/I157773358 |
| authorships[1].institutions[0].country_code | CN |
| authorships[1].institutions[0].display_name | Sun Yat-sen University |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Yanghui Rao |
| authorships[1].is_corresponding | True |
| authorships[1].raw_affiliation_strings | School of Computer Science and Engineering, Sun Yat-sen University, Guangzhou, China |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://aclanthology.org/2023.findings-acl.340.pdf |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Exploring Robust Overfitting for Pre-trained Language Models |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10028 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9908999800682068 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Topic Modeling |
| related_works | https://openalex.org/W2944839767, https://openalex.org/W2989932438, https://openalex.org/W3099765033, https://openalex.org/W4210794429, https://openalex.org/W3132767683, https://openalex.org/W1996541855, https://openalex.org/W4311734044, https://openalex.org/W2970275441, https://openalex.org/W3204747554, https://openalex.org/W4295037467 |
| cited_by_count | 3 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 2 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 1 |
| locations_count | 1 |
| best_oa_location.id | doi:10.18653/v1/2023.findings-acl.340 |
| best_oa_location.is_oa | True |
| best_oa_location.source | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://aclanthology.org/2023.findings-acl.340.pdf |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | proceedings-article |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | Findings of the Association for Computational Linguistics: ACL 2023 |
| best_oa_location.landing_page_url | https://doi.org/10.18653/v1/2023.findings-acl.340 |
| primary_location.id | doi:10.18653/v1/2023.findings-acl.340 |
| primary_location.is_oa | True |
| primary_location.source | |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://aclanthology.org/2023.findings-acl.340.pdf |
| primary_location.version | publishedVersion |
| primary_location.raw_type | proceedings-article |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | Findings of the Association for Computational Linguistics: ACL 2023 |
| primary_location.landing_page_url | https://doi.org/10.18653/v1/2023.findings-acl.340 |
| publication_date | 2023-01-01 |
| publication_year | 2023 |
| referenced_works | https://openalex.org/W1945616565, https://openalex.org/W4221161154, https://openalex.org/W2963083752, https://openalex.org/W2095705004, https://openalex.org/W3169948074, https://openalex.org/W3008313523, https://openalex.org/W3092642435, https://openalex.org/W3174318528, https://openalex.org/W2923014074, https://openalex.org/W2963518130, https://openalex.org/W2996564870, https://openalex.org/W2732724430, https://openalex.org/W3092873005, https://openalex.org/W4226148503, https://openalex.org/W3197868468, https://openalex.org/W3035032188, https://openalex.org/W4288360049, https://openalex.org/W3101449015, https://openalex.org/W2963341956, https://openalex.org/W3124432598, https://openalex.org/W2170240176, https://openalex.org/W3120061794, https://openalex.org/W3006622081, https://openalex.org/W3211686893, https://openalex.org/W3084992427, https://openalex.org/W2963389226, https://openalex.org/W4283215156, https://openalex.org/W2144513243, https://openalex.org/W2970078867, https://openalex.org/W2996851481, https://openalex.org/W4285232587, https://openalex.org/W2949128310, https://openalex.org/W2963859254, https://openalex.org/W3035736465, https://openalex.org/W3177190797, https://openalex.org/W2964159205, https://openalex.org/W4288363925, https://openalex.org/W2913266441, https://openalex.org/W4287125811, https://openalex.org/W3105604018, https://openalex.org/W1673923490, https://openalex.org/W3122890974, https://openalex.org/W2251939518, https://openalex.org/W3213881810, https://openalex.org/W4287864733, https://openalex.org/W3008189420, https://openalex.org/W4307207457, https://openalex.org/W2970449623, https://openalex.org/W3035164976, https://openalex.org/W2180612164, https://openalex.org/W3137695714, https://openalex.org/W2963834268, https://openalex.org/W3168586460, https://openalex.org/W4293846201 |
| referenced_works_count | 54 |
| abstract_inverted_index.a | 75 |
| abstract_inverted_index.We | 0, 46 |
| abstract_inverted_index.as | 18 |
| abstract_inverted_index.by | 10, 52 |
| abstract_inverted_index.of | 25, 41, 94 |
| abstract_inverted_index.on | 29 |
| abstract_inverted_index.to | 37, 48 |
| abstract_inverted_index.we | 33, 71, 90 |
| abstract_inverted_index.and | 97, 122 |
| abstract_inverted_index.can | 81, 112 |
| abstract_inverted_index.for | 6, 85 |
| abstract_inverted_index.our | 110 |
| abstract_inverted_index.the | 2, 13, 19, 26, 30, 38, 42, 60, 64, 68, 92, 99 |
| abstract_inverted_index.top | 118 |
| abstract_inverted_index.find | 72 |
| abstract_inverted_index.from | 66 |
| abstract_inverted_index.loss | 16, 28, 79 |
| abstract_inverted_index.set, | 32 |
| abstract_inverted_index.test | 15 |
| abstract_inverted_index.that | 12, 62, 73, 109 |
| abstract_inverted_index.upon | 116 |
| abstract_inverted_index.with | 56, 78 |
| abstract_inverted_index.data, | 70 |
| abstract_inverted_index.data. | 45 |
| abstract_inverted_index.epoch | 20 |
| abstract_inverted_index.issue | 5 |
| abstract_inverted_index.model | 65 |
| abstract_inverted_index.three | 117 |
| abstract_inverted_index.under | 103 |
| abstract_inverted_index.effect | 93 |
| abstract_inverted_index.grows. | 21 |
| abstract_inverted_index.levels | 96 |
| abstract_inverted_index.method | 77 |
| abstract_inverted_index.models | 9 |
| abstract_inverted_index.robust | 3, 14, 27, 35, 50, 83, 114 |
| abstract_inverted_index.Through | 22 |
| abstract_inverted_index.attempt | 47 |
| abstract_inverted_index.further | 123 |
| abstract_inverted_index.methods | 55, 111, 121 |
| abstract_inverted_index.models. | 88 |
| abstract_inverted_index.promote | 124 |
| abstract_inverted_index.showing | 11 |
| abstract_inverted_index.textual | 104 |
| abstract_inverted_index.attacks. | 105 |
| abstract_inverted_index.evaluate | 98 |
| abstract_inverted_index.flooding | 95 |
| abstract_inverted_index.identify | 1 |
| abstract_inverted_index.language | 8, 87 |
| abstract_inverted_index.mitigate | 49, 82, 113 |
| abstract_inverted_index.prevents | 63 |
| abstract_inverted_index.scaling, | 80 |
| abstract_inverted_index.training | 31, 44, 120 |
| abstract_inverted_index.Extensive | 106 |
| abstract_inverted_index.Following | 59 |
| abstract_inverted_index.attribute | 34 |
| abstract_inverted_index.combining | 53 |
| abstract_inverted_index.flooding, | 74 |
| abstract_inverted_index.increases | 17 |
| abstract_inverted_index.models’ | 100 |
| abstract_inverted_index.model’s | 39 |
| abstract_inverted_index.training. | 58 |
| abstract_inverted_index.memorizing | 67 |
| abstract_inverted_index.philosophy | 61 |
| abstract_inverted_index.robustness | 102 |
| abstract_inverted_index.Eventually, | 89 |
| abstract_inverted_index.adversarial | 43, 57, 69, 101, 119, 125 |
| abstract_inverted_index.demonstrate | 108 |
| abstract_inverted_index.experiments | 107 |
| abstract_inverted_index.exploration | 24 |
| abstract_inverted_index.investigate | 91 |
| abstract_inverted_index.overfitting | 4, 36, 51, 84, 115 |
| abstract_inverted_index.pre-trained | 7, 86 |
| abstract_inverted_index.robustness. | 126 |
| abstract_inverted_index.memorization | 40 |
| abstract_inverted_index.comprehensive | 23 |
| abstract_inverted_index.regularization | 54, 76 |
| cited_by_percentile_year.max | 97 |
| cited_by_percentile_year.min | 90 |
| corresponding_author_ids | https://openalex.org/A5058291454 |
| countries_distinct_count | 1 |
| institutions_distinct_count | 2 |
| corresponding_institution_ids | https://openalex.org/I157773358 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.6200000047683716 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile.value | 0.73575462 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |