Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance Article Swipe
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2311.01108
Adopting a two-stage paradigm of pretraining followed by fine-tuning, Pretrained Language Models (PLMs) have achieved substantial advancements in the field of natural language processing. However, in real-world scenarios, data labels are often noisy due to the complex annotation process, making it essential to develop strategies for fine-tuning PLMs with such noisy labels. To this end, we introduce an innovative approach for fine-tuning PLMs using noisy labels, which incorporates the guidance of Large Language Models (LLMs) like ChatGPT. This guidance assists in accurately distinguishing between clean and noisy samples and provides supplementary information beyond the noisy labels, thereby boosting the learning process during fine-tuning PLMs. Extensive experiments on synthetic and real-world noisy datasets further demonstrate the superior advantages of our framework over the state-of-the-art baselines.
Related Topics
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2311.01108
- https://arxiv.org/pdf/2311.01108
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4388329095
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4388329095Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2311.01108Digital Object Identifier
- Title
-
Noise-Robust Fine-Tuning of Pretrained Language Models via External GuidanceWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-11-02Full publication date if available
- Authors
-
Song Wang, Zhen Tan, Ruocheng Guo, Jundong LiList of authors in order
- Landing page
-
https://arxiv.org/abs/2311.01108Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2311.01108Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2311.01108Direct OA link when available
- Concepts
-
Computer science, Fine-tuning, Boosting (machine learning), Artificial intelligence, Noisy data, Noise (video), Process (computing), Machine learning, Field (mathematics), Language model, Annotation, Robustness (evolution), Physics, Gene, Operating system, Image (mathematics), Biochemistry, Quantum mechanics, Chemistry, Pure mathematics, MathematicsTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4388329095 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2311.01108 |
| ids.doi | https://doi.org/10.48550/arxiv.2311.01108 |
| ids.openalex | https://openalex.org/W4388329095 |
| fwci | |
| type | preprint |
| title | Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10181 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9983999729156494 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Natural Language Processing Techniques |
| topics[1].id | https://openalex.org/T10028 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9934999942779541 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Topic Modeling |
| topics[2].id | https://openalex.org/T10201 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9527999758720398 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Speech Recognition and Synthesis |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.7795356512069702 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C157524613 |
| concepts[1].level | 2 |
| concepts[1].score | 0.6921828985214233 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q2828883 |
| concepts[1].display_name | Fine-tuning |
| concepts[2].id | https://openalex.org/C46686674 |
| concepts[2].level | 2 |
| concepts[2].score | 0.631691038608551 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q466303 |
| concepts[2].display_name | Boosting (machine learning) |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.6146757006645203 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C2781170535 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5651628375053406 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q30587856 |
| concepts[4].display_name | Noisy data |
| concepts[5].id | https://openalex.org/C99498987 |
| concepts[5].level | 3 |
| concepts[5].score | 0.5247949361801147 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q2210247 |
| concepts[5].display_name | Noise (video) |
| concepts[6].id | https://openalex.org/C98045186 |
| concepts[6].level | 2 |
| concepts[6].score | 0.5190216302871704 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q205663 |
| concepts[6].display_name | Process (computing) |
| concepts[7].id | https://openalex.org/C119857082 |
| concepts[7].level | 1 |
| concepts[7].score | 0.4693494737148285 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[7].display_name | Machine learning |
| concepts[8].id | https://openalex.org/C9652623 |
| concepts[8].level | 2 |
| concepts[8].score | 0.44576019048690796 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q190109 |
| concepts[8].display_name | Field (mathematics) |
| concepts[9].id | https://openalex.org/C137293760 |
| concepts[9].level | 2 |
| concepts[9].score | 0.42698436975479126 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q3621696 |
| concepts[9].display_name | Language model |
| concepts[10].id | https://openalex.org/C2776321320 |
| concepts[10].level | 2 |
| concepts[10].score | 0.4168662130832672 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q857525 |
| concepts[10].display_name | Annotation |
| concepts[11].id | https://openalex.org/C63479239 |
| concepts[11].level | 3 |
| concepts[11].score | 0.41627585887908936 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q7353546 |
| concepts[11].display_name | Robustness (evolution) |
| concepts[12].id | https://openalex.org/C121332964 |
| concepts[12].level | 0 |
| concepts[12].score | 0.0 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[12].display_name | Physics |
| concepts[13].id | https://openalex.org/C104317684 |
| concepts[13].level | 2 |
| concepts[13].score | 0.0 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q7187 |
| concepts[13].display_name | Gene |
| concepts[14].id | https://openalex.org/C111919701 |
| concepts[14].level | 1 |
| concepts[14].score | 0.0 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q9135 |
| concepts[14].display_name | Operating system |
| concepts[15].id | https://openalex.org/C115961682 |
| concepts[15].level | 2 |
| concepts[15].score | 0.0 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q860623 |
| concepts[15].display_name | Image (mathematics) |
| concepts[16].id | https://openalex.org/C55493867 |
| concepts[16].level | 1 |
| concepts[16].score | 0.0 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q7094 |
| concepts[16].display_name | Biochemistry |
| concepts[17].id | https://openalex.org/C62520636 |
| concepts[17].level | 1 |
| concepts[17].score | 0.0 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q944 |
| concepts[17].display_name | Quantum mechanics |
| concepts[18].id | https://openalex.org/C185592680 |
| concepts[18].level | 0 |
| concepts[18].score | 0.0 |
| concepts[18].wikidata | https://www.wikidata.org/wiki/Q2329 |
| concepts[18].display_name | Chemistry |
| concepts[19].id | https://openalex.org/C202444582 |
| concepts[19].level | 1 |
| concepts[19].score | 0.0 |
| concepts[19].wikidata | https://www.wikidata.org/wiki/Q837863 |
| concepts[19].display_name | Pure mathematics |
| concepts[20].id | https://openalex.org/C33923547 |
| concepts[20].level | 0 |
| concepts[20].score | 0.0 |
| concepts[20].wikidata | https://www.wikidata.org/wiki/Q395 |
| concepts[20].display_name | Mathematics |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.7795356512069702 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/fine-tuning |
| keywords[1].score | 0.6921828985214233 |
| keywords[1].display_name | Fine-tuning |
| keywords[2].id | https://openalex.org/keywords/boosting |
| keywords[2].score | 0.631691038608551 |
| keywords[2].display_name | Boosting (machine learning) |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.6146757006645203 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/noisy-data |
| keywords[4].score | 0.5651628375053406 |
| keywords[4].display_name | Noisy data |
| keywords[5].id | https://openalex.org/keywords/noise |
| keywords[5].score | 0.5247949361801147 |
| keywords[5].display_name | Noise (video) |
| keywords[6].id | https://openalex.org/keywords/process |
| keywords[6].score | 0.5190216302871704 |
| keywords[6].display_name | Process (computing) |
| keywords[7].id | https://openalex.org/keywords/machine-learning |
| keywords[7].score | 0.4693494737148285 |
| keywords[7].display_name | Machine learning |
| keywords[8].id | https://openalex.org/keywords/field |
| keywords[8].score | 0.44576019048690796 |
| keywords[8].display_name | Field (mathematics) |
| keywords[9].id | https://openalex.org/keywords/language-model |
| keywords[9].score | 0.42698436975479126 |
| keywords[9].display_name | Language model |
| keywords[10].id | https://openalex.org/keywords/annotation |
| keywords[10].score | 0.4168662130832672 |
| keywords[10].display_name | Annotation |
| keywords[11].id | https://openalex.org/keywords/robustness |
| keywords[11].score | 0.41627585887908936 |
| keywords[11].display_name | Robustness (evolution) |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2311.01108 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2311.01108 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2311.01108 |
| locations[1].id | doi:10.48550/arxiv.2311.01108 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2311.01108 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5082259804 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-4152-5295 |
| authorships[0].author.display_name | Song Wang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Wang, Song |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5060371329 |
| authorships[1].author.orcid | https://orcid.org/0000-0001-8643-4683 |
| authorships[1].author.display_name | Zhen Tan |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Tan, Zhen |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5054719216 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-8522-6142 |
| authorships[2].author.display_name | Ruocheng Guo |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Guo, Ruocheng |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5029588473 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-1878-817X |
| authorships[3].author.display_name | Jundong Li |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Li, Jundong |
| authorships[3].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2311.01108 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2023-11-04T00:00:00 |
| display_name | Noise-Robust Fine-Tuning of Pretrained Language Models via External Guidance |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10181 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9983999729156494 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Natural Language Processing Techniques |
| related_works | https://openalex.org/W3037551068, https://openalex.org/W3023285645, https://openalex.org/W3023594376, https://openalex.org/W4287802662, https://openalex.org/W2474028989, https://openalex.org/W1995196013, https://openalex.org/W3154646238, https://openalex.org/W4388329095, https://openalex.org/W2899852118, https://openalex.org/W4309877123 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2311.01108 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2311.01108 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2311.01108 |
| primary_location.id | pmh:oai:arXiv.org:2311.01108 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2311.01108 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2311.01108 |
| publication_date | 2023-11-02 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 1 |
| abstract_inverted_index.To | 52 |
| abstract_inverted_index.an | 57 |
| abstract_inverted_index.by | 7 |
| abstract_inverted_index.in | 17, 25, 80 |
| abstract_inverted_index.it | 40 |
| abstract_inverted_index.of | 4, 20, 70, 117 |
| abstract_inverted_index.on | 106 |
| abstract_inverted_index.to | 34, 42 |
| abstract_inverted_index.we | 55 |
| abstract_inverted_index.and | 85, 88, 108 |
| abstract_inverted_index.are | 30 |
| abstract_inverted_index.due | 33 |
| abstract_inverted_index.for | 45, 60 |
| abstract_inverted_index.our | 118 |
| abstract_inverted_index.the | 18, 35, 68, 93, 98, 114, 121 |
| abstract_inverted_index.PLMs | 47, 62 |
| abstract_inverted_index.This | 77 |
| abstract_inverted_index.data | 28 |
| abstract_inverted_index.end, | 54 |
| abstract_inverted_index.have | 13 |
| abstract_inverted_index.like | 75 |
| abstract_inverted_index.over | 120 |
| abstract_inverted_index.such | 49 |
| abstract_inverted_index.this | 53 |
| abstract_inverted_index.with | 48 |
| abstract_inverted_index.Large | 71 |
| abstract_inverted_index.PLMs. | 103 |
| abstract_inverted_index.clean | 84 |
| abstract_inverted_index.field | 19 |
| abstract_inverted_index.noisy | 32, 50, 64, 86, 94, 110 |
| abstract_inverted_index.often | 31 |
| abstract_inverted_index.using | 63 |
| abstract_inverted_index.which | 66 |
| abstract_inverted_index.(LLMs) | 74 |
| abstract_inverted_index.(PLMs) | 12 |
| abstract_inverted_index.Models | 11, 73 |
| abstract_inverted_index.beyond | 92 |
| abstract_inverted_index.during | 101 |
| abstract_inverted_index.labels | 29 |
| abstract_inverted_index.making | 39 |
| abstract_inverted_index.assists | 79 |
| abstract_inverted_index.between | 83 |
| abstract_inverted_index.complex | 36 |
| abstract_inverted_index.develop | 43 |
| abstract_inverted_index.further | 112 |
| abstract_inverted_index.labels, | 65, 95 |
| abstract_inverted_index.labels. | 51 |
| abstract_inverted_index.natural | 21 |
| abstract_inverted_index.process | 100 |
| abstract_inverted_index.samples | 87 |
| abstract_inverted_index.thereby | 96 |
| abstract_inverted_index.Adopting | 0 |
| abstract_inverted_index.ChatGPT. | 76 |
| abstract_inverted_index.However, | 24 |
| abstract_inverted_index.Language | 10, 72 |
| abstract_inverted_index.achieved | 14 |
| abstract_inverted_index.approach | 59 |
| abstract_inverted_index.boosting | 97 |
| abstract_inverted_index.datasets | 111 |
| abstract_inverted_index.followed | 6 |
| abstract_inverted_index.guidance | 69, 78 |
| abstract_inverted_index.language | 22 |
| abstract_inverted_index.learning | 99 |
| abstract_inverted_index.paradigm | 3 |
| abstract_inverted_index.process, | 38 |
| abstract_inverted_index.provides | 89 |
| abstract_inverted_index.superior | 115 |
| abstract_inverted_index.Extensive | 104 |
| abstract_inverted_index.essential | 41 |
| abstract_inverted_index.framework | 119 |
| abstract_inverted_index.introduce | 56 |
| abstract_inverted_index.synthetic | 107 |
| abstract_inverted_index.two-stage | 2 |
| abstract_inverted_index.Pretrained | 9 |
| abstract_inverted_index.accurately | 81 |
| abstract_inverted_index.advantages | 116 |
| abstract_inverted_index.annotation | 37 |
| abstract_inverted_index.baselines. | 123 |
| abstract_inverted_index.innovative | 58 |
| abstract_inverted_index.real-world | 26, 109 |
| abstract_inverted_index.scenarios, | 27 |
| abstract_inverted_index.strategies | 44 |
| abstract_inverted_index.demonstrate | 113 |
| abstract_inverted_index.experiments | 105 |
| abstract_inverted_index.fine-tuning | 46, 61, 102 |
| abstract_inverted_index.information | 91 |
| abstract_inverted_index.pretraining | 5 |
| abstract_inverted_index.processing. | 23 |
| abstract_inverted_index.substantial | 15 |
| abstract_inverted_index.advancements | 16 |
| abstract_inverted_index.fine-tuning, | 8 |
| abstract_inverted_index.incorporates | 67 |
| abstract_inverted_index.supplementary | 90 |
| abstract_inverted_index.distinguishing | 82 |
| abstract_inverted_index.state-of-the-art | 122 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.7599999904632568 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile |