Improving Label Error Detection and Elimination with Uncertainty Quantification Article Swipe

PDF

Johannes Jakubik , Michael Vössing , Manil Maskey , Christopher Wölfle , Gerhard Satzger ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2405.09602

Identifying and handling label errors can significantly enhance the accuracy of supervised machine learning models. Recent approaches for identifying label errors demonstrate that a low self-confidence of models with respect to a certain label represents a good indicator of an erroneous label. However, latest work has built on softmax probabilities to measure self-confidence. In this paper, we argue that -- as softmax probabilities do not reflect a model's predictive uncertainty accurately -- label error detection requires more sophisticated measures of model uncertainty. Therefore, we develop a range of novel, model-agnostic algorithms for Uncertainty Quantification-Based Label Error Detection (UQ-LED), which combine the techniques of confident learning (CL), Monte Carlo Dropout (MCD), model uncertainty measures (e.g., entropy), and ensemble learning to enhance label error detection. We comprehensively evaluate our algorithms on four image classification benchmark datasets in two stages. In the first stage, we demonstrate that our UQ-LED algorithms outperform state-of-the-art confident learning in identifying label errors. In the second stage, we show that removing all identified errors from the training data based on our approach results in higher accuracies than training on all available labeled data. Importantly, besides our contributions to the detection of label errors, we particularly propose a novel approach to generate realistic, class-dependent label errors synthetically. Overall, our study demonstrates that selectively cleaning datasets with UQ-LED algorithms leads to more accurate classifications than using larger, noisier datasets.

Related Topics

Computer Science

Concepts

Computer science

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2405.09602
PDF: https://arxiv.org/pdf/2405.09602
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4397028119

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4397028119

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2405.09602

Digital Object Identifier
Title: Improving Label Error Detection and Elimination with Uncertainty Quantification

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-05-15

Full publication date if available
Authors: Johannes Jakubik, Michael Vössing, Manil Maskey, Christopher Wölfle, Gerhard Satzger

List of authors in order
Landing page: https://arxiv.org/abs/2405.09602

Publisher landing page
PDF URL: https://arxiv.org/pdf/2405.09602

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2405.09602

Direct OA link when available
Concepts: Computer science

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4397028119
doi	https://doi.org/10.48550/arxiv.2405.09602
ids.doi	https://doi.org/10.48550/arxiv.2405.09602
ids.openalex	https://openalex.org/W4397028119
fwci
type	preprint
title	Improving Label Error Detection and Elimination with Uncertainty Quantification
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10876
topics[0].field.id	https://openalex.org/fields/22
topics[0].field.display_name	Engineering
topics[0].score	0.9218000173568726
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/2207
topics[0].subfield.display_name	Control and Systems Engineering
topics[0].display_name	Fault Detection and Control Systems
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C41008148
concepts[0].level	0
concepts[0].score	0.4816388189792633
concepts[0].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[0].display_name	Computer science
keywords[0].id	https://openalex.org/keywords/computer-science
keywords[0].score	0.4816388189792633
keywords[0].display_name	Computer science
language	en
locations[0].id	pmh:oai:arXiv.org:2405.09602
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2405.09602
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2405.09602
locations[1].id	doi:10.48550/arxiv.2405.09602
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2405.09602
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5014669175
authorships[0].author.orcid	https://orcid.org/0000-0002-6235-0300
authorships[0].author.display_name	Johannes Jakubik
authorships[0].author_position	first
authorships[0].raw_author_name	Jakubik, Johannes
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5039548036
authorships[1].author.orcid	https://orcid.org/0000-0002-7722-6142
authorships[1].author.display_name	Michael Vössing
authorships[1].author_position	middle
authorships[1].raw_author_name	Vössing, Michael
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5002530990
authorships[2].author.orcid	https://orcid.org/0000-0002-5087-6903
authorships[2].author.display_name	Manil Maskey
authorships[2].author_position	middle
authorships[2].raw_author_name	Maskey, Manil
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5098684015
authorships[3].author.orcid
authorships[3].author.display_name	Christopher Wölfle
authorships[3].author_position	middle
authorships[3].raw_author_name	Wölfle, Christopher
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5000245557
authorships[4].author.orcid	https://orcid.org/0000-0001-8731-654X
authorships[4].author.display_name	Gerhard Satzger
authorships[4].author_position	last
authorships[4].raw_author_name	Satzger, Gerhard
authorships[4].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2405.09602
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2024-05-18T00:00:00
display_name	Improving Label Error Detection and Elimination with Uncertainty Quantification
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10876
primary_topic.field.id	https://openalex.org/fields/22
primary_topic.field.display_name	Engineering
primary_topic.score	0.9218000173568726
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/2207
primary_topic.subfield.display_name	Control and Systems Engineering
primary_topic.display_name	Fault Detection and Control Systems
related_works	https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052, https://openalex.org/W2382290278, https://openalex.org/W4395014643
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2405.09602
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2405.09602
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2405.09602
primary_location.id	pmh:oai:arXiv.org:2405.09602
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2405.09602
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2405.09602
publication_date	2024-05-15
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	23, 31, 35, 66, 85, 198
abstract_inverted_index.--	59, 71
abstract_inverted_index.In	53, 137, 155
abstract_inverted_index.We	123
abstract_inverted_index.an	39
abstract_inverted_index.as	60
abstract_inverted_index.do	63
abstract_inverted_index.in	134, 151, 175
abstract_inverted_index.of	10, 26, 38, 79, 87, 102, 192
abstract_inverted_index.on	47, 128, 171, 180
abstract_inverted_index.to	30, 50, 118, 189, 201, 220
abstract_inverted_index.we	56, 83, 141, 159, 195
abstract_inverted_index.all	163, 181
abstract_inverted_index.and	1, 115
abstract_inverted_index.can	5
abstract_inverted_index.for	17, 91
abstract_inverted_index.has	45
abstract_inverted_index.low	24
abstract_inverted_index.not	64
abstract_inverted_index.our	126, 144, 172, 187, 209
abstract_inverted_index.the	8, 100, 138, 156, 167, 190
abstract_inverted_index.two	135
abstract_inverted_index.data	169
abstract_inverted_index.four	129
abstract_inverted_index.from	166
abstract_inverted_index.good	36
abstract_inverted_index.more	76, 221
abstract_inverted_index.show	160
abstract_inverted_index.than	178, 224
abstract_inverted_index.that	22, 58, 143, 161, 212
abstract_inverted_index.this	54
abstract_inverted_index.with	28, 216
abstract_inverted_index.work	44
abstract_inverted_index.(CL),	105
abstract_inverted_index.Carlo	107
abstract_inverted_index.Error	95
abstract_inverted_index.Label	94
abstract_inverted_index.Monte	106
abstract_inverted_index.argue	57
abstract_inverted_index.based	170
abstract_inverted_index.built	46
abstract_inverted_index.data.	184
abstract_inverted_index.error	73, 121
abstract_inverted_index.first	139
abstract_inverted_index.image	130
abstract_inverted_index.label	3, 19, 33, 72, 120, 153, 193, 205
abstract_inverted_index.leads	219
abstract_inverted_index.model	80, 110
abstract_inverted_index.novel	199
abstract_inverted_index.range	86
abstract_inverted_index.study	210
abstract_inverted_index.using	225
abstract_inverted_index.which	98
abstract_inverted_index.(MCD),	109
abstract_inverted_index.(e.g.,	113
abstract_inverted_index.Recent	15
abstract_inverted_index.UQ-LED	145, 217
abstract_inverted_index.errors	4, 20, 165, 206
abstract_inverted_index.higher	176
abstract_inverted_index.label.	41
abstract_inverted_index.latest	43
abstract_inverted_index.models	27
abstract_inverted_index.novel,	88
abstract_inverted_index.paper,	55
abstract_inverted_index.second	157
abstract_inverted_index.stage,	140, 158
abstract_inverted_index.Dropout	108
abstract_inverted_index.besides	186
abstract_inverted_index.certain	32
abstract_inverted_index.combine	99
abstract_inverted_index.develop	84
abstract_inverted_index.enhance	7, 119
abstract_inverted_index.errors,	194
abstract_inverted_index.errors.	154
abstract_inverted_index.labeled	183
abstract_inverted_index.larger,	226
abstract_inverted_index.machine	12
abstract_inverted_index.measure	51
abstract_inverted_index.model's	67
abstract_inverted_index.models.	14
abstract_inverted_index.noisier	227
abstract_inverted_index.propose	197
abstract_inverted_index.reflect	65
abstract_inverted_index.respect	29
abstract_inverted_index.results	174
abstract_inverted_index.softmax	48, 61
abstract_inverted_index.stages.	136
abstract_inverted_index.However,	42
abstract_inverted_index.Overall,	208
abstract_inverted_index.accuracy	9
abstract_inverted_index.accurate	222
abstract_inverted_index.approach	173, 200
abstract_inverted_index.cleaning	214
abstract_inverted_index.datasets	133, 215
abstract_inverted_index.ensemble	116
abstract_inverted_index.evaluate	125
abstract_inverted_index.generate	202
abstract_inverted_index.handling	2
abstract_inverted_index.learning	13, 104, 117, 150
abstract_inverted_index.measures	78, 112
abstract_inverted_index.removing	162
abstract_inverted_index.requires	75
abstract_inverted_index.training	168, 179
abstract_inverted_index.(UQ-LED),	97
abstract_inverted_index.Detection	96
abstract_inverted_index.available	182
abstract_inverted_index.benchmark	132
abstract_inverted_index.confident	103, 149
abstract_inverted_index.datasets.	228
abstract_inverted_index.detection	74, 191
abstract_inverted_index.entropy),	114
abstract_inverted_index.erroneous	40
abstract_inverted_index.indicator	37
abstract_inverted_index.Therefore,	82
abstract_inverted_index.accuracies	177
abstract_inverted_index.accurately	70
abstract_inverted_index.algorithms	90, 127, 146, 218
abstract_inverted_index.approaches	16
abstract_inverted_index.detection.	122
abstract_inverted_index.identified	164
abstract_inverted_index.outperform	147
abstract_inverted_index.predictive	68
abstract_inverted_index.realistic,	203
abstract_inverted_index.represents	34
abstract_inverted_index.supervised	11
abstract_inverted_index.techniques	101
abstract_inverted_index.Identifying	0
abstract_inverted_index.Uncertainty	92
abstract_inverted_index.demonstrate	21, 142
abstract_inverted_index.identifying	18, 152
abstract_inverted_index.selectively	213
abstract_inverted_index.uncertainty	69, 111
abstract_inverted_index.Importantly,	185
abstract_inverted_index.demonstrates	211
abstract_inverted_index.particularly	196
abstract_inverted_index.uncertainty.	81
abstract_inverted_index.contributions	188
abstract_inverted_index.probabilities	49, 62
abstract_inverted_index.significantly	6
abstract_inverted_index.sophisticated	77
abstract_inverted_index.classification	131
abstract_inverted_index.model-agnostic	89
abstract_inverted_index.synthetically.	207
abstract_inverted_index.class-dependent	204
abstract_inverted_index.classifications	223
abstract_inverted_index.comprehensively	124
abstract_inverted_index.self-confidence	25
abstract_inverted_index.self-confidence.	52
abstract_inverted_index.state-of-the-art	148
abstract_inverted_index.Quantification-Based	93
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	5
citation_normalized_percentile