Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop Article Swipe

PDF

Ekaterina Artemova , Akim Tsvigun , Dominik Schlechtweg , Natalia Fedorova , Sergei Tilga , Boris Obmoroshev ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2411.04637

Training and deploying machine learning models relies on a large amount of human-annotated data. As human labeling becomes increasingly expensive and time-consuming, recent research has developed multiple strategies to speed up annotation and reduce costs and human workload: generating synthetic training data, active learning, and hybrid labeling. This tutorial is oriented toward practical applications: we will present the basics of each strategy, highlight their benefits and limitations, and discuss in detail real-life case studies. Additionally, we will walk through best practices for managing human annotators and controlling the quality of the final dataset. The tutorial includes a hands-on workshop, where attendees will be guided in implementing a hybrid annotation setup. This tutorial is designed for NLP practitioners from both research and industry backgrounds who are involved in or interested in optimizing data labeling projects.

Related Topics

Computer Science

Artificial Intelligence

Mathematics

Combinatorics

Concepts

Loop (graph theory) Human-in-the-loop Computer science Artificial intelligence Mathematics Combinatorics

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2411.04637
PDF: https://arxiv.org/pdf/2411.04637
OA Status: green
Cited By: 1
Related Works: 10
OpenAlex ID: https://openalex.org/W4404396960

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4404396960

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2411.04637

Digital Object Identifier
Title: Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-11-07

Full publication date if available
Authors: Ekaterina Artemova, Akim Tsvigun, Dominik Schlechtweg, Natalia Fedorova, Sergei Tilga, Boris Obmoroshev

List of authors in order
Landing page: https://arxiv.org/abs/2411.04637

Publisher landing page
PDF URL: https://arxiv.org/pdf/2411.04637

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2411.04637

Direct OA link when available
Concepts: Loop (graph theory), Human-in-the-loop, Computer science, Artificial intelligence, Mathematics, Combinatorics

Top concepts (fields/topics) attached by OpenAlex
Cited by: 1

Total citation count in OpenAlex
Citations by year (recent): 2025: 1

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4404396960
doi	https://doi.org/10.48550/arxiv.2411.04637
ids.doi	https://doi.org/10.48550/arxiv.2411.04637
ids.openalex	https://openalex.org/W4404396960
fwci
type	preprint
title	Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10181
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9675999879837036
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Natural Language Processing Techniques
topics[1].id	https://openalex.org/T13643
topics[1].field.id	https://openalex.org/fields/33
topics[1].field.display_name	Social Sciences
topics[1].score	0.9358000159263611
topics[1].domain.id	https://openalex.org/domains/2
topics[1].domain.display_name	Social Sciences
topics[1].subfield.id	https://openalex.org/subfields/3320
topics[1].subfield.display_name	Political Science and International Relations
topics[1].display_name	Artificial Intelligence in Law
topics[2].id	https://openalex.org/T13999
topics[2].field.id	https://openalex.org/fields/17
topics[2].field.display_name	Computer Science
topics[2].score	0.9121000170707703
topics[2].domain.id	https://openalex.org/domains/3
topics[2].domain.display_name	Physical Sciences
topics[2].subfield.id	https://openalex.org/subfields/1710
topics[2].subfield.display_name	Information Systems
topics[2].display_name	Digital Rights Management and Security
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C184670325
concepts[0].level	2
concepts[0].score	0.6043068170547485
concepts[0].wikidata	https://www.wikidata.org/wiki/Q512604
concepts[0].display_name	Loop (graph theory)
concepts[1].id	https://openalex.org/C2780626000
concepts[1].level	2
concepts[1].score	0.5263468027114868
concepts[1].wikidata	https://www.wikidata.org/wiki/Q5936775
concepts[1].display_name	Human-in-the-loop
concepts[2].id	https://openalex.org/C41008148
concepts[2].level	0
concepts[2].score	0.48692095279693604
concepts[2].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[2].display_name	Computer science
concepts[3].id	https://openalex.org/C154945302
concepts[3].level	1
concepts[3].score	0.30164623260498047
concepts[3].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[3].display_name	Artificial intelligence
concepts[4].id	https://openalex.org/C33923547
concepts[4].level	0
concepts[4].score	0.13379907608032227
concepts[4].wikidata	https://www.wikidata.org/wiki/Q395
concepts[4].display_name	Mathematics
concepts[5].id	https://openalex.org/C114614502
concepts[5].level	1
concepts[5].score	0.0
concepts[5].wikidata	https://www.wikidata.org/wiki/Q76592
concepts[5].display_name	Combinatorics
keywords[0].id	https://openalex.org/keywords/loop
keywords[0].score	0.6043068170547485
keywords[0].display_name	Loop (graph theory)
keywords[1].id	https://openalex.org/keywords/human-in-the-loop
keywords[1].score	0.5263468027114868
keywords[1].display_name	Human-in-the-loop
keywords[2].id	https://openalex.org/keywords/computer-science
keywords[2].score	0.48692095279693604
keywords[2].display_name	Computer science
keywords[3].id	https://openalex.org/keywords/artificial-intelligence
keywords[3].score	0.30164623260498047
keywords[3].display_name	Artificial intelligence
keywords[4].id	https://openalex.org/keywords/mathematics
keywords[4].score	0.13379907608032227
keywords[4].display_name	Mathematics
language	en
locations[0].id	pmh:oai:arXiv.org:2411.04637
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2411.04637
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2411.04637
locations[1].id	doi:10.48550/arxiv.2411.04637
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2411.04637
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5071539494
authorships[0].author.orcid
authorships[0].author.display_name	Ekaterina Artemova
authorships[0].author_position	first
authorships[0].raw_author_name	Artemova, Ekaterina
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5026319614
authorships[1].author.orcid
authorships[1].author.display_name	Akim Tsvigun
authorships[1].author_position	middle
authorships[1].raw_author_name	Tsvigun, Akim
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5013366042
authorships[2].author.orcid	https://orcid.org/0000-0002-0685-2576
authorships[2].author.display_name	Dominik Schlechtweg
authorships[2].author_position	middle
authorships[2].raw_author_name	Schlechtweg, Dominik
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5070486905
authorships[3].author.orcid	https://orcid.org/0000-0002-2085-8771
authorships[3].author.display_name	Natalia Fedorova
authorships[3].author_position	middle
authorships[3].raw_author_name	Fedorova, Natalia
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5114655777
authorships[4].author.orcid
authorships[4].author.display_name	Sergei Tilga
authorships[4].author_position	middle
authorships[4].raw_author_name	Tilga, Sergei
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5114655778
authorships[5].author.orcid
authorships[5].author.display_name	Boris Obmoroshev
authorships[5].author_position	last
authorships[5].raw_author_name	Obmoroshev, Boris
authorships[5].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2411.04637
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Hands-On Tutorial: Labeling with LLM and Human-in-the-Loop
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10181
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9675999879837036
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Natural Language Processing Techniques
related_works	https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W24774503, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W4288099294, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W4367173559
cited_by_count	1
counts_by_year[0].year	2025
counts_by_year[0].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2411.04637
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2411.04637
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2411.04637
primary_location.id	pmh:oai:arXiv.org:2411.04637
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2411.04637
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2411.04637
publication_date	2024-11-07
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	8, 96, 106
abstract_inverted_index.As	14
abstract_inverted_index.be	102
abstract_inverted_index.in	69, 104, 126, 129
abstract_inverted_index.is	49, 112
abstract_inverted_index.of	11, 59, 89
abstract_inverted_index.on	7
abstract_inverted_index.or	127
abstract_inverted_index.to	28
abstract_inverted_index.up	30
abstract_inverted_index.we	54, 75
abstract_inverted_index.NLP	115
abstract_inverted_index.The	93
abstract_inverted_index.and	1, 20, 32, 35, 44, 65, 67, 85, 120
abstract_inverted_index.are	124
abstract_inverted_index.for	81, 114
abstract_inverted_index.has	24
abstract_inverted_index.the	57, 87, 90
abstract_inverted_index.who	123
abstract_inverted_index.This	47, 110
abstract_inverted_index.best	79
abstract_inverted_index.both	118
abstract_inverted_index.case	72
abstract_inverted_index.data	131
abstract_inverted_index.each	60
abstract_inverted_index.from	117
abstract_inverted_index.walk	77
abstract_inverted_index.will	55, 76, 101
abstract_inverted_index.costs	34
abstract_inverted_index.data,	41
abstract_inverted_index.data.	13
abstract_inverted_index.final	91
abstract_inverted_index.human	15, 36, 83
abstract_inverted_index.large	9
abstract_inverted_index.speed	29
abstract_inverted_index.their	63
abstract_inverted_index.where	99
abstract_inverted_index.active	42
abstract_inverted_index.amount	10
abstract_inverted_index.basics	58
abstract_inverted_index.detail	70
abstract_inverted_index.guided	103
abstract_inverted_index.hybrid	45, 107
abstract_inverted_index.models	5
abstract_inverted_index.recent	22
abstract_inverted_index.reduce	33
abstract_inverted_index.relies	6
abstract_inverted_index.setup.	109
abstract_inverted_index.toward	51
abstract_inverted_index.becomes	17
abstract_inverted_index.discuss	68
abstract_inverted_index.machine	3
abstract_inverted_index.present	56
abstract_inverted_index.quality	88
abstract_inverted_index.through	78
abstract_inverted_index.Training	0
abstract_inverted_index.benefits	64
abstract_inverted_index.dataset.	92
abstract_inverted_index.designed	113
abstract_inverted_index.hands-on	97
abstract_inverted_index.includes	95
abstract_inverted_index.industry	121
abstract_inverted_index.involved	125
abstract_inverted_index.labeling	16, 132
abstract_inverted_index.learning	4
abstract_inverted_index.managing	82
abstract_inverted_index.multiple	26
abstract_inverted_index.oriented	50
abstract_inverted_index.research	23, 119
abstract_inverted_index.studies.	73
abstract_inverted_index.training	40
abstract_inverted_index.tutorial	48, 94, 111
abstract_inverted_index.attendees	100
abstract_inverted_index.deploying	2
abstract_inverted_index.developed	25
abstract_inverted_index.expensive	19
abstract_inverted_index.highlight	62
abstract_inverted_index.labeling.	46
abstract_inverted_index.learning,	43
abstract_inverted_index.practical	52
abstract_inverted_index.practices	80
abstract_inverted_index.projects.	133
abstract_inverted_index.real-life	71
abstract_inverted_index.strategy,	61
abstract_inverted_index.synthetic	39
abstract_inverted_index.workload:	37
abstract_inverted_index.workshop,	98
abstract_inverted_index.annotation	31, 108
abstract_inverted_index.annotators	84
abstract_inverted_index.generating	38
abstract_inverted_index.interested	128
abstract_inverted_index.optimizing	130
abstract_inverted_index.strategies	27
abstract_inverted_index.backgrounds	122
abstract_inverted_index.controlling	86
abstract_inverted_index.implementing	105
abstract_inverted_index.increasingly	18
abstract_inverted_index.limitations,	66
abstract_inverted_index.Additionally,	74
abstract_inverted_index.applications:	53
abstract_inverted_index.practitioners	116
abstract_inverted_index.human-annotated	12
abstract_inverted_index.time-consuming,	21
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	6
citation_normalized_percentile