Designing Informative Metrics for Few-Shot Example Selection Article Swipe

PDF

Rishabh Adiga , Lakshminarayanan Subramanian , Varun Chandrasekaran ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2403.03861

Pretrained language models (PLMs) have shown remarkable few-shot learning capabilities when provided with properly formatted examples. However, selecting the "best" examples remains an open challenge. We propose a complexity-based prompt selection approach for sequence tagging tasks. This approach avoids the training of a dedicated model for selection of examples, and instead uses certain metrics to align the syntactico-semantic complexity of test sentences and examples. We use both sentence- and word-level metrics to match the complexity of examples to the (test) sentence being considered. Our results demonstrate that our approach extracts greater performance from PLMs: it achieves state-of-the-art performance on few-shot NER, achieving a 5% absolute improvement in F1 score on the CoNLL2003 dataset for GPT-4. We also see large gains of upto 28.85 points (F1/Acc.) in smaller models like GPT-j-6B.

Related Topics

Shot (Pellet)

Computer Science

Artificial Intelligence

Machine Learning

Chemistry

Organic Chemistry

Concepts

Selection (genetic algorithm) Shot (pellet) Computer science Artificial intelligence Machine learning Chemistry Organic chemistry

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2403.03861
PDF: https://arxiv.org/pdf/2403.03861
OA Status: green
Cited By: 1
Related Works: 10
OpenAlex ID: https://openalex.org/W4392576613

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4392576613

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2403.03861

Digital Object Identifier
Title: Designing Informative Metrics for Few-Shot Example Selection

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-03-06

Full publication date if available
Authors: Rishabh Adiga, Lakshminarayanan Subramanian, Varun Chandrasekaran

List of authors in order
Landing page: https://arxiv.org/abs/2403.03861

Publisher landing page
PDF URL: https://arxiv.org/pdf/2403.03861

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2403.03861

Direct OA link when available
Concepts: Selection (genetic algorithm), Shot (pellet), Computer science, Artificial intelligence, Machine learning, Chemistry, Organic chemistry

Top concepts (fields/topics) attached by OpenAlex
Cited by: 1

Total citation count in OpenAlex
Citations by year (recent): 2024: 1

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4392576613
doi	https://doi.org/10.48550/arxiv.2403.03861
ids.doi	https://doi.org/10.48550/arxiv.2403.03861
ids.openalex	https://openalex.org/W4392576613
fwci
type	preprint
title	Designing Informative Metrics for Few-Shot Example Selection
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10848
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.5443000197410583
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1703
topics[0].subfield.display_name	Computational Theory and Mathematics
topics[0].display_name	Advanced Multi-Objective Optimization Algorithms
topics[1].id	https://openalex.org/T11159
topics[1].field.id	https://openalex.org/fields/22
topics[1].field.display_name	Engineering
topics[1].score	0.5349000096321106
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/2209
topics[1].subfield.display_name	Industrial and Manufacturing Engineering
topics[1].display_name	Manufacturing Process and Optimization
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C81917197
concepts[0].level	2
concepts[0].score	0.7790572047233582
concepts[0].wikidata	https://www.wikidata.org/wiki/Q628760
concepts[0].display_name	Selection (genetic algorithm)
concepts[1].id	https://openalex.org/C2778344882
concepts[1].level	2
concepts[1].score	0.6434872150421143
concepts[1].wikidata	https://www.wikidata.org/wiki/Q278938
concepts[1].display_name	Shot (pellet)
concepts[2].id	https://openalex.org/C41008148
concepts[2].level	0
concepts[2].score	0.5281712412834167
concepts[2].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[2].display_name	Computer science
concepts[3].id	https://openalex.org/C154945302
concepts[3].level	1
concepts[3].score	0.4602922201156616
concepts[3].wikidata	https://www.wikidata.org/wiki/Q11660
concepts[3].display_name	Artificial intelligence
concepts[4].id	https://openalex.org/C119857082
concepts[4].level	1
concepts[4].score	0.39453649520874023
concepts[4].wikidata	https://www.wikidata.org/wiki/Q2539
concepts[4].display_name	Machine learning
concepts[5].id	https://openalex.org/C185592680
concepts[5].level	0
concepts[5].score	0.09643593430519104
concepts[5].wikidata	https://www.wikidata.org/wiki/Q2329
concepts[5].display_name	Chemistry
concepts[6].id	https://openalex.org/C178790620
concepts[6].level	1
concepts[6].score	0.0
concepts[6].wikidata	https://www.wikidata.org/wiki/Q11351
concepts[6].display_name	Organic chemistry
keywords[0].id	https://openalex.org/keywords/selection
keywords[0].score	0.7790572047233582
keywords[0].display_name	Selection (genetic algorithm)
keywords[1].id	https://openalex.org/keywords/shot
keywords[1].score	0.6434872150421143
keywords[1].display_name	Shot (pellet)
keywords[2].id	https://openalex.org/keywords/computer-science
keywords[2].score	0.5281712412834167
keywords[2].display_name	Computer science
keywords[3].id	https://openalex.org/keywords/artificial-intelligence
keywords[3].score	0.4602922201156616
keywords[3].display_name	Artificial intelligence
keywords[4].id	https://openalex.org/keywords/machine-learning
keywords[4].score	0.39453649520874023
keywords[4].display_name	Machine learning
keywords[5].id	https://openalex.org/keywords/chemistry
keywords[5].score	0.09643593430519104
keywords[5].display_name	Chemistry
language	en
locations[0].id	pmh:oai:arXiv.org:2403.03861
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2403.03861
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2403.03861
locations[1].id	doi:10.48550/arxiv.2403.03861
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2403.03861
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5094095396
authorships[0].author.orcid
authorships[0].author.display_name	Rishabh Adiga
authorships[0].author_position	first
authorships[0].raw_author_name	Adiga, Rishabh
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5072232894
authorships[1].author.orcid	https://orcid.org/0000-0001-8101-1243
authorships[1].author.display_name	Lakshminarayanan Subramanian
authorships[1].author_position	middle
authorships[1].raw_author_name	Subramanian, Lakshminarayanan
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5009092703
authorships[2].author.orcid	https://orcid.org/0000-0003-0661-6378
authorships[2].author.display_name	Varun Chandrasekaran
authorships[2].author_position	last
authorships[2].raw_author_name	Chandrasekaran, Varun
authorships[2].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2403.03861
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	Designing Informative Metrics for Few-Shot Example Selection
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10848
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.5443000197410583
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1703
primary_topic.subfield.display_name	Computational Theory and Mathematics
primary_topic.display_name	Advanced Multi-Objective Optimization Algorithms
related_works	https://openalex.org/W2961085424, https://openalex.org/W4306674287, https://openalex.org/W3046775127, https://openalex.org/W3107602296, https://openalex.org/W3170094116, https://openalex.org/W4386462264, https://openalex.org/W4364306694, https://openalex.org/W4312192474, https://openalex.org/W4283697347, https://openalex.org/W4210805261
cited_by_count	1
counts_by_year[0].year	2024
counts_by_year[0].cited_by_count	1
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2403.03861
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2403.03861
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2403.03861
primary_location.id	pmh:oai:arXiv.org:2403.03861
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2403.03861
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2403.03861
publication_date	2024-03-06
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	27, 42, 102
abstract_inverted_index.5%	103
abstract_inverted_index.F1	107
abstract_inverted_index.We	25, 64, 115
abstract_inverted_index.an	22
abstract_inverted_index.in	106, 125
abstract_inverted_index.it	94
abstract_inverted_index.of	41, 47, 59, 75, 120
abstract_inverted_index.on	98, 109
abstract_inverted_index.to	54, 71, 77
abstract_inverted_index.Our	83
abstract_inverted_index.and	49, 62, 68
abstract_inverted_index.for	32, 45, 113
abstract_inverted_index.our	87
abstract_inverted_index.see	117
abstract_inverted_index.the	18, 39, 56, 73, 78, 110
abstract_inverted_index.use	65
abstract_inverted_index.NER,	100
abstract_inverted_index.This	36
abstract_inverted_index.also	116
abstract_inverted_index.both	66
abstract_inverted_index.from	92
abstract_inverted_index.have	4
abstract_inverted_index.like	128
abstract_inverted_index.open	23
abstract_inverted_index.test	60
abstract_inverted_index.that	86
abstract_inverted_index.upto	121
abstract_inverted_index.uses	51
abstract_inverted_index.when	10
abstract_inverted_index.with	12
abstract_inverted_index.28.85	122
abstract_inverted_index.PLMs:	93
abstract_inverted_index.align	55
abstract_inverted_index.being	81
abstract_inverted_index.gains	119
abstract_inverted_index.large	118
abstract_inverted_index.match	72
abstract_inverted_index.model	44
abstract_inverted_index.score	108
abstract_inverted_index.shown	5
abstract_inverted_index."best"	19
abstract_inverted_index.(PLMs)	3
abstract_inverted_index.(test)	79
abstract_inverted_index.GPT-4.	114
abstract_inverted_index.avoids	38
abstract_inverted_index.models	2, 127
abstract_inverted_index.points	123
abstract_inverted_index.prompt	29
abstract_inverted_index.tasks.	35
abstract_inverted_index.certain	52
abstract_inverted_index.dataset	112
abstract_inverted_index.greater	90
abstract_inverted_index.instead	50
abstract_inverted_index.metrics	53, 70
abstract_inverted_index.propose	26
abstract_inverted_index.remains	21
abstract_inverted_index.results	84
abstract_inverted_index.smaller	126
abstract_inverted_index.tagging	34
abstract_inverted_index.However,	16
abstract_inverted_index.absolute	104
abstract_inverted_index.achieves	95
abstract_inverted_index.approach	31, 37, 88
abstract_inverted_index.examples	20, 76
abstract_inverted_index.extracts	89
abstract_inverted_index.few-shot	7, 99
abstract_inverted_index.language	1
abstract_inverted_index.learning	8
abstract_inverted_index.properly	13
abstract_inverted_index.provided	11
abstract_inverted_index.sentence	80
abstract_inverted_index.sequence	33
abstract_inverted_index.training	40
abstract_inverted_index.(F1/Acc.)	124
abstract_inverted_index.CoNLL2003	111
abstract_inverted_index.GPT-j-6B.	129
abstract_inverted_index.achieving	101
abstract_inverted_index.dedicated	43
abstract_inverted_index.examples,	48
abstract_inverted_index.examples.	15, 63
abstract_inverted_index.formatted	14
abstract_inverted_index.selecting	17
abstract_inverted_index.selection	30, 46
abstract_inverted_index.sentence-	67
abstract_inverted_index.sentences	61
abstract_inverted_index.Pretrained	0
abstract_inverted_index.challenge.	24
abstract_inverted_index.complexity	58, 74
abstract_inverted_index.remarkable	6
abstract_inverted_index.word-level	69
abstract_inverted_index.considered.	82
abstract_inverted_index.demonstrate	85
abstract_inverted_index.improvement	105
abstract_inverted_index.performance	91, 97
abstract_inverted_index.capabilities	9
abstract_inverted_index.complexity-based	28
abstract_inverted_index.state-of-the-art	96
abstract_inverted_index.syntactico-semantic	57
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	3
citation_normalized_percentile