In-Context Principle Learning from Mistakes Article Swipe

PDF

Tianjun Zhang , Aman Madaan , Luyu Gao , Steven Zheng , Swaroop Mishra , Yiming Yang , Niket Tandon , Uri Alon ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2402.05403

In-context learning (ICL, also known as few-shot prompting) has been the standard method of adapting LLMs to downstream tasks, by learning from a few input-output examples. Nonetheless, all ICL-based approaches only learn from correct input-output pairs. In this paper, we revisit this paradigm, by learning more from the few given input-output examples. We introduce Learning Principles (LEAP): First, we intentionally induce the model to make mistakes on these few examples; then we reflect on these mistakes, and learn explicit task-specific "principles" from them, which help solve similar problems and avoid common mistakes; finally, we prompt the model to answer unseen test questions using the original few-shot examples and these learned general principles. We evaluate LEAP on a wide range of benchmarks, including multi-hop question answering (Hotpot QA), textual QA (DROP), Big-Bench Hard reasoning, and math problems (GSM8K and MATH); in all these benchmarks, LEAP improves the strongest available LLMs such as GPT-3.5-turbo, GPT-4, GPT-4 turbo and Claude-2.1. For example, LEAP improves over the standard few-shot prompting using GPT-4 by 7.5% in DROP, and by 3.3% in HotpotQA. Importantly, LEAP does not require any more input or examples than the standard few-shot prompting settings.

Related Topics

Concepts

Context (archaeology) Psychology Cognitive science Computer science Epistemology Geography Philosophy Archaeology

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2402.05403
PDF: https://arxiv.org/pdf/2402.05403
OA Status: green
Cited By: 2
Related Works: 10
OpenAlex ID: https://openalex.org/W4391709567

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4391709567

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2402.05403

Digital Object Identifier
Title: In-Context Principle Learning from Mistakes

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-02-08

Full publication date if available
Authors: Tianjun Zhang, Aman Madaan, Luyu Gao, Steven Zheng, Swaroop Mishra, Yiming Yang, Niket Tandon, Uri Alon

List of authors in order
Landing page: https://arxiv.org/abs/2402.05403

Publisher landing page
PDF URL: https://arxiv.org/pdf/2402.05403

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2402.05403

Direct OA link when available
Concepts: Context (archaeology), Psychology, Cognitive science, Computer science, Epistemology, Geography, Philosophy, Archaeology

Top concepts (fields/topics) attached by OpenAlex
Cited by: 2

Total citation count in OpenAlex
Citations by year (recent): 2024: 2

Per-year citation counts (last 5 years)
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4391709567
doi	https://doi.org/10.48550/arxiv.2402.05403
ids.doi	https://doi.org/10.48550/arxiv.2402.05403
ids.openalex	https://openalex.org/W4391709567
fwci
type	preprint
title	In-Context Principle Learning from Mistakes
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10809
topics[0].field.id	https://openalex.org/fields/36
topics[0].field.display_name	Health Professions
topics[0].score	0.8011000156402588
topics[0].domain.id	https://openalex.org/domains/4
topics[0].domain.display_name	Health Sciences
topics[0].subfield.id	https://openalex.org/subfields/3614
topics[0].subfield.display_name	Radiological and Ultrasound Technology
topics[0].display_name	Occupational Health and Safety Research
topics[1].id	https://openalex.org/T13295
topics[1].field.id	https://openalex.org/fields/22
topics[1].field.display_name	Engineering
topics[1].score	0.7965999841690063
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/2213
topics[1].subfield.display_name	Safety, Risk, Reliability and Quality
topics[1].display_name	Safety Systems Engineering in Autonomy
topics[2].id	https://openalex.org/T11357
topics[2].field.id	https://openalex.org/fields/18
topics[2].field.display_name	Decision Sciences
topics[2].score	0.7692000269889832
topics[2].domain.id	https://openalex.org/domains/2
topics[2].domain.display_name	Social Sciences
topics[2].subfield.id	https://openalex.org/subfields/1804
topics[2].subfield.display_name	Statistics, Probability and Uncertainty
topics[2].display_name	Risk and Safety Analysis
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C2779343474
concepts[0].level	2
concepts[0].score	0.604851245880127
concepts[0].wikidata	https://www.wikidata.org/wiki/Q3109175
concepts[0].display_name	Context (archaeology)
concepts[1].id	https://openalex.org/C15744967
concepts[1].level	0
concepts[1].score	0.4268264174461365
concepts[1].wikidata	https://www.wikidata.org/wiki/Q9418
concepts[1].display_name	Psychology
concepts[2].id	https://openalex.org/C188147891
concepts[2].level	1
concepts[2].score	0.38393837213516235
concepts[2].wikidata	https://www.wikidata.org/wiki/Q147638
concepts[2].display_name	Cognitive science
concepts[3].id	https://openalex.org/C41008148
concepts[3].level	0
concepts[3].score	0.3508259057998657
concepts[3].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[3].display_name	Computer science
concepts[4].id	https://openalex.org/C111472728
concepts[4].level	1
concepts[4].score	0.32557186484336853
concepts[4].wikidata	https://www.wikidata.org/wiki/Q9471
concepts[4].display_name	Epistemology
concepts[5].id	https://openalex.org/C205649164
concepts[5].level	0
concepts[5].score	0.1824665069580078
concepts[5].wikidata	https://www.wikidata.org/wiki/Q1071
concepts[5].display_name	Geography
concepts[6].id	https://openalex.org/C138885662
concepts[6].level	0
concepts[6].score	0.1502012312412262
concepts[6].wikidata	https://www.wikidata.org/wiki/Q5891
concepts[6].display_name	Philosophy
concepts[7].id	https://openalex.org/C166957645
concepts[7].level	1
concepts[7].score	0.0
concepts[7].wikidata	https://www.wikidata.org/wiki/Q23498
concepts[7].display_name	Archaeology
keywords[0].id	https://openalex.org/keywords/context
keywords[0].score	0.604851245880127
keywords[0].display_name	Context (archaeology)
keywords[1].id	https://openalex.org/keywords/psychology
keywords[1].score	0.4268264174461365
keywords[1].display_name	Psychology
keywords[2].id	https://openalex.org/keywords/cognitive-science
keywords[2].score	0.38393837213516235
keywords[2].display_name	Cognitive science
keywords[3].id	https://openalex.org/keywords/computer-science
keywords[3].score	0.3508259057998657
keywords[3].display_name	Computer science
keywords[4].id	https://openalex.org/keywords/epistemology
keywords[4].score	0.32557186484336853
keywords[4].display_name	Epistemology
keywords[5].id	https://openalex.org/keywords/geography
keywords[5].score	0.1824665069580078
keywords[5].display_name	Geography
keywords[6].id	https://openalex.org/keywords/philosophy
keywords[6].score	0.1502012312412262
keywords[6].display_name	Philosophy
language	en
locations[0].id	pmh:oai:arXiv.org:2402.05403
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2402.05403
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2402.05403
locations[1].id	doi:10.48550/arxiv.2402.05403
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	cc-by
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/cc-by
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2402.05403
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5101571114
authorships[0].author.orcid	https://orcid.org/0000-0003-2059-0709
authorships[0].author.display_name	Tianjun Zhang
authorships[0].author_position	first
authorships[0].raw_author_name	Zhang, Tianjun
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5091998799
authorships[1].author.orcid
authorships[1].author.display_name	Aman Madaan
authorships[1].author_position	middle
authorships[1].raw_author_name	Madaan, Aman
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5060885692
authorships[2].author.orcid	https://orcid.org/0009-0006-5806-3022
authorships[2].author.display_name	Luyu Gao
authorships[2].author_position	middle
authorships[2].raw_author_name	Gao, Luyu
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5112919416
authorships[3].author.orcid
authorships[3].author.display_name	Steven Zheng
authorships[3].author_position	middle
authorships[3].raw_author_name	Zheng, Steven
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5063722751
authorships[4].author.orcid	https://orcid.org/0009-0001-6413-7001
authorships[4].author.display_name	Swaroop Mishra
authorships[4].author_position	middle
authorships[4].raw_author_name	Mishra, Swaroop
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5041434096
authorships[5].author.orcid	https://orcid.org/0000-0003-1359-0364
authorships[5].author.display_name	Yiming Yang
authorships[5].author_position	middle
authorships[5].raw_author_name	Yang, Yiming
authorships[5].is_corresponding	False
authorships[6].author.id	https://openalex.org/A5052882677
authorships[6].author.orcid
authorships[6].author.display_name	Niket Tandon
authorships[6].author_position	middle
authorships[6].raw_author_name	Tandon, Niket
authorships[6].is_corresponding	False
authorships[7].author.id	https://openalex.org/A5107243336
authorships[7].author.orcid	https://orcid.org/0000-0001-5796-9988
authorships[7].author.display_name	Uri Alon
authorships[7].author_position	last
authorships[7].raw_author_name	Alon, Uri
authorships[7].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2402.05403
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	In-Context Principle Learning from Mistakes
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10809
primary_topic.field.id	https://openalex.org/fields/36
primary_topic.field.display_name	Health Professions
primary_topic.score	0.8011000156402588
primary_topic.domain.id	https://openalex.org/domains/4
primary_topic.domain.display_name	Health Sciences
primary_topic.subfield.id	https://openalex.org/subfields/3614
primary_topic.subfield.display_name	Radiological and Ultrasound Technology
primary_topic.display_name	Occupational Health and Safety Research
related_works	https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2382290278, https://openalex.org/W2478288626, https://openalex.org/W2350741829, https://openalex.org/W2530322880, https://openalex.org/W1596801655
cited_by_count	2
counts_by_year[0].year	2024
counts_by_year[0].cited_by_count	2
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2402.05403
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2402.05403
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2402.05403
primary_location.id	pmh:oai:arXiv.org:2402.05403
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2402.05403
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2402.05403
publication_date	2024-02-08
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	22, 116
abstract_inverted_index.In	36
abstract_inverted_index.QA	128
abstract_inverted_index.We	52, 112
abstract_inverted_index.as	5, 150
abstract_inverted_index.by	19, 43, 168, 173
abstract_inverted_index.in	139, 170, 175
abstract_inverted_index.of	13, 119
abstract_inverted_index.on	66, 73, 115
abstract_inverted_index.or	185
abstract_inverted_index.to	16, 63, 97
abstract_inverted_index.we	39, 58, 71, 93
abstract_inverted_index.For	157
abstract_inverted_index.all	27, 140
abstract_inverted_index.and	76, 88, 107, 133, 137, 155, 172
abstract_inverted_index.any	182
abstract_inverted_index.few	23, 48, 68
abstract_inverted_index.has	8
abstract_inverted_index.not	180
abstract_inverted_index.the	10, 47, 61, 95, 103, 145, 162, 188
abstract_inverted_index.3.3%	174
abstract_inverted_index.7.5%	169
abstract_inverted_index.Hard	131
abstract_inverted_index.LEAP	114, 143, 159, 178
abstract_inverted_index.LLMs	15, 148
abstract_inverted_index.QA),	126
abstract_inverted_index.also	3
abstract_inverted_index.been	9
abstract_inverted_index.does	179
abstract_inverted_index.from	21, 32, 46, 81
abstract_inverted_index.help	84
abstract_inverted_index.make	64
abstract_inverted_index.math	134
abstract_inverted_index.more	45, 183
abstract_inverted_index.only	30
abstract_inverted_index.over	161
abstract_inverted_index.such	149
abstract_inverted_index.test	100
abstract_inverted_index.than	187
abstract_inverted_index.then	70
abstract_inverted_index.this	37, 41
abstract_inverted_index.wide	117
abstract_inverted_index.(ICL,	2
abstract_inverted_index.DROP,	171
abstract_inverted_index.GPT-4	153, 167
abstract_inverted_index.avoid	89
abstract_inverted_index.given	49
abstract_inverted_index.input	184
abstract_inverted_index.known	4
abstract_inverted_index.learn	31, 77
abstract_inverted_index.model	62, 96
abstract_inverted_index.range	118
abstract_inverted_index.solve	85
abstract_inverted_index.them,	82
abstract_inverted_index.these	67, 74, 108, 141
abstract_inverted_index.turbo	154
abstract_inverted_index.using	102, 166
abstract_inverted_index.which	83
abstract_inverted_index.(GSM8K	136
abstract_inverted_index.First,	57
abstract_inverted_index.GPT-4,	152
abstract_inverted_index.MATH);	138
abstract_inverted_index.answer	98
abstract_inverted_index.common	90
abstract_inverted_index.induce	60
abstract_inverted_index.method	12
abstract_inverted_index.pairs.	35
abstract_inverted_index.paper,	38
abstract_inverted_index.prompt	94
abstract_inverted_index.tasks,	18
abstract_inverted_index.unseen	99
abstract_inverted_index.(DROP),	129
abstract_inverted_index.(Hotpot	125
abstract_inverted_index.(LEAP):	56
abstract_inverted_index.correct	33
abstract_inverted_index.general	110
abstract_inverted_index.learned	109
abstract_inverted_index.reflect	72
abstract_inverted_index.require	181
abstract_inverted_index.revisit	40
abstract_inverted_index.similar	86
abstract_inverted_index.textual	127
abstract_inverted_index.Learning	54
abstract_inverted_index.adapting	14
abstract_inverted_index.evaluate	113
abstract_inverted_index.example,	158
abstract_inverted_index.examples	106, 186
abstract_inverted_index.explicit	78
abstract_inverted_index.few-shot	6, 105, 164, 190
abstract_inverted_index.finally,	92
abstract_inverted_index.improves	144, 160
abstract_inverted_index.learning	1, 20, 44
abstract_inverted_index.mistakes	65
abstract_inverted_index.original	104
abstract_inverted_index.problems	87, 135
abstract_inverted_index.question	123
abstract_inverted_index.standard	11, 163, 189
abstract_inverted_index.Big-Bench	130
abstract_inverted_index.HotpotQA.	176
abstract_inverted_index.ICL-based	28
abstract_inverted_index.answering	124
abstract_inverted_index.available	147
abstract_inverted_index.examples.	25, 51
abstract_inverted_index.examples;	69
abstract_inverted_index.including	121
abstract_inverted_index.introduce	53
abstract_inverted_index.mistakes,	75
abstract_inverted_index.mistakes;	91
abstract_inverted_index.multi-hop	122
abstract_inverted_index.paradigm,	42
abstract_inverted_index.prompting	165, 191
abstract_inverted_index.questions	101
abstract_inverted_index.settings.	192
abstract_inverted_index.strongest	146
abstract_inverted_index.In-context	0
abstract_inverted_index.Principles	55
abstract_inverted_index.approaches	29
abstract_inverted_index.downstream	17
abstract_inverted_index.prompting)	7
abstract_inverted_index.reasoning,	132
abstract_inverted_index.Claude-2.1.	156
abstract_inverted_index.benchmarks,	120, 142
abstract_inverted_index.principles.	111
abstract_inverted_index."principles"	80
abstract_inverted_index.Importantly,	177
abstract_inverted_index.Nonetheless,	26
abstract_inverted_index.input-output	24, 34, 50
abstract_inverted_index.intentionally	59
abstract_inverted_index.task-specific	79
abstract_inverted_index.GPT-3.5-turbo,	151
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	8
citation_normalized_percentile