Cutting Through the Noise: Boosting LLM Performance on Math Word Problems Article Swipe

PDF

Ujjwala Anantheswaran , Himanshu Gupta , Kevin Scaria , Shreyas Verma , Chitta Baral , Swaroop Mishra ·

YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2406.15444

Large Language Models (LLMs) excel at various tasks, including solving math word problems (MWPs), but struggle with real-world problems containing irrelevant information. To address this, we propose a prompting framework that generates adversarial variants of MWPs by adding irrelevant variables. We introduce a dataset, PROBLEMATHIC, containing both adversarial and non-adversarial MWPs. Our experiments reveal that LLMs are susceptible to distraction by numerical noise, resulting in an average relative performance drop of ~26% on adversarial MWPs. To mitigate this, we fine-tune LLMs (Llama-2, Mistral) on the adversarial samples from our dataset. Fine-tuning on adversarial training instances improves performance on adversarial MWPs by ~8%, indicating increased robustness to noise and improved ability to identify relevant data for reasoning. Finally, to assess the generalizability of our prompting framework, we introduce GSM-8K-Adv, an adversarial variant of the GSM-8K benchmark. LLMs continue to struggle when faced with adversarial information, reducing performance by up to 6%.

Related Topics

Mathematics

Computer Science

Mathematics Education

Biology

Biochemistry

Concepts

Robustness (evolution) Mathematics Computer science Mathematics education Biology Gene Biochemistry

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2406.15444
PDF: https://arxiv.org/pdf/2406.15444
OA Status: green
Related Works: 10
OpenAlex ID: https://openalex.org/W4400025230

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4400025230

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2406.15444

Digital Object Identifier
Title: Cutting Through the Noise: Boosting LLM Performance on Math Word Problems

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2024

Year of publication
Publication date: 2024-05-30

Full publication date if available
Authors: Ujjwala Anantheswaran, Himanshu Gupta, Kevin Scaria, Shreyas Verma, Chitta Baral, Swaroop Mishra

List of authors in order
Landing page: https://arxiv.org/abs/2406.15444

Publisher landing page
PDF URL: https://arxiv.org/pdf/2406.15444

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2406.15444

Direct OA link when available
Concepts: Robustness (evolution), Mathematics, Computer science, Mathematics education, Biology, Gene, Biochemistry

Top concepts (fields/topics) attached by OpenAlex
Cited by: 0

Total citation count in OpenAlex
Related works (count): 10

Other works algorithmically related by OpenAlex

Full payload

id	https://openalex.org/W4400025230
doi	https://doi.org/10.48550/arxiv.2406.15444
ids.doi	https://doi.org/10.48550/arxiv.2406.15444
ids.openalex	https://openalex.org/W4400025230
fwci
type	preprint
title	Cutting Through the Noise: Boosting LLM Performance on Math Word Problems
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10181
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.9962999820709229
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Natural Language Processing Techniques
topics[1].id	https://openalex.org/T13523
topics[1].field.id	https://openalex.org/fields/17
topics[1].field.display_name	Computer Science
topics[1].score	0.9753999710083008
topics[1].domain.id	https://openalex.org/domains/3
topics[1].domain.display_name	Physical Sciences
topics[1].subfield.id	https://openalex.org/subfields/1703
topics[1].subfield.display_name	Computational Theory and Mathematics
topics[1].display_name	Mathematics, Computing, and Information Processing
is_xpac	False
apc_list
apc_paid
concepts[0].id	https://openalex.org/C63479239
concepts[0].level	3
concepts[0].score	0.6492016315460205
concepts[0].wikidata	https://www.wikidata.org/wiki/Q7353546
concepts[0].display_name	Robustness (evolution)
concepts[1].id	https://openalex.org/C33923547
concepts[1].level	0
concepts[1].score	0.380764365196228
concepts[1].wikidata	https://www.wikidata.org/wiki/Q395
concepts[1].display_name	Mathematics
concepts[2].id	https://openalex.org/C41008148
concepts[2].level	0
concepts[2].score	0.3582776188850403
concepts[2].wikidata	https://www.wikidata.org/wiki/Q21198
concepts[2].display_name	Computer science
concepts[3].id	https://openalex.org/C145420912
concepts[3].level	1
concepts[3].score	0.33840715885162354
concepts[3].wikidata	https://www.wikidata.org/wiki/Q853077
concepts[3].display_name	Mathematics education
concepts[4].id	https://openalex.org/C86803240
concepts[4].level	0
concepts[4].score	0.12174001336097717
concepts[4].wikidata	https://www.wikidata.org/wiki/Q420
concepts[4].display_name	Biology
concepts[5].id	https://openalex.org/C104317684
concepts[5].level	2
concepts[5].score	0.0
concepts[5].wikidata	https://www.wikidata.org/wiki/Q7187
concepts[5].display_name	Gene
concepts[6].id	https://openalex.org/C55493867
concepts[6].level	1
concepts[6].score	0.0
concepts[6].wikidata	https://www.wikidata.org/wiki/Q7094
concepts[6].display_name	Biochemistry
keywords[0].id	https://openalex.org/keywords/robustness
keywords[0].score	0.6492016315460205
keywords[0].display_name	Robustness (evolution)
keywords[1].id	https://openalex.org/keywords/mathematics
keywords[1].score	0.380764365196228
keywords[1].display_name	Mathematics
keywords[2].id	https://openalex.org/keywords/computer-science
keywords[2].score	0.3582776188850403
keywords[2].display_name	Computer science
keywords[3].id	https://openalex.org/keywords/mathematics-education
keywords[3].score	0.33840715885162354
keywords[3].display_name	Mathematics education
keywords[4].id	https://openalex.org/keywords/biology
keywords[4].score	0.12174001336097717
keywords[4].display_name	Biology
language	en
locations[0].id	pmh:oai:arXiv.org:2406.15444
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2406.15444
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2406.15444
locations[1].id	doi:10.48550/arxiv.2406.15444
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license	public-domain
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id	https://openalex.org/licenses/public-domain
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2406.15444
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5059844013
authorships[0].author.orcid
authorships[0].author.display_name	Ujjwala Anantheswaran
authorships[0].author_position	first
authorships[0].raw_author_name	Anantheswaran, Ujjwala
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5101643160
authorships[1].author.orcid	https://orcid.org/0000-0003-3271-3032
authorships[1].author.display_name	Himanshu Gupta
authorships[1].author_position	middle
authorships[1].raw_author_name	Gupta, Himanshu
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5038616311
authorships[2].author.orcid
authorships[2].author.display_name	Kevin Scaria
authorships[2].author_position	middle
authorships[2].raw_author_name	Scaria, Kevin
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5013095307
authorships[3].author.orcid
authorships[3].author.display_name	Shreyas Verma
authorships[3].author_position	middle
authorships[3].raw_author_name	Verma, Shreyas
authorships[3].is_corresponding	False
authorships[4].author.id	https://openalex.org/A5083735830
authorships[4].author.orcid	https://orcid.org/0000-0002-7549-723X
authorships[4].author.display_name	Chitta Baral
authorships[4].author_position	middle
authorships[4].raw_author_name	Baral, Chitta
authorships[4].is_corresponding	False
authorships[5].author.id	https://openalex.org/A5063722751
authorships[5].author.orcid	https://orcid.org/0009-0001-6413-7001
authorships[5].author.display_name	Swaroop Mishra
authorships[5].author_position	last
authorships[5].raw_author_name	Mishra, Swaroop
authorships[5].is_corresponding	False
has_content.pdf	False
has_content.grobid_xml	False
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2406.15444
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2024-06-26T00:00:00
display_name	Cutting Through the Noise: Boosting LLM Performance on Math Word Problems
has_fulltext	False
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10181
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.9962999820709229
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Natural Language Processing Techniques
related_works	https://openalex.org/W2748952813, https://openalex.org/W4391375266, https://openalex.org/W1979597421, https://openalex.org/W2007980826, https://openalex.org/W2061531152, https://openalex.org/W3002753104, https://openalex.org/W2077600819, https://openalex.org/W2142036596, https://openalex.org/W2072657027, https://openalex.org/W2600246793
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2406.15444
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2406.15444
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2406.15444
primary_location.id	pmh:oai:arXiv.org:2406.15444
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2406.15444
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2406.15444
publication_date	2024-05-30
publication_year	2024
referenced_works_count	0
abstract_inverted_index.a	27, 42
abstract_inverted_index.To	22, 75
abstract_inverted_index.We	40
abstract_inverted_index.an	65, 128
abstract_inverted_index.at	5
abstract_inverted_index.by	36, 60, 100, 146
abstract_inverted_index.in	64
abstract_inverted_index.of	34, 70, 121, 131
abstract_inverted_index.on	72, 83, 91, 97
abstract_inverted_index.to	58, 105, 110, 117, 137, 148
abstract_inverted_index.up	147
abstract_inverted_index.we	25, 78, 125
abstract_inverted_index.6%.	149
abstract_inverted_index.Our	51
abstract_inverted_index.and	48, 107
abstract_inverted_index.are	56
abstract_inverted_index.but	14
abstract_inverted_index.for	114
abstract_inverted_index.our	88, 122
abstract_inverted_index.the	84, 119, 132
abstract_inverted_index.LLMs	55, 80, 135
abstract_inverted_index.MWPs	35, 99
abstract_inverted_index.both	46
abstract_inverted_index.data	113
abstract_inverted_index.drop	69
abstract_inverted_index.from	87
abstract_inverted_index.math	10
abstract_inverted_index.that	30, 54
abstract_inverted_index.when	139
abstract_inverted_index.with	16, 141
abstract_inverted_index.word	11
abstract_inverted_index.~26%	71
abstract_inverted_index.~8%,	101
abstract_inverted_index.Large	0
abstract_inverted_index.MWPs.	50, 74
abstract_inverted_index.excel	4
abstract_inverted_index.faced	140
abstract_inverted_index.noise	106
abstract_inverted_index.this,	24, 77
abstract_inverted_index.(LLMs)	3
abstract_inverted_index.GSM-8K	133
abstract_inverted_index.Models	2
abstract_inverted_index.adding	37
abstract_inverted_index.assess	118
abstract_inverted_index.noise,	62
abstract_inverted_index.reveal	53
abstract_inverted_index.tasks,	7
abstract_inverted_index.(MWPs),	13
abstract_inverted_index.ability	109
abstract_inverted_index.address	23
abstract_inverted_index.average	66
abstract_inverted_index.propose	26
abstract_inverted_index.samples	86
abstract_inverted_index.solving	9
abstract_inverted_index.variant	130
abstract_inverted_index.various	6
abstract_inverted_index.Finally,	116
abstract_inverted_index.Language	1
abstract_inverted_index.Mistral)	82
abstract_inverted_index.continue	136
abstract_inverted_index.dataset,	43
abstract_inverted_index.dataset.	89
abstract_inverted_index.identify	111
abstract_inverted_index.improved	108
abstract_inverted_index.improves	95
abstract_inverted_index.mitigate	76
abstract_inverted_index.problems	12, 18
abstract_inverted_index.reducing	144
abstract_inverted_index.relative	67
abstract_inverted_index.relevant	112
abstract_inverted_index.struggle	15, 138
abstract_inverted_index.training	93
abstract_inverted_index.variants	33
abstract_inverted_index.(Llama-2,	81
abstract_inverted_index.fine-tune	79
abstract_inverted_index.framework	29
abstract_inverted_index.generates	31
abstract_inverted_index.including	8
abstract_inverted_index.increased	103
abstract_inverted_index.instances	94
abstract_inverted_index.introduce	41, 126
abstract_inverted_index.numerical	61
abstract_inverted_index.prompting	28, 123
abstract_inverted_index.resulting	63
abstract_inverted_index.benchmark.	134
abstract_inverted_index.containing	19, 45
abstract_inverted_index.framework,	124
abstract_inverted_index.indicating	102
abstract_inverted_index.irrelevant	20, 38
abstract_inverted_index.real-world	17
abstract_inverted_index.reasoning.	115
abstract_inverted_index.robustness	104
abstract_inverted_index.variables.	39
abstract_inverted_index.Fine-tuning	90
abstract_inverted_index.GSM-8K-Adv,	127
abstract_inverted_index.adversarial	32, 47, 73, 85, 92, 98, 129, 142
abstract_inverted_index.distraction	59
abstract_inverted_index.experiments	52
abstract_inverted_index.performance	68, 96, 145
abstract_inverted_index.susceptible	57
abstract_inverted_index.information,	143
abstract_inverted_index.information.	21
abstract_inverted_index.PROBLEMATHIC,	44
abstract_inverted_index.non-adversarial	49
abstract_inverted_index.generalizability	120
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	6
citation_normalized_percentile