A Fragile Number Sense: Probing the Elemental Limits of Numerical Reasoning in LLMs Article Swipe
Roussel Rahman
,
Aashwin Mishra
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2509.06332
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2509.06332
Large Language Models (LLMs) have demonstrated remarkable emergent capabilities, yet the robustness of their numerical reasoning remains an open question. While standard benchmarks evaluate LLM reasoning on complex problem sets using aggregated metrics, they often obscure foundational weaknesses. In this work, we probe LLM mathematical numeracy by evaluating performance on problems of escalating complexity, from constituent operations to combinatorial puzzles. We test several state-of-the-art LLM-based agents on a 100-problem challenge comprising four categories: (1) basic arithmetic, (2) advanced operations, (3) primality checking, and (4) the Game of 24 number puzzle. Our results show that while the agents achieved high accuracy on the first three categories, which require deterministic algorithmic execution, they consistently failed at the number puzzle, underlining its demand for a heuristic search over a large combinatorial space to be a significant bottleneck. These findings reveal that the agents' proficiency is largely confined to recalling and executing known algorithms, rather than performing generative problem-solving. This suggests their apparent numerical reasoning is more akin to sophisticated pattern-matching than flexible, analytical thought, limiting their potential for tasks that require novel or creative numerical insights.
Related Topics
Concepts
No concepts available.
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2509.06332
- https://arxiv.org/pdf/2509.06332
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W4415055548
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4415055548Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2509.06332Digital Object Identifier
- Title
-
A Fragile Number Sense: Probing the Elemental Limits of Numerical Reasoning in LLMsWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-09-08Full publication date if available
- Authors
-
Roussel Rahman, Aashwin MishraList of authors in order
- Landing page
-
https://arxiv.org/abs/2509.06332Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2509.06332Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2509.06332Direct OA link when available
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W4415055548 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2509.06332 |
| ids.doi | https://doi.org/10.48550/arxiv.2509.06332 |
| ids.openalex | https://openalex.org/W4415055548 |
| fwci | |
| type | preprint |
| title | A Fragile Number Sense: Probing the Elemental Limits of Numerical Reasoning in LLMs |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T13523 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.967199981212616 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1703 |
| topics[0].subfield.display_name | Computational Theory and Mathematics |
| topics[0].display_name | Mathematics, Computing, and Information Processing |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2509.06332 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2509.06332 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2509.06332 |
| locations[1].id | doi:10.48550/arxiv.2509.06332 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2509.06332 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5066597149 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-2009-4246 |
| authorships[0].author.display_name | Roussel Rahman |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Rahman, Roussel |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5101639379 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-7257-6784 |
| authorships[1].author.display_name | Aashwin Mishra |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | Mishra, Aashwin Ananda |
| authorships[1].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2509.06332 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-11T00:00:00 |
| display_name | A Fragile Number Sense: Probing the Elemental Limits of Numerical Reasoning in LLMs |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T13523 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.967199981212616 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1703 |
| primary_topic.subfield.display_name | Computational Theory and Mathematics |
| primary_topic.display_name | Mathematics, Computing, and Information Processing |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2509.06332 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2509.06332 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2509.06332 |
| primary_location.id | pmh:oai:arXiv.org:2509.06332 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2509.06332 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2509.06332 |
| publication_date | 2025-09-08 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 67, 121, 125, 131 |
| abstract_inverted_index.24 | 87 |
| abstract_inverted_index.In | 38 |
| abstract_inverted_index.We | 60 |
| abstract_inverted_index.an | 17 |
| abstract_inverted_index.at | 113 |
| abstract_inverted_index.be | 130 |
| abstract_inverted_index.by | 46 |
| abstract_inverted_index.is | 141, 161 |
| abstract_inverted_index.of | 12, 51, 86 |
| abstract_inverted_index.on | 26, 49, 66, 100 |
| abstract_inverted_index.or | 179 |
| abstract_inverted_index.to | 57, 129, 144, 164 |
| abstract_inverted_index.we | 41 |
| abstract_inverted_index.(1) | 73 |
| abstract_inverted_index.(2) | 76 |
| abstract_inverted_index.(3) | 79 |
| abstract_inverted_index.(4) | 83 |
| abstract_inverted_index.LLM | 24, 43 |
| abstract_inverted_index.Our | 90 |
| abstract_inverted_index.and | 82, 146 |
| abstract_inverted_index.for | 120, 174 |
| abstract_inverted_index.its | 118 |
| abstract_inverted_index.the | 10, 84, 95, 101, 114, 138 |
| abstract_inverted_index.yet | 9 |
| abstract_inverted_index.Game | 85 |
| abstract_inverted_index.This | 155 |
| abstract_inverted_index.akin | 163 |
| abstract_inverted_index.four | 71 |
| abstract_inverted_index.from | 54 |
| abstract_inverted_index.have | 4 |
| abstract_inverted_index.high | 98 |
| abstract_inverted_index.more | 162 |
| abstract_inverted_index.open | 18 |
| abstract_inverted_index.over | 124 |
| abstract_inverted_index.sets | 29 |
| abstract_inverted_index.show | 92 |
| abstract_inverted_index.test | 61 |
| abstract_inverted_index.than | 151, 167 |
| abstract_inverted_index.that | 93, 137, 176 |
| abstract_inverted_index.they | 33, 110 |
| abstract_inverted_index.this | 39 |
| abstract_inverted_index.Large | 0 |
| abstract_inverted_index.These | 134 |
| abstract_inverted_index.While | 20 |
| abstract_inverted_index.basic | 74 |
| abstract_inverted_index.first | 102 |
| abstract_inverted_index.known | 148 |
| abstract_inverted_index.large | 126 |
| abstract_inverted_index.novel | 178 |
| abstract_inverted_index.often | 34 |
| abstract_inverted_index.probe | 42 |
| abstract_inverted_index.space | 128 |
| abstract_inverted_index.tasks | 175 |
| abstract_inverted_index.their | 13, 157, 172 |
| abstract_inverted_index.three | 103 |
| abstract_inverted_index.using | 30 |
| abstract_inverted_index.which | 105 |
| abstract_inverted_index.while | 94 |
| abstract_inverted_index.work, | 40 |
| abstract_inverted_index.(LLMs) | 3 |
| abstract_inverted_index.Models | 2 |
| abstract_inverted_index.agents | 65, 96 |
| abstract_inverted_index.demand | 119 |
| abstract_inverted_index.failed | 112 |
| abstract_inverted_index.number | 88, 115 |
| abstract_inverted_index.rather | 150 |
| abstract_inverted_index.reveal | 136 |
| abstract_inverted_index.search | 123 |
| abstract_inverted_index.agents' | 139 |
| abstract_inverted_index.complex | 27 |
| abstract_inverted_index.largely | 142 |
| abstract_inverted_index.obscure | 35 |
| abstract_inverted_index.problem | 28 |
| abstract_inverted_index.puzzle, | 116 |
| abstract_inverted_index.puzzle. | 89 |
| abstract_inverted_index.remains | 16 |
| abstract_inverted_index.require | 106, 177 |
| abstract_inverted_index.results | 91 |
| abstract_inverted_index.several | 62 |
| abstract_inverted_index.Language | 1 |
| abstract_inverted_index.accuracy | 99 |
| abstract_inverted_index.achieved | 97 |
| abstract_inverted_index.advanced | 77 |
| abstract_inverted_index.apparent | 158 |
| abstract_inverted_index.confined | 143 |
| abstract_inverted_index.creative | 180 |
| abstract_inverted_index.emergent | 7 |
| abstract_inverted_index.evaluate | 23 |
| abstract_inverted_index.findings | 135 |
| abstract_inverted_index.limiting | 171 |
| abstract_inverted_index.metrics, | 32 |
| abstract_inverted_index.numeracy | 45 |
| abstract_inverted_index.problems | 50 |
| abstract_inverted_index.puzzles. | 59 |
| abstract_inverted_index.standard | 21 |
| abstract_inverted_index.suggests | 156 |
| abstract_inverted_index.thought, | 170 |
| abstract_inverted_index.LLM-based | 64 |
| abstract_inverted_index.challenge | 69 |
| abstract_inverted_index.checking, | 81 |
| abstract_inverted_index.executing | 147 |
| abstract_inverted_index.flexible, | 168 |
| abstract_inverted_index.heuristic | 122 |
| abstract_inverted_index.insights. | 182 |
| abstract_inverted_index.numerical | 14, 159, 181 |
| abstract_inverted_index.potential | 173 |
| abstract_inverted_index.primality | 80 |
| abstract_inverted_index.question. | 19 |
| abstract_inverted_index.reasoning | 15, 25, 160 |
| abstract_inverted_index.recalling | 145 |
| abstract_inverted_index.aggregated | 31 |
| abstract_inverted_index.analytical | 169 |
| abstract_inverted_index.benchmarks | 22 |
| abstract_inverted_index.comprising | 70 |
| abstract_inverted_index.escalating | 52 |
| abstract_inverted_index.evaluating | 47 |
| abstract_inverted_index.execution, | 109 |
| abstract_inverted_index.generative | 153 |
| abstract_inverted_index.operations | 56 |
| abstract_inverted_index.performing | 152 |
| abstract_inverted_index.remarkable | 6 |
| abstract_inverted_index.robustness | 11 |
| abstract_inverted_index.100-problem | 68 |
| abstract_inverted_index.algorithmic | 108 |
| abstract_inverted_index.algorithms, | 149 |
| abstract_inverted_index.arithmetic, | 75 |
| abstract_inverted_index.bottleneck. | 133 |
| abstract_inverted_index.categories, | 104 |
| abstract_inverted_index.categories: | 72 |
| abstract_inverted_index.complexity, | 53 |
| abstract_inverted_index.constituent | 55 |
| abstract_inverted_index.operations, | 78 |
| abstract_inverted_index.performance | 48 |
| abstract_inverted_index.proficiency | 140 |
| abstract_inverted_index.significant | 132 |
| abstract_inverted_index.underlining | 117 |
| abstract_inverted_index.weaknesses. | 37 |
| abstract_inverted_index.consistently | 111 |
| abstract_inverted_index.demonstrated | 5 |
| abstract_inverted_index.foundational | 36 |
| abstract_inverted_index.mathematical | 44 |
| abstract_inverted_index.capabilities, | 8 |
| abstract_inverted_index.combinatorial | 58, 127 |
| abstract_inverted_index.deterministic | 107 |
| abstract_inverted_index.sophisticated | 165 |
| abstract_inverted_index.pattern-matching | 166 |
| abstract_inverted_index.problem-solving. | 154 |
| abstract_inverted_index.state-of-the-art | 63 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 2 |
| citation_normalized_percentile |