Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers Article Swipe

MAKE IMPACT

Alice Rueda , Maha M. Hassan , Argyrios Perivolaris , Bazen Gashaw Teferra , Reza Samavi , Sirisha Rambhatla , Yucheng Wu , Yanbo Zhang , Bo Cao , Divya Sharma , Sridhar Krishnan , Venkat Bhat · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2505.01482

Large language models (LLMs) have demonstrated remarkable capabilities in natural language understanding, reasoning, and problem-solving across various domains. However, their ability to perform complex, multi-step reasoning task-essential for applications in science, medicine, and law-remains an area of active investigation. This paper examines the reasoning capabilities of contemporary LLMs, analyzing their strengths, limitations, and potential for improvement. The study uses prompt engineering techniques on the Graduate-Level GoogleProof Q&A (GPQA) dataset to assess the scientific reasoning of GPT-4o. Five popular prompt engineering techniques and two tailored promptings were tested: baseline direct answer (zero-shot), chain-of-thought (CoT), zero-shot CoT, self-ask, self-consistency, decomposition, and multipath promptings. Our findings indicate that while LLMs exhibit emergent reasoning abilities, they often rely on pattern recognition rather than true logical inference, leading to inconsistencies in complex problem-solving. The results indicated that self-consistency outperformed the other prompt engineering technique with an accuracy of 52.99%, followed by direct answer (52.23%). Zero-shot CoT (50%) outperformed multipath (48.44%), decomposition (47.77%), self-ask (46.88%), and CoT (43.75%). Self-consistency performed the second worst in explaining the answers. Simple techniques such as direct answer, CoT, and zero-shot CoT have the best scientific reasoning. We propose a research agenda aimed at bridging these gaps by integrating structured reasoning frameworks, hybrid AI approaches, and human-in-the-loop methodologies. By critically evaluating the reasoning mechanisms of LLMs, this paper contributes to the ongoing discourse on the future of artificial general intelligence and the development of more robust, trustworthy AI systems.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2505.01482

PDF

https://arxiv.org/pdf/2505.01482

OA Status

green

OpenAlex ID

https://openalex.org/W4415026124

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4415026124

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2505.01482

Digital Object Identifier

Title

Understanding LLM Scientific Reasoning through Promptings and Model's Explanation on the Answers

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2025

Year of publication

Publication date

2025-05-02

Full publication date if available

Authors

Alice Rueda, Maha M. Hassan, Argyrios Perivolaris, Bazen Gashaw Teferra, Reza Samavi, Sirisha Rambhatla, Yucheng Wu, Yanbo Zhang, Bo Cao, Divya Sharma, Sridhar Krishnan, Venkat Bhat

List of authors in order

Landing page

https://arxiv.org/abs/2505.01482

Publisher landing page

PDF URL

https://arxiv.org/pdf/2505.01482

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2505.01482

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload