THiNK: Can Large Language Models Think-aloud? Article Swipe

PDF

Yongan Yu , Mengqian Wu , Yi‐Chung Lin , Nikki G. Lobczowski ·

YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2505.20184

Assessing higher-order thinking skills in large language models (LLMs) remains a fundamental challenge, especially in tasks that go beyond surface-level accuracy. In this work, we propose THiNK (Testing Higher-order Notion of Knowledge), a multi-agent, feedback-driven evaluation framework grounded in Bloom's Taxonomy. THiNK frames reasoning assessment as an iterative task of problem generation, critique, and revision, encouraging LLMs to think-aloud through step-by-step reflection and refinement. This enables a systematic evaluation of both lower-order (e.g., remember, understand) and higher-order (e.g., evaluate, create) thinking skills. We apply THiNK to seven state-of-the-art LLMs and perform a detailed cognitive analysis of their outputs. Results reveal that while models reliably perform lower-order categories well, they struggle with applying knowledge in realistic contexts and exhibit limited abstraction. Structured feedback loops significantly improve reasoning performance, particularly in higher-order thinking. Qualitative evaluations further confirm that THiNK-guided outputs better align with domain logic and problem structure. The code of our framework provides a scalable methodology for probing and enhancing LLM reasoning, offering new directions for evaluation grounded in learning science, which is available at our GitHub repository.

Related Topics

C (Programming Language)

Llama (Language Model)

Serbian Language

Large Magellanic Cloud

Jerry & Marge Go Large

Language Isolate

First Language

Manx Language

Concepts

No concepts available.

Metadata

Type: preprint
Language: en
Landing Page: http://arxiv.org/abs/2505.20184
PDF: https://arxiv.org/pdf/2505.20184
OA Status: green
OpenAlex ID: https://openalex.org/W4414588292

All OpenAlex metadata

Raw OpenAlex JSON

OpenAlex ID: https://openalex.org/W4414588292

Canonical identifier for this work in OpenAlex
DOI: https://doi.org/10.48550/arxiv.2505.20184

Digital Object Identifier
Title: THiNK: Can Large Language Models Think-aloud?

Work title
Type: preprint

OpenAlex work type
Language: en

Primary language
Publication year: 2025

Year of publication
Publication date: 2025-05-26

Full publication date if available
Authors: Yongan Yu, Mengqian Wu, Yi‐Chung Lin, Nikki G. Lobczowski

List of authors in order
Landing page: https://arxiv.org/abs/2505.20184

Publisher landing page
PDF URL: https://arxiv.org/pdf/2505.20184

Direct link to full text PDF
Open access: Yes

Whether a free full text is available
OA status: green

Open access status per OpenAlex
OA URL: https://arxiv.org/pdf/2505.20184

Direct OA link when available
Cited by: 0

Total citation count in OpenAlex

Full payload

id	https://openalex.org/W4414588292
doi	https://doi.org/10.48550/arxiv.2505.20184
ids.doi	https://doi.org/10.48550/arxiv.2505.20184
ids.openalex	https://openalex.org/W4414588292
fwci
type	preprint
title	THiNK: Can Large Language Models Think-aloud?
biblio.issue
biblio.volume
biblio.last_page
biblio.first_page
topics[0].id	https://openalex.org/T10181
topics[0].field.id	https://openalex.org/fields/17
topics[0].field.display_name	Computer Science
topics[0].score	0.7766000032424927
topics[0].domain.id	https://openalex.org/domains/3
topics[0].domain.display_name	Physical Sciences
topics[0].subfield.id	https://openalex.org/subfields/1702
topics[0].subfield.display_name	Artificial Intelligence
topics[0].display_name	Natural Language Processing Techniques
is_xpac	False
apc_list
apc_paid
language	en
locations[0].id	pmh:oai:arXiv.org:2505.20184
locations[0].is_oa	True
locations[0].source.id	https://openalex.org/S4306400194
locations[0].source.issn
locations[0].source.type	repository
locations[0].source.is_oa	True
locations[0].source.issn_l
locations[0].source.is_core	False
locations[0].source.is_in_doaj	False
locations[0].source.display_name	arXiv (Cornell University)
locations[0].source.host_organization	https://openalex.org/I205783295
locations[0].source.host_organization_name	Cornell University
locations[0].source.host_organization_lineage	https://openalex.org/I205783295
locations[0].license
locations[0].pdf_url	https://arxiv.org/pdf/2505.20184
locations[0].version	submittedVersion
locations[0].raw_type	text
locations[0].license_id
locations[0].is_accepted	False
locations[0].is_published	False
locations[0].raw_source_name
locations[0].landing_page_url	http://arxiv.org/abs/2505.20184
locations[1].id	doi:10.48550/arxiv.2505.20184
locations[1].is_oa	True
locations[1].source.id	https://openalex.org/S4306400194
locations[1].source.issn
locations[1].source.type	repository
locations[1].source.is_oa	True
locations[1].source.issn_l
locations[1].source.is_core	False
locations[1].source.is_in_doaj	False
locations[1].source.display_name	arXiv (Cornell University)
locations[1].source.host_organization	https://openalex.org/I205783295
locations[1].source.host_organization_name	Cornell University
locations[1].source.host_organization_lineage	https://openalex.org/I205783295
locations[1].license
locations[1].pdf_url
locations[1].version
locations[1].raw_type	article
locations[1].license_id
locations[1].is_accepted	False
locations[1].is_published
locations[1].raw_source_name
locations[1].landing_page_url	https://doi.org/10.48550/arxiv.2505.20184
indexed_in	arxiv, datacite
authorships[0].author.id	https://openalex.org/A5112768031
authorships[0].author.orcid	https://orcid.org/0000-0002-4160-9344
authorships[0].author.display_name	Yongan Yu
authorships[0].author_position	first
authorships[0].raw_author_name	Yu, Yongan
authorships[0].is_corresponding	False
authorships[1].author.id	https://openalex.org/A5111125757
authorships[1].author.orcid
authorships[1].author.display_name	Mengqian Wu
authorships[1].author_position	middle
authorships[1].raw_author_name	Wu, Mengqian
authorships[1].is_corresponding	False
authorships[2].author.id	https://openalex.org/A5058441595
authorships[2].author.orcid	https://orcid.org/0000-0002-0352-2694
authorships[2].author.display_name	Yi‐Chung Lin
authorships[2].author_position	middle
authorships[2].raw_author_name	Lin, Yiran
authorships[2].is_corresponding	False
authorships[3].author.id	https://openalex.org/A5005519512
authorships[3].author.orcid	https://orcid.org/0000-0002-9018-2957
authorships[3].author.display_name	Nikki G. Lobczowski
authorships[3].author_position	last
authorships[3].raw_author_name	Lobczowski, Nikki G.
authorships[3].is_corresponding	False
has_content.pdf	True
has_content.grobid_xml	True
is_paratext	False
open_access.is_oa	True
open_access.oa_url	https://arxiv.org/pdf/2505.20184
open_access.oa_status	green
open_access.any_repository_has_fulltext	False
created_date	2025-10-10T00:00:00
display_name	THiNK: Can Large Language Models Think-aloud?
has_fulltext	True
is_retracted	False
updated_date	2025-11-06T06:51:31.235846
primary_topic.id	https://openalex.org/T10181
primary_topic.field.id	https://openalex.org/fields/17
primary_topic.field.display_name	Computer Science
primary_topic.score	0.7766000032424927
primary_topic.domain.id	https://openalex.org/domains/3
primary_topic.domain.display_name	Physical Sciences
primary_topic.subfield.id	https://openalex.org/subfields/1702
primary_topic.subfield.display_name	Artificial Intelligence
primary_topic.display_name	Natural Language Processing Techniques
cited_by_count	0
locations_count	2
best_oa_location.id	pmh:oai:arXiv.org:2505.20184
best_oa_location.is_oa	True
best_oa_location.source.id	https://openalex.org/S4306400194
best_oa_location.source.issn
best_oa_location.source.type	repository
best_oa_location.source.is_oa	True
best_oa_location.source.issn_l
best_oa_location.source.is_core	False
best_oa_location.source.is_in_doaj	False
best_oa_location.source.display_name	arXiv (Cornell University)
best_oa_location.source.host_organization	https://openalex.org/I205783295
best_oa_location.source.host_organization_name	Cornell University
best_oa_location.source.host_organization_lineage	https://openalex.org/I205783295
best_oa_location.license
best_oa_location.pdf_url	https://arxiv.org/pdf/2505.20184
best_oa_location.version	submittedVersion
best_oa_location.raw_type	text
best_oa_location.license_id
best_oa_location.is_accepted	False
best_oa_location.is_published	False
best_oa_location.raw_source_name
best_oa_location.landing_page_url	http://arxiv.org/abs/2505.20184
primary_location.id	pmh:oai:arXiv.org:2505.20184
primary_location.is_oa	True
primary_location.source.id	https://openalex.org/S4306400194
primary_location.source.issn
primary_location.source.type	repository
primary_location.source.is_oa	True
primary_location.source.issn_l
primary_location.source.is_core	False
primary_location.source.is_in_doaj	False
primary_location.source.display_name	arXiv (Cornell University)
primary_location.source.host_organization	https://openalex.org/I205783295
primary_location.source.host_organization_name	Cornell University
primary_location.source.host_organization_lineage	https://openalex.org/I205783295
primary_location.license
primary_location.pdf_url	https://arxiv.org/pdf/2505.20184
primary_location.version	submittedVersion
primary_location.raw_type	text
primary_location.license_id
primary_location.is_accepted	False
primary_location.is_published	False
primary_location.raw_source_name
primary_location.landing_page_url	http://arxiv.org/abs/2505.20184
publication_date	2025-05-26
publication_year	2025
referenced_works_count	0
abstract_inverted_index.a	10, 32, 66, 91, 152
abstract_inverted_index.In	21
abstract_inverted_index.We	82
abstract_inverted_index.an	46
abstract_inverted_index.as	45
abstract_inverted_index.at	173
abstract_inverted_index.go	17
abstract_inverted_index.in	4, 14, 38, 113, 128, 167
abstract_inverted_index.is	171
abstract_inverted_index.of	30, 49, 69, 95, 148
abstract_inverted_index.to	57, 85
abstract_inverted_index.we	24
abstract_inverted_index.LLM	159
abstract_inverted_index.The	146
abstract_inverted_index.and	53, 62, 75, 89, 116, 143, 157
abstract_inverted_index.for	155, 164
abstract_inverted_index.new	162
abstract_inverted_index.our	149, 174
abstract_inverted_index.LLMs	56, 88
abstract_inverted_index.This	64
abstract_inverted_index.both	70
abstract_inverted_index.code	147
abstract_inverted_index.task	48
abstract_inverted_index.that	16, 100, 135
abstract_inverted_index.they	108
abstract_inverted_index.this	22
abstract_inverted_index.with	110, 140
abstract_inverted_index.THiNK	26, 41, 84
abstract_inverted_index.align	139
abstract_inverted_index.apply	83
abstract_inverted_index.large	5
abstract_inverted_index.logic	142
abstract_inverted_index.loops	122
abstract_inverted_index.seven	86
abstract_inverted_index.tasks	15
abstract_inverted_index.their	96
abstract_inverted_index.well,	107
abstract_inverted_index.which	170
abstract_inverted_index.while	101
abstract_inverted_index.work,	23
abstract_inverted_index.(LLMs)	8
abstract_inverted_index.(e.g.,	72, 77
abstract_inverted_index.GitHub	175
abstract_inverted_index.Notion	29
abstract_inverted_index.better	138
abstract_inverted_index.beyond	18
abstract_inverted_index.domain	141
abstract_inverted_index.frames	42
abstract_inverted_index.models	7, 102
abstract_inverted_index.reveal	99
abstract_inverted_index.skills	3
abstract_inverted_index.Bloom's	39
abstract_inverted_index.Results	98
abstract_inverted_index.confirm	134
abstract_inverted_index.create)	79
abstract_inverted_index.enables	65
abstract_inverted_index.exhibit	117
abstract_inverted_index.further	133
abstract_inverted_index.improve	124
abstract_inverted_index.limited	118
abstract_inverted_index.outputs	137
abstract_inverted_index.perform	90, 104
abstract_inverted_index.probing	156
abstract_inverted_index.problem	50, 144
abstract_inverted_index.propose	25
abstract_inverted_index.remains	9
abstract_inverted_index.skills.	81
abstract_inverted_index.through	59
abstract_inverted_index.(Testing	27
abstract_inverted_index.analysis	94
abstract_inverted_index.applying	111
abstract_inverted_index.contexts	115
abstract_inverted_index.detailed	92
abstract_inverted_index.feedback	121
abstract_inverted_index.grounded	37, 166
abstract_inverted_index.language	6
abstract_inverted_index.learning	168
abstract_inverted_index.offering	161
abstract_inverted_index.outputs.	97
abstract_inverted_index.provides	151
abstract_inverted_index.reliably	103
abstract_inverted_index.scalable	153
abstract_inverted_index.science,	169
abstract_inverted_index.struggle	109
abstract_inverted_index.thinking	2, 80
abstract_inverted_index.Assessing	0
abstract_inverted_index.Taxonomy.	40
abstract_inverted_index.accuracy.	20
abstract_inverted_index.available	172
abstract_inverted_index.cognitive	93
abstract_inverted_index.critique,	52
abstract_inverted_index.enhancing	158
abstract_inverted_index.evaluate,	78
abstract_inverted_index.framework	36, 150
abstract_inverted_index.iterative	47
abstract_inverted_index.knowledge	112
abstract_inverted_index.realistic	114
abstract_inverted_index.reasoning	43, 125
abstract_inverted_index.remember,	73
abstract_inverted_index.revision,	54
abstract_inverted_index.thinking.	130
abstract_inverted_index.Structured	120
abstract_inverted_index.assessment	44
abstract_inverted_index.categories	106
abstract_inverted_index.challenge,	12
abstract_inverted_index.directions	163
abstract_inverted_index.especially	13
abstract_inverted_index.evaluation	35, 68, 165
abstract_inverted_index.reasoning,	160
abstract_inverted_index.reflection	61
abstract_inverted_index.structure.	145
abstract_inverted_index.systematic	67
abstract_inverted_index.Knowledge),	31
abstract_inverted_index.Qualitative	131
abstract_inverted_index.encouraging	55
abstract_inverted_index.evaluations	132
abstract_inverted_index.fundamental	11
abstract_inverted_index.generation,	51
abstract_inverted_index.lower-order	71, 105
abstract_inverted_index.methodology	154
abstract_inverted_index.refinement.	63
abstract_inverted_index.repository.	176
abstract_inverted_index.think-aloud	58
abstract_inverted_index.understand)	74
abstract_inverted_index.Higher-order	28
abstract_inverted_index.THiNK-guided	136
abstract_inverted_index.abstraction.	119
abstract_inverted_index.higher-order	1, 76, 129
abstract_inverted_index.multi-agent,	33
abstract_inverted_index.particularly	127
abstract_inverted_index.performance,	126
abstract_inverted_index.step-by-step	60
abstract_inverted_index.significantly	123
abstract_inverted_index.surface-level	19
abstract_inverted_index.feedback-driven	34
abstract_inverted_index.state-of-the-art	87
cited_by_percentile_year
countries_distinct_count	0
institutions_distinct_count	4
citation_normalized_percentile