Applying Large Language Models and Chain-of-Thought for Automatic Scoring Article Swipe
Gyeong-Geon Lee
,
Ehsan Latif
,
Xuansheng Wu
,
Ninghao Liu
,
Xiaoming Zhaı
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2312.03748
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2312.03748
This study investigates the application of large language models (LLMs), specifically GPT-3.5 and GPT-4, with Chain-of-Though (CoT) in the automatic scoring of student-written responses to science assessments. We focused on overcoming the challenges of accessibility, technical complexity, and lack of explainability that have previously limited the use of artificial intelligence-based automatic scoring tools among researchers and educators. With a testing dataset comprising six assessment tasks (three binomial and three trinomial) with 1,650 student responses, we employed six prompt engineering strategies to automatically score student responses. The six strategies combined zero-shot or few-shot learning with CoT, either alone or alongside item stem and scoring rubrics. Results indicated that few-shot (acc = .67) outperformed zero-shot learning (acc = .60), with 12.6% increase. CoT, when used without item stem and scoring rubrics, did not significantly affect scoring accuracy (acc = .60). However, CoT prompting paired with contextual item stems and rubrics proved to be a significant contributor to scoring accuracy (13.44% increase for zero-shot; 3.7% increase for few-shot). We found a more balanced accuracy across different proficiency categories when CoT was used with a scoring rubric, highlighting the importance of domain-specific reasoning in enhancing the effectiveness of LLMs in scoring tasks. We also found that GPT-4 demonstrated superior performance over GPT -3.5 in various scoring tasks when combined with the single-call greedy sampling or ensemble voting nucleus sampling strategy, showing 8.64% difference. Particularly, the single-call greedy sampling strategy with GPT-4 outperformed other approaches.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2312.03748
- https://arxiv.org/pdf/2312.03748
- OA Status
- green
- Cited By
- 11
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4389500600
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4389500600Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2312.03748Digital Object Identifier
- Title
-
Applying Large Language Models and Chain-of-Thought for Automatic ScoringWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-11-30Full publication date if available
- Authors
-
Gyeong-Geon Lee, Ehsan Latif, Xuansheng Wu, Ninghao Liu, Xiaoming ZhaıList of authors in order
- Landing page
-
https://arxiv.org/abs/2312.03748Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2312.03748Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2312.03748Direct OA link when available
- Concepts
-
Computer science, Natural language processing, Language model, Chain (unit), Artificial intelligence, Physics, AstronomyTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
11Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 7, 2024: 4Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4389500600 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2312.03748 |
| ids.doi | https://doi.org/10.48550/arxiv.2312.03748 |
| ids.openalex | https://openalex.org/W4389500600 |
| fwci | |
| type | preprint |
| title | Applying Large Language Models and Chain-of-Thought for Automatic Scoring |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10456 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.4334000051021576 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Multi-Agent Systems and Negotiation |
| topics[1].id | https://openalex.org/T10215 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.415800005197525 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Semantic Web and Ontologies |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.5944797396659851 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C204321447 |
| concepts[1].level | 1 |
| concepts[1].score | 0.5257578492164612 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[1].display_name | Natural language processing |
| concepts[2].id | https://openalex.org/C137293760 |
| concepts[2].level | 2 |
| concepts[2].score | 0.4301431477069855 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q3621696 |
| concepts[2].display_name | Language model |
| concepts[3].id | https://openalex.org/C199185054 |
| concepts[3].level | 2 |
| concepts[3].score | 0.42384248971939087 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q552299 |
| concepts[3].display_name | Chain (unit) |
| concepts[4].id | https://openalex.org/C154945302 |
| concepts[4].level | 1 |
| concepts[4].score | 0.37163224816322327 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[4].display_name | Artificial intelligence |
| concepts[5].id | https://openalex.org/C121332964 |
| concepts[5].level | 0 |
| concepts[5].score | 0.0 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q413 |
| concepts[5].display_name | Physics |
| concepts[6].id | https://openalex.org/C1276947 |
| concepts[6].level | 1 |
| concepts[6].score | 0.0 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q333 |
| concepts[6].display_name | Astronomy |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.5944797396659851 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/natural-language-processing |
| keywords[1].score | 0.5257578492164612 |
| keywords[1].display_name | Natural language processing |
| keywords[2].id | https://openalex.org/keywords/language-model |
| keywords[2].score | 0.4301431477069855 |
| keywords[2].display_name | Language model |
| keywords[3].id | https://openalex.org/keywords/chain |
| keywords[3].score | 0.42384248971939087 |
| keywords[3].display_name | Chain (unit) |
| keywords[4].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[4].score | 0.37163224816322327 |
| keywords[4].display_name | Artificial intelligence |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2312.03748 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2312.03748 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2312.03748 |
| locations[1].id | doi:10.48550/arxiv.2312.03748 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2312.03748 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5005042692 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-7844-9412 |
| authorships[0].author.display_name | Gyeong-Geon Lee |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Lee, Gyeong-Geon |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5026133451 |
| authorships[1].author.orcid | https://orcid.org/0009-0008-6553-4093 |
| authorships[1].author.display_name | Ehsan Latif |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Latif, Ehsan |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5089884284 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-7816-7658 |
| authorships[2].author.display_name | Xuansheng Wu |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Wu, Xuansheng |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5007489034 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-9170-2424 |
| authorships[3].author.display_name | Ninghao Liu |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Liu, Ninghao |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5013379229 |
| authorships[4].author.orcid | https://orcid.org/0000-0003-4519-1931 |
| authorships[4].author.display_name | Xiaoming Zhaı |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Zhai, Xiaoming |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2312.03748 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Applying Large Language Models and Chain-of-Thought for Automatic Scoring |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10456 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.4334000051021576 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Multi-Agent Systems and Negotiation |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W2358668433, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W2382290278, https://openalex.org/W2478288626, https://openalex.org/W4391913857, https://openalex.org/W3204019825 |
| cited_by_count | 11 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 7 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 4 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2312.03748 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2312.03748 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2312.03748 |
| primary_location.id | pmh:oai:arXiv.org:2312.03748 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2312.03748 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2312.03748 |
| publication_date | 2023-11-30 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.= | 109, 115, 136 |
| abstract_inverted_index.a | 58, 151, 167, 180 |
| abstract_inverted_index.We | 27, 165, 198 |
| abstract_inverted_index.be | 150 |
| abstract_inverted_index.in | 17, 189, 195, 209 |
| abstract_inverted_index.of | 5, 21, 33, 39, 47, 186, 193 |
| abstract_inverted_index.on | 29 |
| abstract_inverted_index.or | 90, 97, 220 |
| abstract_inverted_index.to | 24, 80, 149, 154 |
| abstract_inverted_index.we | 74 |
| abstract_inverted_index.CoT | 139, 176 |
| abstract_inverted_index.GPT | 207 |
| abstract_inverted_index.The | 85 |
| abstract_inverted_index.and | 12, 37, 55, 67, 101, 126, 146 |
| abstract_inverted_index.did | 129 |
| abstract_inverted_index.for | 159, 163 |
| abstract_inverted_index.not | 130 |
| abstract_inverted_index.six | 62, 76, 86 |
| abstract_inverted_index.the | 3, 18, 31, 45, 184, 191, 216, 230 |
| abstract_inverted_index.use | 46 |
| abstract_inverted_index.was | 177 |
| abstract_inverted_index.(acc | 108, 114, 135 |
| abstract_inverted_index.-3.5 | 208 |
| abstract_inverted_index..67) | 110 |
| abstract_inverted_index.3.7% | 161 |
| abstract_inverted_index.CoT, | 94, 120 |
| abstract_inverted_index.LLMs | 194 |
| abstract_inverted_index.This | 0 |
| abstract_inverted_index.With | 57 |
| abstract_inverted_index.also | 199 |
| abstract_inverted_index.have | 42 |
| abstract_inverted_index.item | 99, 124, 144 |
| abstract_inverted_index.lack | 38 |
| abstract_inverted_index.more | 168 |
| abstract_inverted_index.over | 206 |
| abstract_inverted_index.stem | 100, 125 |
| abstract_inverted_index.that | 41, 106, 201 |
| abstract_inverted_index.used | 122, 178 |
| abstract_inverted_index.when | 121, 175, 213 |
| abstract_inverted_index.with | 14, 70, 93, 117, 142, 179, 215, 235 |
| abstract_inverted_index.(CoT) | 16 |
| abstract_inverted_index..60), | 116 |
| abstract_inverted_index..60). | 137 |
| abstract_inverted_index.1,650 | 71 |
| abstract_inverted_index.12.6% | 118 |
| abstract_inverted_index.8.64% | 227 |
| abstract_inverted_index.GPT-4 | 202, 236 |
| abstract_inverted_index.alone | 96 |
| abstract_inverted_index.among | 53 |
| abstract_inverted_index.found | 166, 200 |
| abstract_inverted_index.large | 6 |
| abstract_inverted_index.other | 238 |
| abstract_inverted_index.score | 82 |
| abstract_inverted_index.stems | 145 |
| abstract_inverted_index.study | 1 |
| abstract_inverted_index.tasks | 64, 212 |
| abstract_inverted_index.three | 68 |
| abstract_inverted_index.tools | 52 |
| abstract_inverted_index.(three | 65 |
| abstract_inverted_index.GPT-4, | 13 |
| abstract_inverted_index.across | 171 |
| abstract_inverted_index.affect | 132 |
| abstract_inverted_index.either | 95 |
| abstract_inverted_index.greedy | 218, 232 |
| abstract_inverted_index.models | 8 |
| abstract_inverted_index.paired | 141 |
| abstract_inverted_index.prompt | 77 |
| abstract_inverted_index.proved | 148 |
| abstract_inverted_index.tasks. | 197 |
| abstract_inverted_index.voting | 222 |
| abstract_inverted_index.(13.44% | 157 |
| abstract_inverted_index.(LLMs), | 9 |
| abstract_inverted_index.GPT-3.5 | 11 |
| abstract_inverted_index.Results | 104 |
| abstract_inverted_index.dataset | 60 |
| abstract_inverted_index.focused | 28 |
| abstract_inverted_index.limited | 44 |
| abstract_inverted_index.nucleus | 223 |
| abstract_inverted_index.rubric, | 182 |
| abstract_inverted_index.rubrics | 147 |
| abstract_inverted_index.science | 25 |
| abstract_inverted_index.scoring | 20, 51, 102, 127, 133, 155, 181, 196, 211 |
| abstract_inverted_index.showing | 226 |
| abstract_inverted_index.student | 72, 83 |
| abstract_inverted_index.testing | 59 |
| abstract_inverted_index.various | 210 |
| abstract_inverted_index.without | 123 |
| abstract_inverted_index.However, | 138 |
| abstract_inverted_index.accuracy | 134, 156, 170 |
| abstract_inverted_index.balanced | 169 |
| abstract_inverted_index.binomial | 66 |
| abstract_inverted_index.combined | 88, 214 |
| abstract_inverted_index.employed | 75 |
| abstract_inverted_index.ensemble | 221 |
| abstract_inverted_index.few-shot | 91, 107 |
| abstract_inverted_index.increase | 158, 162 |
| abstract_inverted_index.language | 7 |
| abstract_inverted_index.learning | 92, 113 |
| abstract_inverted_index.rubrics, | 128 |
| abstract_inverted_index.rubrics. | 103 |
| abstract_inverted_index.sampling | 219, 224, 233 |
| abstract_inverted_index.strategy | 234 |
| abstract_inverted_index.superior | 204 |
| abstract_inverted_index.alongside | 98 |
| abstract_inverted_index.automatic | 19, 50 |
| abstract_inverted_index.different | 172 |
| abstract_inverted_index.enhancing | 190 |
| abstract_inverted_index.increase. | 119 |
| abstract_inverted_index.indicated | 105 |
| abstract_inverted_index.prompting | 140 |
| abstract_inverted_index.reasoning | 188 |
| abstract_inverted_index.responses | 23 |
| abstract_inverted_index.strategy, | 225 |
| abstract_inverted_index.technical | 35 |
| abstract_inverted_index.zero-shot | 89, 112 |
| abstract_inverted_index.artificial | 48 |
| abstract_inverted_index.assessment | 63 |
| abstract_inverted_index.categories | 174 |
| abstract_inverted_index.challenges | 32 |
| abstract_inverted_index.comprising | 61 |
| abstract_inverted_index.contextual | 143 |
| abstract_inverted_index.educators. | 56 |
| abstract_inverted_index.few-shot). | 164 |
| abstract_inverted_index.importance | 185 |
| abstract_inverted_index.overcoming | 30 |
| abstract_inverted_index.previously | 43 |
| abstract_inverted_index.responses, | 73 |
| abstract_inverted_index.responses. | 84 |
| abstract_inverted_index.strategies | 79, 87 |
| abstract_inverted_index.trinomial) | 69 |
| abstract_inverted_index.zero-shot; | 160 |
| abstract_inverted_index.application | 4 |
| abstract_inverted_index.approaches. | 239 |
| abstract_inverted_index.complexity, | 36 |
| abstract_inverted_index.contributor | 153 |
| abstract_inverted_index.difference. | 228 |
| abstract_inverted_index.engineering | 78 |
| abstract_inverted_index.performance | 205 |
| abstract_inverted_index.proficiency | 173 |
| abstract_inverted_index.researchers | 54 |
| abstract_inverted_index.significant | 152 |
| abstract_inverted_index.single-call | 217, 231 |
| abstract_inverted_index.assessments. | 26 |
| abstract_inverted_index.demonstrated | 203 |
| abstract_inverted_index.highlighting | 183 |
| abstract_inverted_index.investigates | 2 |
| abstract_inverted_index.outperformed | 111, 237 |
| abstract_inverted_index.specifically | 10 |
| abstract_inverted_index.Particularly, | 229 |
| abstract_inverted_index.automatically | 81 |
| abstract_inverted_index.effectiveness | 192 |
| abstract_inverted_index.significantly | 131 |
| abstract_inverted_index.accessibility, | 34 |
| abstract_inverted_index.explainability | 40 |
| abstract_inverted_index.Chain-of-Though | 15 |
| abstract_inverted_index.domain-specific | 187 |
| abstract_inverted_index.student-written | 22 |
| abstract_inverted_index.intelligence-based | 49 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/4 |
| sustainable_development_goals[0].score | 0.8500000238418579 |
| sustainable_development_goals[0].display_name | Quality Education |
| citation_normalized_percentile |