Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation Effectiveness Article Swipe
Audrey Salmon
,
K.E. Hammer
,
Eddie Antonio Santos
,
Brett A. Becker
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2501.05706
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2501.05706
Making errors is part of the programming process -- even for the most seasoned professionals. Novices in particular are bound to make many errors while learning. It is well known that traditional (compiler/interpreter) programming error messages have been less than helpful for many novices and can have effects such as being frustrating, containing confusing jargon, and being downright misleading. Recent work has found that large language models (LLMs) can generate excellent error explanations, but that the effectiveness of these error messages heavily depends on whether the LLM has been provided with context -- typically the original source code where the problem occurred. Knowing that programming error messages can be misleading and/or contain that serves little-to-no use (particularly for novices) we explore the reverse: what happens when GPT-3.5 is prompted for error explanations on just the erroneous source code itself -- original compiler/interpreter produced error message excluded. We utilized various strategies to make more effective error explanations, including one-shot prompting and fine-tuning. We report the baseline results of how effective the error explanations are at providing feedback, as well as how various prompting strategies might improve the explanations' effectiveness. Our results can help educators by understanding how LLMs respond to such prompts that novices are bound to make, and hopefully lead to more effective use of Generative AI in the classroom.
Related Topics
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2501.05706
- https://arxiv.org/pdf/2501.05706
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4406348142
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4406348142Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2501.05706Digital Object Identifier
- Title
-
Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation EffectivenessWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-01-10Full publication date if available
- Authors
-
Audrey Salmon, K.E. Hammer, Eddie Antonio Santos, Brett A. BeckerList of authors in order
- Landing page
-
https://arxiv.org/abs/2501.05706Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2501.05706Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2501.05706Direct OA link when available
- Concepts
-
Debugging, Computer science, Programming languageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4406348142 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2501.05706 |
| ids.doi | https://doi.org/10.48550/arxiv.2501.05706 |
| ids.openalex | https://openalex.org/W4406348142 |
| fwci | |
| type | preprint |
| title | Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation Effectiveness |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10260 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9927999973297119 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1710 |
| topics[0].subfield.display_name | Information Systems |
| topics[0].display_name | Software Engineering Research |
| topics[1].id | https://openalex.org/T12423 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.974399983882904 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1712 |
| topics[1].subfield.display_name | Software |
| topics[1].display_name | Software Reliability and Analysis Research |
| topics[2].id | https://openalex.org/T12127 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9725000262260437 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1705 |
| topics[2].subfield.display_name | Computer Networks and Communications |
| topics[2].display_name | Software System Performance and Reliability |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C168065819 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8851803541183472 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q845566 |
| concepts[0].display_name | Debugging |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.6461271047592163 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C199360897 |
| concepts[2].level | 1 |
| concepts[2].score | 0.3731617331504822 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[2].display_name | Programming language |
| keywords[0].id | https://openalex.org/keywords/debugging |
| keywords[0].score | 0.8851803541183472 |
| keywords[0].display_name | Debugging |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.6461271047592163 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/programming-language |
| keywords[2].score | 0.3731617331504822 |
| keywords[2].display_name | Programming language |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2501.05706 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2501.05706 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2501.05706 |
| locations[1].id | doi:10.48550/arxiv.2501.05706 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2501.05706 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5115870760 |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Audrey Salmon |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Salmon, Audrey |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5069013994 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | K.E. Hammer |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Hammer, Katie |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5048601698 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-5337-715X |
| authorships[2].author.display_name | Eddie Antonio Santos |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Santos, Eddie Antonio |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5054692443 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-1446-647X |
| authorships[3].author.display_name | Brett A. Becker |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Becker, Brett A. |
| authorships[3].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2501.05706 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Debugging Without Error Messages: How LLM Prompting Strategy Affects Programming Error Explanation Effectiveness |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10260 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9927999973297119 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1710 |
| primary_topic.subfield.display_name | Information Systems |
| primary_topic.display_name | Software Engineering Research |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W4321442002, https://openalex.org/W2015265939, https://openalex.org/W2284072287, https://openalex.org/W2611067230, https://openalex.org/W2480201319, https://openalex.org/W2387706296, https://openalex.org/W2061417947 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2501.05706 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2501.05706 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2501.05706 |
| primary_location.id | pmh:oai:arXiv.org:2501.05706 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2501.05706 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2501.05706 |
| publication_date | 2025-01-10 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.-- | 8, 92, 139 |
| abstract_inverted_index.AI | 216 |
| abstract_inverted_index.It | 26 |
| abstract_inverted_index.We | 146, 161 |
| abstract_inverted_index.as | 49, 176, 178 |
| abstract_inverted_index.at | 173 |
| abstract_inverted_index.be | 108 |
| abstract_inverted_index.by | 193 |
| abstract_inverted_index.in | 16, 217 |
| abstract_inverted_index.is | 2, 27, 127 |
| abstract_inverted_index.of | 4, 77, 166, 214 |
| abstract_inverted_index.on | 83, 132 |
| abstract_inverted_index.to | 20, 150, 198, 205, 210 |
| abstract_inverted_index.we | 119 |
| abstract_inverted_index.LLM | 86 |
| abstract_inverted_index.Our | 188 |
| abstract_inverted_index.and | 44, 55, 159, 207 |
| abstract_inverted_index.are | 18, 172, 203 |
| abstract_inverted_index.but | 73 |
| abstract_inverted_index.can | 45, 68, 107, 190 |
| abstract_inverted_index.for | 10, 41, 117, 129 |
| abstract_inverted_index.has | 61, 87 |
| abstract_inverted_index.how | 167, 179, 195 |
| abstract_inverted_index.the | 5, 11, 75, 85, 94, 99, 121, 134, 163, 169, 185, 218 |
| abstract_inverted_index.use | 115, 213 |
| abstract_inverted_index.LLMs | 196 |
| abstract_inverted_index.been | 37, 88 |
| abstract_inverted_index.code | 97, 137 |
| abstract_inverted_index.even | 9 |
| abstract_inverted_index.have | 36, 46 |
| abstract_inverted_index.help | 191 |
| abstract_inverted_index.just | 133 |
| abstract_inverted_index.lead | 209 |
| abstract_inverted_index.less | 38 |
| abstract_inverted_index.make | 21, 151 |
| abstract_inverted_index.many | 22, 42 |
| abstract_inverted_index.more | 152, 211 |
| abstract_inverted_index.most | 12 |
| abstract_inverted_index.part | 3 |
| abstract_inverted_index.such | 48, 199 |
| abstract_inverted_index.than | 39 |
| abstract_inverted_index.that | 30, 63, 74, 103, 112, 201 |
| abstract_inverted_index.well | 28, 177 |
| abstract_inverted_index.what | 123 |
| abstract_inverted_index.when | 125 |
| abstract_inverted_index.with | 90 |
| abstract_inverted_index.work | 60 |
| abstract_inverted_index.being | 50, 56 |
| abstract_inverted_index.bound | 19, 204 |
| abstract_inverted_index.error | 34, 71, 79, 105, 130, 143, 154, 170 |
| abstract_inverted_index.found | 62 |
| abstract_inverted_index.known | 29 |
| abstract_inverted_index.large | 64 |
| abstract_inverted_index.make, | 206 |
| abstract_inverted_index.might | 183 |
| abstract_inverted_index.these | 78 |
| abstract_inverted_index.where | 98 |
| abstract_inverted_index.while | 24 |
| abstract_inverted_index.(LLMs) | 67 |
| abstract_inverted_index.Making | 0 |
| abstract_inverted_index.Recent | 59 |
| abstract_inverted_index.and/or | 110 |
| abstract_inverted_index.errors | 1, 23 |
| abstract_inverted_index.itself | 138 |
| abstract_inverted_index.models | 66 |
| abstract_inverted_index.report | 162 |
| abstract_inverted_index.serves | 113 |
| abstract_inverted_index.source | 96, 136 |
| abstract_inverted_index.GPT-3.5 | 126 |
| abstract_inverted_index.Knowing | 102 |
| abstract_inverted_index.Novices | 15 |
| abstract_inverted_index.contain | 111 |
| abstract_inverted_index.context | 91 |
| abstract_inverted_index.depends | 82 |
| abstract_inverted_index.effects | 47 |
| abstract_inverted_index.explore | 120 |
| abstract_inverted_index.happens | 124 |
| abstract_inverted_index.heavily | 81 |
| abstract_inverted_index.helpful | 40 |
| abstract_inverted_index.improve | 184 |
| abstract_inverted_index.jargon, | 54 |
| abstract_inverted_index.message | 144 |
| abstract_inverted_index.novices | 43, 202 |
| abstract_inverted_index.problem | 100 |
| abstract_inverted_index.process | 7 |
| abstract_inverted_index.prompts | 200 |
| abstract_inverted_index.respond | 197 |
| abstract_inverted_index.results | 165, 189 |
| abstract_inverted_index.various | 148, 180 |
| abstract_inverted_index.whether | 84 |
| abstract_inverted_index.baseline | 164 |
| abstract_inverted_index.generate | 69 |
| abstract_inverted_index.language | 65 |
| abstract_inverted_index.messages | 35, 80, 106 |
| abstract_inverted_index.novices) | 118 |
| abstract_inverted_index.one-shot | 157 |
| abstract_inverted_index.original | 95, 140 |
| abstract_inverted_index.produced | 142 |
| abstract_inverted_index.prompted | 128 |
| abstract_inverted_index.provided | 89 |
| abstract_inverted_index.reverse: | 122 |
| abstract_inverted_index.seasoned | 13 |
| abstract_inverted_index.utilized | 147 |
| abstract_inverted_index.confusing | 53 |
| abstract_inverted_index.downright | 57 |
| abstract_inverted_index.educators | 192 |
| abstract_inverted_index.effective | 153, 168, 212 |
| abstract_inverted_index.erroneous | 135 |
| abstract_inverted_index.excellent | 70 |
| abstract_inverted_index.excluded. | 145 |
| abstract_inverted_index.feedback, | 175 |
| abstract_inverted_index.hopefully | 208 |
| abstract_inverted_index.including | 156 |
| abstract_inverted_index.learning. | 25 |
| abstract_inverted_index.occurred. | 101 |
| abstract_inverted_index.prompting | 158, 181 |
| abstract_inverted_index.providing | 174 |
| abstract_inverted_index.typically | 93 |
| abstract_inverted_index.Generative | 215 |
| abstract_inverted_index.classroom. | 219 |
| abstract_inverted_index.containing | 52 |
| abstract_inverted_index.misleading | 109 |
| abstract_inverted_index.particular | 17 |
| abstract_inverted_index.strategies | 149, 182 |
| abstract_inverted_index.misleading. | 58 |
| abstract_inverted_index.programming | 6, 33, 104 |
| abstract_inverted_index.traditional | 31 |
| abstract_inverted_index.explanations | 131, 171 |
| abstract_inverted_index.fine-tuning. | 160 |
| abstract_inverted_index.frustrating, | 51 |
| abstract_inverted_index.little-to-no | 114 |
| abstract_inverted_index.(particularly | 116 |
| abstract_inverted_index.effectiveness | 76 |
| abstract_inverted_index.explanations' | 186 |
| abstract_inverted_index.explanations, | 72, 155 |
| abstract_inverted_index.understanding | 194 |
| abstract_inverted_index.effectiveness. | 187 |
| abstract_inverted_index.professionals. | 14 |
| abstract_inverted_index.compiler/interpreter | 141 |
| abstract_inverted_index.(compiler/interpreter) | 32 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile |