Calibration of Large Language Models on Code Summarization Article Swipe

MAKE IMPACT

Yuvraj Virk , Prémkumar Dévanbu , Toufique Ahmed · YOU? · · 2024 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2404.19318

A brief, fluent, and relevant summary can be helpful during program comprehension; however, such a summary does require significant human effort to produce. Often, good summaries are unavailable in software projects, which makes maintenance more difficult. There has been a considerable body of research into automated AI-based methods, using Large Language models (LLMs), to generate summaries of code; there also has been quite a bit of work on ways to measure the performance of such summarization methods, with special attention paid to how closely these AI-generated summaries resemble a summary a human might have produced. Measures such as BERTScore and BLEU have been suggested and evaluated with human-subject studies. However, LLM-generated summaries can be inaccurate, incomplete, etc.: generally, too dissimilar to one that a good developer might write. Given an LLM-generated code summary, how can a user rationally judge if a summary is sufficiently good and reliable? Given just some input source code, and an LLM-generated summary, existing approaches can help judge brevity, fluency and relevance of the summary; however, it's difficult to gauge whether an LLM-generated summary sufficiently resembles what a human might produce, without a "golden" human-produced summary to compare against. We study this resemblance question as calibration problem: given just the code & the summary from an LLM, can we compute a confidence measure, that provides a reliable indication of whether the summary sufficiently resembles what a human would have produced in this situation? We examine this question using several LLMs, for several languages, and in several different settings. Our investigation suggests approaches to provide reliable predictions of the likelihood that an LLM-generated summary would sufficiently resemble a summary a human might write for the same code.

Type

preprint

Language

en

Landing Page

http://arxiv.org/abs/2404.19318

PDF

https://arxiv.org/pdf/2404.19318

OA Status

green

Cited By

1

Related Works

10

OpenAlex ID

https://openalex.org/W4396600598

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4396600598

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2404.19318

Digital Object Identifier

Title

Calibration of Large Language Models on Code Summarization

Work title

Type

preprint

OpenAlex work type

Language

en

Primary language

Publication year

2024

Year of publication

Publication date

2024-04-30

Full publication date if available

Authors

Yuvraj Virk, Prémkumar Dévanbu, Toufique Ahmed

List of authors in order

Landing page

https://arxiv.org/abs/2404.19318

Publisher landing page

PDF URL

https://arxiv.org/pdf/2404.19318

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2404.19318

Direct OA link when available

Concepts

Code (set theory), Computer science, Confidence interval, Psychology, Statistics, Programming language, Mathematics, Set (abstract data type)

Top concepts (fields/topics) attached by OpenAlex

Cited by

1

Total citation count in OpenAlex

Citations by year (recent)

2025: 1

Per-year citation counts (last 5 years)

Related works (count)

10

Other works algorithmically related by OpenAlex

Full payload