CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection Article Swipe

MAKE IMPACT

Qingyu Zhang , Puzhuo Liu , Di Peng , Chenxiong Qian · YOU? · · 2025 · Open Access · · DOI: https://doi.org/10.48550/arxiv.2511.19875

Version control relies on commit messages to convey the rationale for code changes, but these messages are often low quality and, more critically, inconsistent with their diffs-known as message-code inconsistency (MCI). MCIs mislead reviewers, hinder maintenance, contaminate research datasets, and may obscure security patches. Yet, no dedicated benchmark exists to evaluate models for MCI detection. We introduce CODEFUSE-COMMITEVAL, the first benchmark designed for MCI detection using large language models (LLMs). Built on the ApacheCM dataset for diversity and quality, we generate seven types of inconsistent messages through rule-guided mutations of originally consistent commits and apply two-fold validation to verify both positive and negative samples. Using this labeled dataset of message-diff pairs, we evaluate six state-of-the-art open-source LLMs under a vanilla setting and with three augmentation strategies: few-shot prompting, chain-of-thought, and extended context. Results show models detect inconsistent commits more reliably than consistent ones (average Recall 85.95%, Precision 80.28%, Specificity 63.8%); gpt-oss-20B performs best overall but uses over twice the tokens of others. Augmentation effects vary: adjacent context helps larger models but adds noise for smaller ones; few-shot improves accuracy and reduces token use, yet increases universally incorrect predictions; chain-of-thought boosts precision and specificity at the cost of recall and higher token consumption. Type-wise analysis reveals higher detectability for component, file-path, and operation inconsistencies, but lower accuracy and higher token cost for intent-level "purpose" inconsistencies. CODEFUSE-COMMITEVAL provides a rigorous foundation for measuring, comparing, and advancing MCI detection, highlighting the need for richer context and balanced data to capture high-level semantic gaps.

Type

preprint

Landing Page

http://arxiv.org/abs/2511.19875

PDF

https://arxiv.org/pdf/2511.19875

OA Status

green

OpenAlex ID

https://openalex.org/W4416766960

Raw OpenAlex JSON

OpenAlex ID

https://openalex.org/W4416766960

Canonical identifier for this work in OpenAlex

DOI

https://doi.org/10.48550/arxiv.2511.19875

Digital Object Identifier

Title

CodeFuse-CommitEval: Towards Benchmarking LLM's Power on Commit Message and Code Change Inconsistency Detection

Work title

Type

preprint

OpenAlex work type

Publication year

2025

Year of publication

Publication date

2025-11-25

Full publication date if available

Authors

Qingyu Zhang, Puzhuo Liu, Di Peng, Chenxiong Qian

List of authors in order

Landing page

https://arxiv.org/abs/2511.19875

Publisher landing page

PDF URL

https://arxiv.org/pdf/2511.19875

Direct link to full text PDF

Open access

Yes

Whether a free full text is available

OA status

green

Open access status per OpenAlex

OA URL

https://arxiv.org/pdf/2511.19875

Direct OA link when available

Cited by

0

Total citation count in OpenAlex

Full payload