MdEval: Massively Multilingual Code Debugging Article Swipe
Shukai Liu
,
Linzheng Chai
,
Jian Yang
,
Jiajun Shi
,
He Zhu
,
Liran Wang
,
Ke Jin
,
Wei Zhang
,
H H Zhu
,
Shuyue Guo
,
Tao Sun
,
Jiaheng Liu
,
Yi Duan
,
Hao Yu
,
Liqun Yang
,
Guanglin Niu
,
Ge Zhang
,
Zhoujun Li
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2411.02310
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2411.02310
Code large language models (LLMs) have made significant progress in code debugging by directly generating the correct code based on the buggy code snippet. Programming benchmarks, typically consisting of buggy code snippet and their associated test cases, are used to assess the debugging capabilities of LLMs. However, many existing benchmarks primarily focus on Python and are often limited in terms of language diversity (e.g., DebugBench and DebugEval). To advance the field of multilingual debugging with LLMs, we propose the first massively multilingual debugging benchmark, which includes 3.6K test samples of 18 programming languages and covers the automated program repair (APR) task, the code review (CR) task, and the bug identification (BI) task. Further, we introduce the debugging instruction corpora MDEVAL-INSTRUCT by injecting bugs into the correct multilingual queries and solutions (xDebugGen). Further, a multilingual debugger xDebugCoder trained on MDEVAL-INSTRUCT as a strong baseline specifically to handle the bugs of a wide range of programming languages (e.g. "Missing Mut" in language Rust and "Misused Macro Definition" in language C). Our extensive experiments on MDEVAL reveal a notable performance gap between open-source models and closed-source LLMs (e.g., GPT and Claude series), highlighting huge room for improvement in multilingual code debugging scenarios.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2411.02310
- https://arxiv.org/pdf/2411.02310
- OA Status
- green
- Cited By
- 1
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4404355202
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4404355202Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2411.02310Digital Object Identifier
- Title
-
MdEval: Massively Multilingual Code DebuggingWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-11-04Full publication date if available
- Authors
-
Shukai Liu, Linzheng Chai, Jian Yang, Jiajun Shi, He Zhu, Liran Wang, Ke Jin, Wei Zhang, H H Zhu, Shuyue Guo, Tao Sun, Jiaheng Liu, Yi Duan, Hao Yu, Liqun Yang, Guanglin Niu, Ge Zhang, Zhoujun LiList of authors in order
- Landing page
-
https://arxiv.org/abs/2411.02310Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2411.02310Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2411.02310Direct OA link when available
- Concepts
-
Debugging, Computer science, Programming language, Code (set theory), Massively parallel, Parallel computing, Set (abstract data type)Top concepts (fields/topics) attached by OpenAlex
- Cited by
-
1Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 1Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4404355202 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2411.02310 |
| ids.doi | https://doi.org/10.48550/arxiv.2411.02310 |
| ids.openalex | https://openalex.org/W4404355202 |
| fwci | |
| type | preprint |
| title | MdEval: Massively Multilingual Code Debugging |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10181 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9909999966621399 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Natural Language Processing Techniques |
| topics[1].id | https://openalex.org/T12031 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9498000144958496 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Speech and dialogue systems |
| topics[2].id | https://openalex.org/T10028 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9276999831199646 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Topic Modeling |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C168065819 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8164635300636292 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q845566 |
| concepts[0].display_name | Debugging |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.6881629228591919 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C199360897 |
| concepts[2].level | 1 |
| concepts[2].score | 0.6658205986022949 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[2].display_name | Programming language |
| concepts[3].id | https://openalex.org/C2776760102 |
| concepts[3].level | 3 |
| concepts[3].score | 0.5990877747535706 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q5139990 |
| concepts[3].display_name | Code (set theory) |
| concepts[4].id | https://openalex.org/C190475519 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5419114828109741 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q544384 |
| concepts[4].display_name | Massively parallel |
| concepts[5].id | https://openalex.org/C173608175 |
| concepts[5].level | 1 |
| concepts[5].score | 0.3863928020000458 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q232661 |
| concepts[5].display_name | Parallel computing |
| concepts[6].id | https://openalex.org/C177264268 |
| concepts[6].level | 2 |
| concepts[6].score | 0.0 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q1514741 |
| concepts[6].display_name | Set (abstract data type) |
| keywords[0].id | https://openalex.org/keywords/debugging |
| keywords[0].score | 0.8164635300636292 |
| keywords[0].display_name | Debugging |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.6881629228591919 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/programming-language |
| keywords[2].score | 0.6658205986022949 |
| keywords[2].display_name | Programming language |
| keywords[3].id | https://openalex.org/keywords/code |
| keywords[3].score | 0.5990877747535706 |
| keywords[3].display_name | Code (set theory) |
| keywords[4].id | https://openalex.org/keywords/massively-parallel |
| keywords[4].score | 0.5419114828109741 |
| keywords[4].display_name | Massively parallel |
| keywords[5].id | https://openalex.org/keywords/parallel-computing |
| keywords[5].score | 0.3863928020000458 |
| keywords[5].display_name | Parallel computing |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2411.02310 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2411.02310 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2411.02310 |
| locations[1].id | doi:10.48550/arxiv.2411.02310 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2411.02310 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5035078709 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-7369-2958 |
| authorships[0].author.display_name | Shukai Liu |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Liu, Shukai |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5077248623 |
| authorships[1].author.orcid | https://orcid.org/0009-0001-2129-3207 |
| authorships[1].author.display_name | Linzheng Chai |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Chai, Linzheng |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5079652214 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-9590-447X |
| authorships[2].author.display_name | Jian Yang |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Yang, Jian |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5103072326 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Jiajun Shi |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Shi, Jiajun |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5103028950 |
| authorships[4].author.orcid | |
| authorships[4].author.display_name | He Zhu |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Zhu, He |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5034592186 |
| authorships[5].author.orcid | https://orcid.org/0000-0002-6684-0120 |
| authorships[5].author.display_name | Liran Wang |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Wang, Liran |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5103444312 |
| authorships[6].author.orcid | |
| authorships[6].author.display_name | Ke Jin |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Jin, Ke |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5100723667 |
| authorships[7].author.orcid | https://orcid.org/0000-0001-5450-6056 |
| authorships[7].author.display_name | Wei Zhang |
| authorships[7].author_position | middle |
| authorships[7].raw_author_name | Zhang, Wei |
| authorships[7].is_corresponding | False |
| authorships[8].author.id | https://openalex.org/A5060984708 |
| authorships[8].author.orcid | https://orcid.org/0000-0001-6647-1532 |
| authorships[8].author.display_name | H H Zhu |
| authorships[8].author_position | middle |
| authorships[8].raw_author_name | Zhu, Hualei |
| authorships[8].is_corresponding | False |
| authorships[9].author.id | https://openalex.org/A5104280301 |
| authorships[9].author.orcid | |
| authorships[9].author.display_name | Shuyue Guo |
| authorships[9].author_position | middle |
| authorships[9].raw_author_name | Guo, Shuyue |
| authorships[9].is_corresponding | False |
| authorships[10].author.id | https://openalex.org/A5100639224 |
| authorships[10].author.orcid | https://orcid.org/0000-0002-4667-1737 |
| authorships[10].author.display_name | Tao Sun |
| authorships[10].author_position | middle |
| authorships[10].raw_author_name | Sun, Tao |
| authorships[10].is_corresponding | False |
| authorships[11].author.id | https://openalex.org/A5032858379 |
| authorships[11].author.orcid | https://orcid.org/0000-0002-5183-8538 |
| authorships[11].author.display_name | Jiaheng Liu |
| authorships[11].author_position | middle |
| authorships[11].raw_author_name | Liu, Jiaheng |
| authorships[11].is_corresponding | False |
| authorships[12].author.id | https://openalex.org/A5063150466 |
| authorships[12].author.orcid | https://orcid.org/0000-0002-6229-9461 |
| authorships[12].author.display_name | Yi Duan |
| authorships[12].author_position | middle |
| authorships[12].raw_author_name | Duan, Yunlong |
| authorships[12].is_corresponding | False |
| authorships[13].author.id | https://openalex.org/A5028660878 |
| authorships[13].author.orcid | https://orcid.org/0000-0002-9113-2425 |
| authorships[13].author.display_name | Hao Yu |
| authorships[13].author_position | middle |
| authorships[13].raw_author_name | Hao, Yu |
| authorships[13].is_corresponding | False |
| authorships[14].author.id | https://openalex.org/A5101750528 |
| authorships[14].author.orcid | https://orcid.org/0000-0002-5498-3474 |
| authorships[14].author.display_name | Liqun Yang |
| authorships[14].author_position | middle |
| authorships[14].raw_author_name | Yang, Liqun |
| authorships[14].is_corresponding | False |
| authorships[15].author.id | https://openalex.org/A5054092813 |
| authorships[15].author.orcid | |
| authorships[15].author.display_name | Guanglin Niu |
| authorships[15].author_position | middle |
| authorships[15].raw_author_name | Niu, Guanglin |
| authorships[15].is_corresponding | False |
| authorships[16].author.id | https://openalex.org/A5100326077 |
| authorships[16].author.orcid | https://orcid.org/0000-0003-0704-0657 |
| authorships[16].author.display_name | Ge Zhang |
| authorships[16].author_position | middle |
| authorships[16].raw_author_name | Zhang, Ge |
| authorships[16].is_corresponding | False |
| authorships[17].author.id | https://openalex.org/A5036786337 |
| authorships[17].author.orcid | https://orcid.org/0000-0002-9603-9713 |
| authorships[17].author.display_name | Zhoujun Li |
| authorships[17].author_position | last |
| authorships[17].raw_author_name | Li, Zhoujun |
| authorships[17].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2411.02310 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | MdEval: Massively Multilingual Code Debugging |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10181 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9909999966621399 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Natural Language Processing Techniques |
| related_works | https://openalex.org/W2061417947, https://openalex.org/W2023839151, https://openalex.org/W1774183074, https://openalex.org/W2057488824, https://openalex.org/W2334687145, https://openalex.org/W2178011914, https://openalex.org/W4235962491, https://openalex.org/W2061778832, https://openalex.org/W1513001507, https://openalex.org/W2257153718 |
| cited_by_count | 1 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 1 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2411.02310 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2411.02310 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2411.02310 |
| primary_location.id | pmh:oai:arXiv.org:2411.02310 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2411.02310 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2411.02310 |
| publication_date | 2024-11-04 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 132, 140, 149, 174 |
| abstract_inverted_index.18 | 90 |
| abstract_inverted_index.To | 67 |
| abstract_inverted_index.as | 139 |
| abstract_inverted_index.by | 12, 120 |
| abstract_inverted_index.in | 9, 58, 158, 165, 194 |
| abstract_inverted_index.of | 28, 44, 60, 71, 89, 148, 152 |
| abstract_inverted_index.on | 19, 52, 137, 171 |
| abstract_inverted_index.to | 39, 144 |
| abstract_inverted_index.we | 76, 113 |
| abstract_inverted_index.C). | 167 |
| abstract_inverted_index.GPT | 185 |
| abstract_inverted_index.Our | 168 |
| abstract_inverted_index.and | 32, 54, 65, 93, 106, 128, 161, 181, 186 |
| abstract_inverted_index.are | 37, 55 |
| abstract_inverted_index.bug | 108 |
| abstract_inverted_index.for | 192 |
| abstract_inverted_index.gap | 177 |
| abstract_inverted_index.the | 15, 20, 41, 69, 78, 95, 101, 107, 115, 124, 146 |
| abstract_inverted_index.(BI) | 110 |
| abstract_inverted_index.(CR) | 104 |
| abstract_inverted_index.3.6K | 86 |
| abstract_inverted_index.Code | 0 |
| abstract_inverted_index.LLMs | 183 |
| abstract_inverted_index.Mut" | 157 |
| abstract_inverted_index.Rust | 160 |
| abstract_inverted_index.bugs | 122, 147 |
| abstract_inverted_index.code | 10, 17, 22, 30, 102, 196 |
| abstract_inverted_index.have | 5 |
| abstract_inverted_index.huge | 190 |
| abstract_inverted_index.into | 123 |
| abstract_inverted_index.made | 6 |
| abstract_inverted_index.many | 47 |
| abstract_inverted_index.room | 191 |
| abstract_inverted_index.test | 35, 87 |
| abstract_inverted_index.used | 38 |
| abstract_inverted_index.wide | 150 |
| abstract_inverted_index.with | 74 |
| abstract_inverted_index.(APR) | 99 |
| abstract_inverted_index.(e.g. | 155 |
| abstract_inverted_index.LLMs, | 75 |
| abstract_inverted_index.LLMs. | 45 |
| abstract_inverted_index.Macro | 163 |
| abstract_inverted_index.based | 18 |
| abstract_inverted_index.buggy | 21, 29 |
| abstract_inverted_index.field | 70 |
| abstract_inverted_index.first | 79 |
| abstract_inverted_index.focus | 51 |
| abstract_inverted_index.large | 1 |
| abstract_inverted_index.often | 56 |
| abstract_inverted_index.range | 151 |
| abstract_inverted_index.task, | 100, 105 |
| abstract_inverted_index.task. | 111 |
| abstract_inverted_index.terms | 59 |
| abstract_inverted_index.their | 33 |
| abstract_inverted_index.which | 84 |
| abstract_inverted_index.(LLMs) | 4 |
| abstract_inverted_index.(e.g., | 63, 184 |
| abstract_inverted_index.Claude | 187 |
| abstract_inverted_index.MDEVAL | 172 |
| abstract_inverted_index.Python | 53 |
| abstract_inverted_index.assess | 40 |
| abstract_inverted_index.cases, | 36 |
| abstract_inverted_index.covers | 94 |
| abstract_inverted_index.handle | 145 |
| abstract_inverted_index.models | 3, 180 |
| abstract_inverted_index.repair | 98 |
| abstract_inverted_index.reveal | 173 |
| abstract_inverted_index.review | 103 |
| abstract_inverted_index.strong | 141 |
| abstract_inverted_index.advance | 68 |
| abstract_inverted_index.between | 178 |
| abstract_inverted_index.corpora | 118 |
| abstract_inverted_index.correct | 16, 125 |
| abstract_inverted_index.limited | 57 |
| abstract_inverted_index.notable | 175 |
| abstract_inverted_index.program | 97 |
| abstract_inverted_index.propose | 77 |
| abstract_inverted_index.queries | 127 |
| abstract_inverted_index.samples | 88 |
| abstract_inverted_index.snippet | 31 |
| abstract_inverted_index.trained | 136 |
| abstract_inverted_index."Missing | 156 |
| abstract_inverted_index."Misused | 162 |
| abstract_inverted_index.Further, | 112, 131 |
| abstract_inverted_index.However, | 46 |
| abstract_inverted_index.baseline | 142 |
| abstract_inverted_index.debugger | 134 |
| abstract_inverted_index.directly | 13 |
| abstract_inverted_index.existing | 48 |
| abstract_inverted_index.includes | 85 |
| abstract_inverted_index.language | 2, 61, 159, 166 |
| abstract_inverted_index.progress | 8 |
| abstract_inverted_index.series), | 188 |
| abstract_inverted_index.snippet. | 23 |
| abstract_inverted_index.automated | 96 |
| abstract_inverted_index.debugging | 11, 42, 73, 82, 116, 197 |
| abstract_inverted_index.diversity | 62 |
| abstract_inverted_index.extensive | 169 |
| abstract_inverted_index.injecting | 121 |
| abstract_inverted_index.introduce | 114 |
| abstract_inverted_index.languages | 92, 154 |
| abstract_inverted_index.massively | 80 |
| abstract_inverted_index.primarily | 50 |
| abstract_inverted_index.solutions | 129 |
| abstract_inverted_index.typically | 26 |
| abstract_inverted_index.DebugBench | 64 |
| abstract_inverted_index.associated | 34 |
| abstract_inverted_index.benchmark, | 83 |
| abstract_inverted_index.benchmarks | 49 |
| abstract_inverted_index.consisting | 27 |
| abstract_inverted_index.generating | 14 |
| abstract_inverted_index.scenarios. | 198 |
| abstract_inverted_index.DebugEval). | 66 |
| abstract_inverted_index.Definition" | 164 |
| abstract_inverted_index.Programming | 24 |
| abstract_inverted_index.benchmarks, | 25 |
| abstract_inverted_index.experiments | 170 |
| abstract_inverted_index.improvement | 193 |
| abstract_inverted_index.instruction | 117 |
| abstract_inverted_index.open-source | 179 |
| abstract_inverted_index.performance | 176 |
| abstract_inverted_index.programming | 91, 153 |
| abstract_inverted_index.significant | 7 |
| abstract_inverted_index.xDebugCoder | 135 |
| abstract_inverted_index.(xDebugGen). | 130 |
| abstract_inverted_index.capabilities | 43 |
| abstract_inverted_index.highlighting | 189 |
| abstract_inverted_index.multilingual | 72, 81, 126, 133, 195 |
| abstract_inverted_index.specifically | 143 |
| abstract_inverted_index.closed-source | 182 |
| abstract_inverted_index.identification | 109 |
| abstract_inverted_index.MDEVAL-INSTRUCT | 119, 138 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 18 |
| citation_normalized_percentile |