A Survey on Large Language Models for Software Engineering Article Swipe
Quanjun Zhang
,
Chunrong Fang
,
Yang Xie
,
Yaxin Zhang
,
Yun Yang
,
Weisong Sun
,
Shengcheng Yu
,
Zhenyu Chen
·
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2312.15223
YOU?
·
· 2023
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2312.15223
Software Engineering (SE) is the systematic design, development, maintenance, and management of software applications underpinning the digital infrastructure of our modern world. Very recently, the SE community has seen a rapidly increasing number of techniques employing Large Language Models (LLMs) to automate a broad range of SE tasks. Nevertheless, existing information of the applications, effects, and possible limitations of LLMs within SE is still not well-studied. In this paper, we provide a systematic survey to summarize the current state-of-the-art research in the LLM-based SE community. We summarize 62 representative LLMs of Code across three model architectures, 15 pre-training objectives across four categories, and 16 downstream tasks across five categories. We then present a detailed summarization of the recent SE studies for which LLMs are commonly utilized, including 947 studies for 112 specific code-related tasks across five crucial phases within the SE workflow. We also discuss several critical aspects during the integration of LLMs into SE, such as empirical evaluation, benchmarking, security and reliability, domain tuning, compressing and distillation. Finally, we highlight several challenges and potential opportunities on applying LLMs for future SE studies, such as exploring domain LLMs and constructing clean evaluation datasets. Overall, our work can help researchers gain a comprehensive understanding about the achievements of the existing LLM-based SE studies and promote the practical application of these techniques. Our artifacts are publicly available and will be continuously updated at the living repository: https://github.com/iSEngLab/AwesomeLLM4SE.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2312.15223
- https://arxiv.org/pdf/2312.15223
- OA Status
- green
- Cited By
- 15
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4390306128
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4390306128Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2312.15223Digital Object Identifier
- Title
-
A Survey on Large Language Models for Software EngineeringWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2023Year of publication
- Publication date
-
2023-12-23Full publication date if available
- Authors
-
Quanjun Zhang, Chunrong Fang, Yang Xie, Yaxin Zhang, Yun Yang, Weisong Sun, Shengcheng Yu, Zhenyu ChenList of authors in order
- Landing page
-
https://arxiv.org/abs/2312.15223Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2312.15223Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2312.15223Direct OA link when available
- Concepts
-
Computer science, Workflow, Data science, DatabaseTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
15Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 8, 2024: 7Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4390306128 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2312.15223 |
| ids.doi | https://doi.org/10.48550/arxiv.2312.15223 |
| ids.openalex | https://openalex.org/W4390306128 |
| fwci | |
| type | preprint |
| title | A Survey on Large Language Models for Software Engineering |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10260 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9991999864578247 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1710 |
| topics[0].subfield.display_name | Information Systems |
| topics[0].display_name | Software Engineering Research |
| topics[1].id | https://openalex.org/T10430 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9772999882698059 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1710 |
| topics[1].subfield.display_name | Information Systems |
| topics[1].display_name | Software Engineering Techniques and Practices |
| topics[2].id | https://openalex.org/T10028 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9645000100135803 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Topic Modeling |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.5839945673942566 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C177212765 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5006003379821777 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q627335 |
| concepts[1].display_name | Workflow |
| concepts[2].id | https://openalex.org/C2522767166 |
| concepts[2].level | 1 |
| concepts[2].score | 0.4402481019496918 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q2374463 |
| concepts[2].display_name | Data science |
| concepts[3].id | https://openalex.org/C77088390 |
| concepts[3].level | 1 |
| concepts[3].score | 0.08883562684059143 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q8513 |
| concepts[3].display_name | Database |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.5839945673942566 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/workflow |
| keywords[1].score | 0.5006003379821777 |
| keywords[1].display_name | Workflow |
| keywords[2].id | https://openalex.org/keywords/data-science |
| keywords[2].score | 0.4402481019496918 |
| keywords[2].display_name | Data science |
| keywords[3].id | https://openalex.org/keywords/database |
| keywords[3].score | 0.08883562684059143 |
| keywords[3].display_name | Database |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2312.15223 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2312.15223 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2312.15223 |
| locations[1].id | doi:10.48550/arxiv.2312.15223 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2312.15223 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5101756397 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-2495-3805 |
| authorships[0].author.display_name | Quanjun Zhang |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Zhang, Quanjun |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5075174750 |
| authorships[1].author.orcid | https://orcid.org/0000-0002-9930-7111 |
| authorships[1].author.display_name | Chunrong Fang |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Fang, Chunrong |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5102568298 |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Yang Xie |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Xie, Yang |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5100736914 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-2265-3614 |
| authorships[3].author.display_name | Yaxin Zhang |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Zhang, Yaxin |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5035343733 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-7868-5471 |
| authorships[4].author.display_name | Yun Yang |
| authorships[4].author_position | middle |
| authorships[4].raw_author_name | Yang, Yun |
| authorships[4].is_corresponding | False |
| authorships[5].author.id | https://openalex.org/A5013856385 |
| authorships[5].author.orcid | https://orcid.org/0000-0001-9236-8264 |
| authorships[5].author.display_name | Weisong Sun |
| authorships[5].author_position | middle |
| authorships[5].raw_author_name | Sun, Weisong |
| authorships[5].is_corresponding | False |
| authorships[6].author.id | https://openalex.org/A5101519109 |
| authorships[6].author.orcid | https://orcid.org/0000-0003-4640-8637 |
| authorships[6].author.display_name | Shengcheng Yu |
| authorships[6].author_position | middle |
| authorships[6].raw_author_name | Yu, Shengcheng |
| authorships[6].is_corresponding | False |
| authorships[7].author.id | https://openalex.org/A5100422935 |
| authorships[7].author.orcid | https://orcid.org/0000-0002-9592-7022 |
| authorships[7].author.display_name | Zhenyu Chen |
| authorships[7].author_position | last |
| authorships[7].raw_author_name | Chen, Zhenyu |
| authorships[7].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2312.15223 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | A Survey on Large Language Models for Software Engineering |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10260 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9991999864578247 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1710 |
| primary_topic.subfield.display_name | Information Systems |
| primary_topic.display_name | Software Engineering Research |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W1981780420, https://openalex.org/W2182707996, https://openalex.org/W45233828, https://openalex.org/W2964988449, https://openalex.org/W2397952901, https://openalex.org/W2029380707, https://openalex.org/W4255934811, https://openalex.org/W2465382974 |
| cited_by_count | 15 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 8 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 7 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2312.15223 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2312.15223 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2312.15223 |
| primary_location.id | pmh:oai:arXiv.org:2312.15223 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2312.15223 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2312.15223 |
| publication_date | 2023-12-23 |
| publication_year | 2023 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 29, 42, 71, 112, 200 |
| abstract_inverted_index.15 | 96 |
| abstract_inverted_index.16 | 103 |
| abstract_inverted_index.62 | 87 |
| abstract_inverted_index.In | 66 |
| abstract_inverted_index.SE | 25, 46, 61, 83, 118, 140, 181, 210 |
| abstract_inverted_index.We | 85, 109, 142 |
| abstract_inverted_index.as | 156, 184 |
| abstract_inverted_index.at | 230 |
| abstract_inverted_index.be | 227 |
| abstract_inverted_index.in | 80 |
| abstract_inverted_index.is | 3, 62 |
| abstract_inverted_index.of | 11, 18, 33, 45, 51, 58, 90, 115, 151, 206, 217 |
| abstract_inverted_index.on | 176 |
| abstract_inverted_index.to | 40, 74 |
| abstract_inverted_index.we | 69, 169 |
| abstract_inverted_index.112 | 130 |
| abstract_inverted_index.947 | 127 |
| abstract_inverted_index.Our | 220 |
| abstract_inverted_index.SE, | 154 |
| abstract_inverted_index.and | 9, 55, 102, 161, 166, 173, 188, 212, 225 |
| abstract_inverted_index.are | 123, 222 |
| abstract_inverted_index.can | 196 |
| abstract_inverted_index.for | 120, 129, 179 |
| abstract_inverted_index.has | 27 |
| abstract_inverted_index.not | 64 |
| abstract_inverted_index.our | 19, 194 |
| abstract_inverted_index.the | 4, 15, 24, 52, 76, 81, 116, 139, 149, 204, 207, 214, 231 |
| abstract_inverted_index.(SE) | 2 |
| abstract_inverted_index.Code | 91 |
| abstract_inverted_index.LLMs | 59, 89, 122, 152, 178, 187 |
| abstract_inverted_index.Very | 22 |
| abstract_inverted_index.also | 143 |
| abstract_inverted_index.five | 107, 135 |
| abstract_inverted_index.four | 100 |
| abstract_inverted_index.gain | 199 |
| abstract_inverted_index.help | 197 |
| abstract_inverted_index.into | 153 |
| abstract_inverted_index.seen | 28 |
| abstract_inverted_index.such | 155, 183 |
| abstract_inverted_index.then | 110 |
| abstract_inverted_index.this | 67 |
| abstract_inverted_index.will | 226 |
| abstract_inverted_index.work | 195 |
| abstract_inverted_index.Large | 36 |
| abstract_inverted_index.about | 203 |
| abstract_inverted_index.broad | 43 |
| abstract_inverted_index.clean | 190 |
| abstract_inverted_index.model | 94 |
| abstract_inverted_index.range | 44 |
| abstract_inverted_index.still | 63 |
| abstract_inverted_index.tasks | 105, 133 |
| abstract_inverted_index.these | 218 |
| abstract_inverted_index.three | 93 |
| abstract_inverted_index.which | 121 |
| abstract_inverted_index.(LLMs) | 39 |
| abstract_inverted_index.Models | 38 |
| abstract_inverted_index.across | 92, 99, 106, 134 |
| abstract_inverted_index.domain | 163, 186 |
| abstract_inverted_index.during | 148 |
| abstract_inverted_index.future | 180 |
| abstract_inverted_index.living | 232 |
| abstract_inverted_index.modern | 20 |
| abstract_inverted_index.number | 32 |
| abstract_inverted_index.paper, | 68 |
| abstract_inverted_index.phases | 137 |
| abstract_inverted_index.recent | 117 |
| abstract_inverted_index.survey | 73 |
| abstract_inverted_index.tasks. | 47 |
| abstract_inverted_index.within | 60, 138 |
| abstract_inverted_index.world. | 21 |
| abstract_inverted_index.aspects | 147 |
| abstract_inverted_index.crucial | 136 |
| abstract_inverted_index.current | 77 |
| abstract_inverted_index.design, | 6 |
| abstract_inverted_index.digital | 16 |
| abstract_inverted_index.discuss | 144 |
| abstract_inverted_index.present | 111 |
| abstract_inverted_index.promote | 213 |
| abstract_inverted_index.provide | 70 |
| abstract_inverted_index.rapidly | 30 |
| abstract_inverted_index.several | 145, 171 |
| abstract_inverted_index.studies | 119, 128, 211 |
| abstract_inverted_index.tuning, | 164 |
| abstract_inverted_index.updated | 229 |
| abstract_inverted_index.Finally, | 168 |
| abstract_inverted_index.Language | 37 |
| abstract_inverted_index.Overall, | 193 |
| abstract_inverted_index.Software | 0 |
| abstract_inverted_index.applying | 177 |
| abstract_inverted_index.automate | 41 |
| abstract_inverted_index.commonly | 124 |
| abstract_inverted_index.critical | 146 |
| abstract_inverted_index.detailed | 113 |
| abstract_inverted_index.effects, | 54 |
| abstract_inverted_index.existing | 49, 208 |
| abstract_inverted_index.possible | 56 |
| abstract_inverted_index.publicly | 223 |
| abstract_inverted_index.research | 79 |
| abstract_inverted_index.security | 160 |
| abstract_inverted_index.software | 12 |
| abstract_inverted_index.specific | 131 |
| abstract_inverted_index.studies, | 182 |
| abstract_inverted_index.LLM-based | 82, 209 |
| abstract_inverted_index.artifacts | 221 |
| abstract_inverted_index.available | 224 |
| abstract_inverted_index.community | 26 |
| abstract_inverted_index.datasets. | 192 |
| abstract_inverted_index.empirical | 157 |
| abstract_inverted_index.employing | 35 |
| abstract_inverted_index.exploring | 185 |
| abstract_inverted_index.highlight | 170 |
| abstract_inverted_index.including | 126 |
| abstract_inverted_index.potential | 174 |
| abstract_inverted_index.practical | 215 |
| abstract_inverted_index.recently, | 23 |
| abstract_inverted_index.summarize | 75, 86 |
| abstract_inverted_index.utilized, | 125 |
| abstract_inverted_index.workflow. | 141 |
| abstract_inverted_index.challenges | 172 |
| abstract_inverted_index.community. | 84 |
| abstract_inverted_index.downstream | 104 |
| abstract_inverted_index.evaluation | 191 |
| abstract_inverted_index.increasing | 31 |
| abstract_inverted_index.management | 10 |
| abstract_inverted_index.objectives | 98 |
| abstract_inverted_index.systematic | 5, 72 |
| abstract_inverted_index.techniques | 34 |
| abstract_inverted_index.Engineering | 1 |
| abstract_inverted_index.application | 216 |
| abstract_inverted_index.categories, | 101 |
| abstract_inverted_index.categories. | 108 |
| abstract_inverted_index.compressing | 165 |
| abstract_inverted_index.evaluation, | 158 |
| abstract_inverted_index.information | 50 |
| abstract_inverted_index.integration | 150 |
| abstract_inverted_index.limitations | 57 |
| abstract_inverted_index.repository: | 233 |
| abstract_inverted_index.researchers | 198 |
| abstract_inverted_index.techniques. | 219 |
| abstract_inverted_index.achievements | 205 |
| abstract_inverted_index.applications | 13 |
| abstract_inverted_index.code-related | 132 |
| abstract_inverted_index.constructing | 189 |
| abstract_inverted_index.continuously | 228 |
| abstract_inverted_index.development, | 7 |
| abstract_inverted_index.maintenance, | 8 |
| abstract_inverted_index.pre-training | 97 |
| abstract_inverted_index.reliability, | 162 |
| abstract_inverted_index.underpinning | 14 |
| abstract_inverted_index.Nevertheless, | 48 |
| abstract_inverted_index.applications, | 53 |
| abstract_inverted_index.benchmarking, | 159 |
| abstract_inverted_index.comprehensive | 201 |
| abstract_inverted_index.distillation. | 167 |
| abstract_inverted_index.opportunities | 175 |
| abstract_inverted_index.summarization | 114 |
| abstract_inverted_index.understanding | 202 |
| abstract_inverted_index.well-studied. | 65 |
| abstract_inverted_index.architectures, | 95 |
| abstract_inverted_index.infrastructure | 17 |
| abstract_inverted_index.representative | 88 |
| abstract_inverted_index.state-of-the-art | 78 |
| abstract_inverted_index.https://github.com/iSEngLab/AwesomeLLM4SE. | 234 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 8 |
| sustainable_development_goals[0].id | https://metadata.un.org/sdg/9 |
| sustainable_development_goals[0].score | 0.6299999952316284 |
| sustainable_development_goals[0].display_name | Industry, innovation and infrastructure |
| citation_normalized_percentile |