Enhancing Pre-Trained Language Models for Vulnerability Detection via Semantic-Preserving Data Augmentation Article Swipe
Wen Qi
,
Jiahao Cao
,
Debasis Poddar
,
Sophia Li
,
Xinda Wang
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2410.00249
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2410.00249
With the rapid development and widespread use of advanced network systems, software vulnerabilities pose a significant threat to secure communications and networking. Learning-based vulnerability detection systems, particularly those leveraging pre-trained language models, have demonstrated significant potential in promptly identifying vulnerabilities in communication networks and reducing the risk of exploitation. However, the shortage of accurately labeled vulnerability datasets hinders further progress in this field. Failing to represent real-world vulnerability data variety and preserve vulnerability semantics, existing augmentation approaches provide limited or even counterproductive contributions to model training. In this paper, we propose a data augmentation technique aimed at enhancing the performance of pre-trained language models for vulnerability detection. Given the vulnerability dataset, our method performs natural semantic-preserving program transformation to generate a large volume of new samples with enriched data diversity and variety. By incorporating our augmented dataset in fine-tuning a series of representative code pre-trained models (i.e., CodeBERT, GraphCodeBERT, UnixCoder, and PDBERT), up to 10.1% increase in accuracy and 23.6% increase in F1 can be achieved in the vulnerability detection task. Comparison results also show that our proposed method can substantially outperform other prominent vulnerability augmentation approaches.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2410.00249
- https://arxiv.org/pdf/2410.00249
- OA Status
- green
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4403853701
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4403853701Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2410.00249Digital Object Identifier
- Title
-
Enhancing Pre-Trained Language Models for Vulnerability Detection via Semantic-Preserving Data AugmentationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-09-30Full publication date if available
- Authors
-
Wen Qi, Jiahao Cao, Debasis Poddar, Sophia Li, Xinda WangList of authors in order
- Landing page
-
https://arxiv.org/abs/2410.00249Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2410.00249Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2410.00249Direct OA link when available
- Concepts
-
Computer science, Vulnerability (computing), Natural language processing, Artificial intelligence, Computer securityTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4403853701 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2410.00249 |
| ids.doi | https://doi.org/10.48550/arxiv.2410.00249 |
| ids.openalex | https://openalex.org/W4403853701 |
| fwci | |
| type | preprint |
| title | Enhancing Pre-Trained Language Models for Vulnerability Detection via Semantic-Preserving Data Augmentation |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T12127 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9120000004768372 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1705 |
| topics[0].subfield.display_name | Computer Networks and Communications |
| topics[0].display_name | Software System Performance and Reliability |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.6697226762771606 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C95713431 |
| concepts[1].level | 2 |
| concepts[1].score | 0.6135843992233276 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q631425 |
| concepts[1].display_name | Vulnerability (computing) |
| concepts[2].id | https://openalex.org/C204321447 |
| concepts[2].level | 1 |
| concepts[2].score | 0.5962981581687927 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[2].display_name | Natural language processing |
| concepts[3].id | https://openalex.org/C154945302 |
| concepts[3].level | 1 |
| concepts[3].score | 0.5238566398620605 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[3].display_name | Artificial intelligence |
| concepts[4].id | https://openalex.org/C38652104 |
| concepts[4].level | 1 |
| concepts[4].score | 0.1566924750804901 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q3510521 |
| concepts[4].display_name | Computer security |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.6697226762771606 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/vulnerability |
| keywords[1].score | 0.6135843992233276 |
| keywords[1].display_name | Vulnerability (computing) |
| keywords[2].id | https://openalex.org/keywords/natural-language-processing |
| keywords[2].score | 0.5962981581687927 |
| keywords[2].display_name | Natural language processing |
| keywords[3].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[3].score | 0.5238566398620605 |
| keywords[3].display_name | Artificial intelligence |
| keywords[4].id | https://openalex.org/keywords/computer-security |
| keywords[4].score | 0.1566924750804901 |
| keywords[4].display_name | Computer security |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2410.00249 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2410.00249 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2410.00249 |
| locations[1].id | doi:10.48550/arxiv.2410.00249 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2410.00249 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5035835836 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-2091-3718 |
| authorships[0].author.display_name | Wen Qi |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Qi, Weiliang |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5031181114 |
| authorships[1].author.orcid | https://orcid.org/0009-0005-7729-5446 |
| authorships[1].author.display_name | Jiahao Cao |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Cao, Jiahao |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5029820307 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-5257-0342 |
| authorships[2].author.display_name | Debasis Poddar |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Poddar, Darsh |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5107212341 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | Sophia Li |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Li, Sophia |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5102758573 |
| authorships[4].author.orcid | https://orcid.org/0009-0003-1463-432X |
| authorships[4].author.display_name | Xinda Wang |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Wang, Xinda |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2410.00249 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2024-10-29T00:00:00 |
| display_name | Enhancing Pre-Trained Language Models for Vulnerability Detection via Semantic-Preserving Data Augmentation |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T12127 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9120000004768372 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1705 |
| primary_topic.subfield.display_name | Computer Networks and Communications |
| primary_topic.display_name | Software System Performance and Reliability |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W3204019825 |
| cited_by_count | 0 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2410.00249 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2410.00249 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2410.00249 |
| primary_location.id | pmh:oai:arXiv.org:2410.00249 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2410.00249 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2410.00249 |
| publication_date | 2024-09-30 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 14, 91, 120, 139 |
| abstract_inverted_index.By | 132 |
| abstract_inverted_index.F1 | 162 |
| abstract_inverted_index.In | 86 |
| abstract_inverted_index.at | 96 |
| abstract_inverted_index.be | 164 |
| abstract_inverted_index.in | 36, 40, 60, 137, 156, 161, 166 |
| abstract_inverted_index.of | 7, 47, 52, 100, 123, 141 |
| abstract_inverted_index.or | 79 |
| abstract_inverted_index.to | 17, 64, 83, 118, 153 |
| abstract_inverted_index.up | 152 |
| abstract_inverted_index.we | 89 |
| abstract_inverted_index.and | 4, 20, 43, 70, 130, 150, 158 |
| abstract_inverted_index.can | 163, 179 |
| abstract_inverted_index.for | 104 |
| abstract_inverted_index.new | 124 |
| abstract_inverted_index.our | 111, 134, 176 |
| abstract_inverted_index.the | 1, 45, 50, 98, 108, 167 |
| abstract_inverted_index.use | 6 |
| abstract_inverted_index.With | 0 |
| abstract_inverted_index.also | 173 |
| abstract_inverted_index.code | 143 |
| abstract_inverted_index.data | 68, 92, 128 |
| abstract_inverted_index.even | 80 |
| abstract_inverted_index.have | 32 |
| abstract_inverted_index.pose | 13 |
| abstract_inverted_index.risk | 46 |
| abstract_inverted_index.show | 174 |
| abstract_inverted_index.that | 175 |
| abstract_inverted_index.this | 61, 87 |
| abstract_inverted_index.with | 126 |
| abstract_inverted_index.10.1% | 154 |
| abstract_inverted_index.23.6% | 159 |
| abstract_inverted_index.Given | 107 |
| abstract_inverted_index.aimed | 95 |
| abstract_inverted_index.large | 121 |
| abstract_inverted_index.model | 84 |
| abstract_inverted_index.other | 182 |
| abstract_inverted_index.rapid | 2 |
| abstract_inverted_index.task. | 170 |
| abstract_inverted_index.those | 27 |
| abstract_inverted_index.(i.e., | 146 |
| abstract_inverted_index.field. | 62 |
| abstract_inverted_index.method | 112, 178 |
| abstract_inverted_index.models | 103, 145 |
| abstract_inverted_index.paper, | 88 |
| abstract_inverted_index.secure | 18 |
| abstract_inverted_index.series | 140 |
| abstract_inverted_index.threat | 16 |
| abstract_inverted_index.volume | 122 |
| abstract_inverted_index.Failing | 63 |
| abstract_inverted_index.dataset | 136 |
| abstract_inverted_index.further | 58 |
| abstract_inverted_index.hinders | 57 |
| abstract_inverted_index.labeled | 54 |
| abstract_inverted_index.limited | 78 |
| abstract_inverted_index.models, | 31 |
| abstract_inverted_index.natural | 114 |
| abstract_inverted_index.network | 9 |
| abstract_inverted_index.program | 116 |
| abstract_inverted_index.propose | 90 |
| abstract_inverted_index.provide | 77 |
| abstract_inverted_index.results | 172 |
| abstract_inverted_index.samples | 125 |
| abstract_inverted_index.variety | 69 |
| abstract_inverted_index.However, | 49 |
| abstract_inverted_index.PDBERT), | 151 |
| abstract_inverted_index.accuracy | 157 |
| abstract_inverted_index.achieved | 165 |
| abstract_inverted_index.advanced | 8 |
| abstract_inverted_index.dataset, | 110 |
| abstract_inverted_index.datasets | 56 |
| abstract_inverted_index.enriched | 127 |
| abstract_inverted_index.existing | 74 |
| abstract_inverted_index.generate | 119 |
| abstract_inverted_index.increase | 155, 160 |
| abstract_inverted_index.language | 30, 102 |
| abstract_inverted_index.networks | 42 |
| abstract_inverted_index.performs | 113 |
| abstract_inverted_index.preserve | 71 |
| abstract_inverted_index.progress | 59 |
| abstract_inverted_index.promptly | 37 |
| abstract_inverted_index.proposed | 177 |
| abstract_inverted_index.reducing | 44 |
| abstract_inverted_index.shortage | 51 |
| abstract_inverted_index.software | 11 |
| abstract_inverted_index.systems, | 10, 25 |
| abstract_inverted_index.variety. | 131 |
| abstract_inverted_index.CodeBERT, | 147 |
| abstract_inverted_index.augmented | 135 |
| abstract_inverted_index.detection | 24, 169 |
| abstract_inverted_index.diversity | 129 |
| abstract_inverted_index.enhancing | 97 |
| abstract_inverted_index.potential | 35 |
| abstract_inverted_index.prominent | 183 |
| abstract_inverted_index.represent | 65 |
| abstract_inverted_index.technique | 94 |
| abstract_inverted_index.training. | 85 |
| abstract_inverted_index.Comparison | 171 |
| abstract_inverted_index.UnixCoder, | 149 |
| abstract_inverted_index.accurately | 53 |
| abstract_inverted_index.approaches | 76 |
| abstract_inverted_index.detection. | 106 |
| abstract_inverted_index.leveraging | 28 |
| abstract_inverted_index.outperform | 181 |
| abstract_inverted_index.real-world | 66 |
| abstract_inverted_index.semantics, | 73 |
| abstract_inverted_index.widespread | 5 |
| abstract_inverted_index.approaches. | 186 |
| abstract_inverted_index.development | 3 |
| abstract_inverted_index.fine-tuning | 138 |
| abstract_inverted_index.identifying | 38 |
| abstract_inverted_index.networking. | 21 |
| abstract_inverted_index.performance | 99 |
| abstract_inverted_index.pre-trained | 29, 101, 144 |
| abstract_inverted_index.significant | 15, 34 |
| abstract_inverted_index.augmentation | 75, 93, 185 |
| abstract_inverted_index.demonstrated | 33 |
| abstract_inverted_index.particularly | 26 |
| abstract_inverted_index.communication | 41 |
| abstract_inverted_index.contributions | 82 |
| abstract_inverted_index.exploitation. | 48 |
| abstract_inverted_index.incorporating | 133 |
| abstract_inverted_index.substantially | 180 |
| abstract_inverted_index.vulnerability | 23, 55, 67, 72, 105, 109, 168, 184 |
| abstract_inverted_index.GraphCodeBERT, | 148 |
| abstract_inverted_index.Learning-based | 22 |
| abstract_inverted_index.communications | 19 |
| abstract_inverted_index.representative | 142 |
| abstract_inverted_index.transformation | 117 |
| abstract_inverted_index.vulnerabilities | 12, 39 |
| abstract_inverted_index.counterproductive | 81 |
| abstract_inverted_index.semantic-preserving | 115 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |