Leveraging Large Language Models to Improve REST API Testing Article Swipe
Myeongsoo Kim
,
Tyler Stennett
,
Dhruv Shah
,
Saurabh Sinha
,
Alessandro Orso
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.1145/3639476.3639769
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.1145/3639476.3639769
The widespread adoption of REST APIs, coupled with their growing complexity and size, has led to the need for automated REST API testing tools. Current tools focus on the structured data in REST API specifications but often neglect valuable insights available in unstructured natural-language descriptions in the specifications, which leads to suboptimal test coverage. Recently, to address this gap, researchers have developed techniques that extract rules from these human-readable descriptions and query knowledge bases to derive meaningful input values. However, these techniques are limited in the types of rules they can extract and prone to produce inaccurate results. This paper presents RESTGPT, an innovative approach that leverages the power and intrinsic context-awareness of Large Language Models (LLMs) to improve REST API testing. RESTGPT takes as input an API specification, extracts machine-interpretable rules, and generates example parameter values from natural-language descriptions in the specification. It then augments the original specification with these rules and values. Our evaluations indicate that RESTGPT outperforms existing techniques in both rule extraction and value generation. Given these promising results, we outline future research directions for advancing REST API testing through LLMs.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.1145/3639476.3639769
- https://dl.acm.org/doi/pdf/10.1145/3639476.3639769
- OA Status
- gold
- Cited By
- 21
- References
- 18
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4398766360
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4398766360Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.1145/3639476.3639769Digital Object Identifier
- Title
-
Leveraging Large Language Models to Improve REST API TestingWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-04-14Full publication date if available
- Authors
-
Myeongsoo Kim, Tyler Stennett, Dhruv Shah, Saurabh Sinha, Alessandro OrsoList of authors in order
- Landing page
-
https://doi.org/10.1145/3639476.3639769Publisher landing page
- PDF URL
-
https://dl.acm.org/doi/pdf/10.1145/3639476.3639769Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://dl.acm.org/doi/pdf/10.1145/3639476.3639769Direct OA link when available
- Concepts
-
Computer science, Rest (music), Programming language, Software engineering, Cardiology, MedicineTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
21Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 18, 2024: 3Per-year citation counts (last 5 years)
- References (count)
-
18Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4398766360 |
|---|---|
| doi | https://doi.org/10.1145/3639476.3639769 |
| ids.doi | https://doi.org/10.1145/3639476.3639769 |
| ids.openalex | https://openalex.org/W4398766360 |
| fwci | 22.2586751 |
| type | article |
| title | Leveraging Large Language Models to Improve REST API Testing |
| awards[0].id | https://openalex.org/G1594933795 |
| awards[0].funder_id | https://openalex.org/F4320306076 |
| awards[0].display_name | |
| awards[0].funder_award_id | CCF-0725202 |
| awards[0].funder_display_name | National Science Foundation |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | 41 |
| biblio.first_page | 37 |
| topics[0].id | https://openalex.org/T10743 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9977999925613403 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1712 |
| topics[0].subfield.display_name | Software |
| topics[0].display_name | Software Testing and Debugging Techniques |
| topics[1].id | https://openalex.org/T11719 |
| topics[1].field.id | https://openalex.org/fields/18 |
| topics[1].field.display_name | Decision Sciences |
| topics[1].score | 0.9976999759674072 |
| topics[1].domain.id | https://openalex.org/domains/2 |
| topics[1].domain.display_name | Social Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1803 |
| topics[1].subfield.display_name | Management Science and Operations Research |
| topics[1].display_name | Data Quality and Management |
| topics[2].id | https://openalex.org/T12127 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9973000288009644 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1705 |
| topics[2].subfield.display_name | Computer Networks and Communications |
| topics[2].display_name | Software System Performance and Reliability |
| funders[0].id | https://openalex.org/F4320306076 |
| funders[0].ror | https://ror.org/021nxhr62 |
| funders[0].display_name | National Science Foundation |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.780173659324646 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C77265313 |
| concepts[1].level | 2 |
| concepts[1].score | 0.6371039748191833 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q879844 |
| concepts[1].display_name | Rest (music) |
| concepts[2].id | https://openalex.org/C199360897 |
| concepts[2].level | 1 |
| concepts[2].score | 0.5078571438789368 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[2].display_name | Programming language |
| concepts[3].id | https://openalex.org/C115903868 |
| concepts[3].level | 1 |
| concepts[3].score | 0.39631387591362 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q80993 |
| concepts[3].display_name | Software engineering |
| concepts[4].id | https://openalex.org/C164705383 |
| concepts[4].level | 1 |
| concepts[4].score | 0.0 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q10379 |
| concepts[4].display_name | Cardiology |
| concepts[5].id | https://openalex.org/C71924100 |
| concepts[5].level | 0 |
| concepts[5].score | 0.0 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q11190 |
| concepts[5].display_name | Medicine |
| keywords[0].id | https://openalex.org/keywords/computer-science |
| keywords[0].score | 0.780173659324646 |
| keywords[0].display_name | Computer science |
| keywords[1].id | https://openalex.org/keywords/rest |
| keywords[1].score | 0.6371039748191833 |
| keywords[1].display_name | Rest (music) |
| keywords[2].id | https://openalex.org/keywords/programming-language |
| keywords[2].score | 0.5078571438789368 |
| keywords[2].display_name | Programming language |
| keywords[3].id | https://openalex.org/keywords/software-engineering |
| keywords[3].score | 0.39631387591362 |
| keywords[3].display_name | Software engineering |
| language | en |
| locations[0].id | doi:10.1145/3639476.3639769 |
| locations[0].is_oa | True |
| locations[0].source | |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://dl.acm.org/doi/pdf/10.1145/3639476.3639769 |
| locations[0].version | publishedVersion |
| locations[0].raw_type | proceedings-article |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results |
| locations[0].landing_page_url | https://doi.org/10.1145/3639476.3639769 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5101736356 |
| authorships[0].author.orcid | https://orcid.org/0000-0002-5018-5280 |
| authorships[0].author.display_name | Myeongsoo Kim |
| authorships[0].countries | US |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I130701444 |
| authorships[0].affiliations[0].raw_affiliation_string | Georgia Institute of Technology, Atlanta, Georgia, United States |
| authorships[0].institutions[0].id | https://openalex.org/I130701444 |
| authorships[0].institutions[0].ror | https://ror.org/01zkghx44 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I130701444 |
| authorships[0].institutions[0].country_code | US |
| authorships[0].institutions[0].display_name | Georgia Institute of Technology |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Myeongsoo Kim |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | Georgia Institute of Technology, Atlanta, Georgia, United States |
| authorships[1].author.id | https://openalex.org/A5093426970 |
| authorships[1].author.orcid | https://orcid.org/0009-0006-9780-9608 |
| authorships[1].author.display_name | Tyler Stennett |
| authorships[1].countries | US |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I130701444 |
| authorships[1].affiliations[0].raw_affiliation_string | Georgia Institute of Technology, Atlanta, Georgia, USA |
| authorships[1].institutions[0].id | https://openalex.org/I130701444 |
| authorships[1].institutions[0].ror | https://ror.org/01zkghx44 |
| authorships[1].institutions[0].type | education |
| authorships[1].institutions[0].lineage | https://openalex.org/I130701444 |
| authorships[1].institutions[0].country_code | US |
| authorships[1].institutions[0].display_name | Georgia Institute of Technology |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Tyler Stennett |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | Georgia Institute of Technology, Atlanta, Georgia, USA |
| authorships[2].author.id | https://openalex.org/A5082341210 |
| authorships[2].author.orcid | https://orcid.org/0009-0003-8141-5856 |
| authorships[2].author.display_name | Dhruv Shah |
| authorships[2].countries | US |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I130701444 |
| authorships[2].affiliations[0].raw_affiliation_string | Georgia Institute of Technology, Atlanta, Georgia, USA |
| authorships[2].institutions[0].id | https://openalex.org/I130701444 |
| authorships[2].institutions[0].ror | https://ror.org/01zkghx44 |
| authorships[2].institutions[0].type | education |
| authorships[2].institutions[0].lineage | https://openalex.org/I130701444 |
| authorships[2].institutions[0].country_code | US |
| authorships[2].institutions[0].display_name | Georgia Institute of Technology |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Dhruv Shah |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Georgia Institute of Technology, Atlanta, Georgia, USA |
| authorships[3].author.id | https://openalex.org/A5103241577 |
| authorships[3].author.orcid | https://orcid.org/0000-0003-4092-2643 |
| authorships[3].author.display_name | Saurabh Sinha |
| authorships[3].affiliations[0].raw_affiliation_string | IBM Research, Yorktown Heights, New York, United States |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Saurabh Sinha |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | IBM Research, Yorktown Heights, New York, United States |
| authorships[4].author.id | https://openalex.org/A5030961858 |
| authorships[4].author.orcid | https://orcid.org/0000-0003-4516-9320 |
| authorships[4].author.display_name | Alessandro Orso |
| authorships[4].countries | US |
| authorships[4].affiliations[0].institution_ids | https://openalex.org/I130701444 |
| authorships[4].affiliations[0].raw_affiliation_string | Georgia Institute of Technology, Atlanta, Georgia, United States |
| authorships[4].institutions[0].id | https://openalex.org/I130701444 |
| authorships[4].institutions[0].ror | https://ror.org/01zkghx44 |
| authorships[4].institutions[0].type | education |
| authorships[4].institutions[0].lineage | https://openalex.org/I130701444 |
| authorships[4].institutions[0].country_code | US |
| authorships[4].institutions[0].display_name | Georgia Institute of Technology |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Alessandro Orso |
| authorships[4].is_corresponding | False |
| authorships[4].raw_affiliation_strings | Georgia Institute of Technology, Atlanta, Georgia, United States |
| has_content.pdf | True |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://dl.acm.org/doi/pdf/10.1145/3639476.3639769 |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Leveraging Large Language Models to Improve REST API Testing |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10743 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9977999925613403 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1712 |
| primary_topic.subfield.display_name | Software |
| primary_topic.display_name | Software Testing and Debugging Techniques |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2748952813, https://openalex.org/W4388147713, https://openalex.org/W4205145096, https://openalex.org/W2977909229, https://openalex.org/W2359059303, https://openalex.org/W3194775706, https://openalex.org/W3110334367, https://openalex.org/W3126656380, https://openalex.org/W1598444294 |
| cited_by_count | 21 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 18 |
| counts_by_year[1].year | 2024 |
| counts_by_year[1].cited_by_count | 3 |
| locations_count | 1 |
| best_oa_location.id | doi:10.1145/3639476.3639769 |
| best_oa_location.is_oa | True |
| best_oa_location.source | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://dl.acm.org/doi/pdf/10.1145/3639476.3639769 |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | proceedings-article |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results |
| best_oa_location.landing_page_url | https://doi.org/10.1145/3639476.3639769 |
| primary_location.id | doi:10.1145/3639476.3639769 |
| primary_location.is_oa | True |
| primary_location.source | |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://dl.acm.org/doi/pdf/10.1145/3639476.3639769 |
| primary_location.version | publishedVersion |
| primary_location.raw_type | proceedings-article |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | Proceedings of the 2024 ACM/IEEE 44th International Conference on Software Engineering: New Ideas and Emerging Results |
| primary_location.landing_page_url | https://doi.org/10.1145/3639476.3639769 |
| publication_date | 2024-04-14 |
| publication_year | 2024 |
| referenced_works | https://openalex.org/W4213274161, https://openalex.org/W2997401484, https://openalex.org/W2955835447, https://openalex.org/W2153225416, https://openalex.org/W4207068811, https://openalex.org/W4386027284, https://openalex.org/W4312297578, https://openalex.org/W3047675717, https://openalex.org/W3173777979, https://openalex.org/W4384154515, https://openalex.org/W4388483057, https://openalex.org/W4224306538, https://openalex.org/W3185341429, https://openalex.org/W3177546869, https://openalex.org/W6949241347, https://openalex.org/W4221143046, https://openalex.org/W4284706564, https://openalex.org/W4376504003 |
| referenced_works_count | 18 |
| abstract_inverted_index.It | 143 |
| abstract_inverted_index.an | 102, 126 |
| abstract_inverted_index.as | 124 |
| abstract_inverted_index.in | 31, 41, 45, 84, 140, 162 |
| abstract_inverted_index.of | 3, 87, 112 |
| abstract_inverted_index.on | 27 |
| abstract_inverted_index.to | 15, 50, 55, 74, 94, 117 |
| abstract_inverted_index.we | 173 |
| abstract_inverted_index.API | 21, 33, 120, 127, 181 |
| abstract_inverted_index.Our | 154 |
| abstract_inverted_index.The | 0 |
| abstract_inverted_index.and | 11, 70, 92, 109, 132, 152, 166 |
| abstract_inverted_index.are | 82 |
| abstract_inverted_index.but | 35 |
| abstract_inverted_index.can | 90 |
| abstract_inverted_index.for | 18, 178 |
| abstract_inverted_index.has | 13 |
| abstract_inverted_index.led | 14 |
| abstract_inverted_index.the | 16, 28, 46, 85, 107, 141, 146 |
| abstract_inverted_index.REST | 4, 20, 32, 119, 180 |
| abstract_inverted_index.This | 98 |
| abstract_inverted_index.both | 163 |
| abstract_inverted_index.data | 30 |
| abstract_inverted_index.from | 66, 137 |
| abstract_inverted_index.gap, | 58 |
| abstract_inverted_index.have | 60 |
| abstract_inverted_index.need | 17 |
| abstract_inverted_index.rule | 164 |
| abstract_inverted_index.test | 52 |
| abstract_inverted_index.that | 63, 105, 157 |
| abstract_inverted_index.then | 144 |
| abstract_inverted_index.they | 89 |
| abstract_inverted_index.this | 57 |
| abstract_inverted_index.with | 7, 149 |
| abstract_inverted_index.APIs, | 5 |
| abstract_inverted_index.Given | 169 |
| abstract_inverted_index.LLMs. | 184 |
| abstract_inverted_index.Large | 113 |
| abstract_inverted_index.bases | 73 |
| abstract_inverted_index.focus | 26 |
| abstract_inverted_index.input | 77, 125 |
| abstract_inverted_index.leads | 49 |
| abstract_inverted_index.often | 36 |
| abstract_inverted_index.paper | 99 |
| abstract_inverted_index.power | 108 |
| abstract_inverted_index.prone | 93 |
| abstract_inverted_index.query | 71 |
| abstract_inverted_index.rules | 65, 88, 151 |
| abstract_inverted_index.size, | 12 |
| abstract_inverted_index.takes | 123 |
| abstract_inverted_index.their | 8 |
| abstract_inverted_index.these | 67, 80, 150, 170 |
| abstract_inverted_index.tools | 25 |
| abstract_inverted_index.types | 86 |
| abstract_inverted_index.value | 167 |
| abstract_inverted_index.which | 48 |
| abstract_inverted_index.(LLMs) | 116 |
| abstract_inverted_index.Models | 115 |
| abstract_inverted_index.derive | 75 |
| abstract_inverted_index.future | 175 |
| abstract_inverted_index.rules, | 131 |
| abstract_inverted_index.tools. | 23 |
| abstract_inverted_index.values | 136 |
| abstract_inverted_index.Current | 24 |
| abstract_inverted_index.RESTGPT | 122, 158 |
| abstract_inverted_index.address | 56 |
| abstract_inverted_index.coupled | 6 |
| abstract_inverted_index.example | 134 |
| abstract_inverted_index.extract | 64, 91 |
| abstract_inverted_index.growing | 9 |
| abstract_inverted_index.improve | 118 |
| abstract_inverted_index.limited | 83 |
| abstract_inverted_index.neglect | 37 |
| abstract_inverted_index.outline | 174 |
| abstract_inverted_index.produce | 95 |
| abstract_inverted_index.testing | 22, 182 |
| abstract_inverted_index.through | 183 |
| abstract_inverted_index.values. | 78, 153 |
| abstract_inverted_index.However, | 79 |
| abstract_inverted_index.Language | 114 |
| abstract_inverted_index.RESTGPT, | 101 |
| abstract_inverted_index.adoption | 2 |
| abstract_inverted_index.approach | 104 |
| abstract_inverted_index.augments | 145 |
| abstract_inverted_index.existing | 160 |
| abstract_inverted_index.extracts | 129 |
| abstract_inverted_index.indicate | 156 |
| abstract_inverted_index.insights | 39 |
| abstract_inverted_index.original | 147 |
| abstract_inverted_index.presents | 100 |
| abstract_inverted_index.research | 176 |
| abstract_inverted_index.results, | 172 |
| abstract_inverted_index.results. | 97 |
| abstract_inverted_index.testing. | 121 |
| abstract_inverted_index.valuable | 38 |
| abstract_inverted_index.Recently, | 54 |
| abstract_inverted_index.advancing | 179 |
| abstract_inverted_index.automated | 19 |
| abstract_inverted_index.available | 40 |
| abstract_inverted_index.coverage. | 53 |
| abstract_inverted_index.developed | 61 |
| abstract_inverted_index.generates | 133 |
| abstract_inverted_index.intrinsic | 110 |
| abstract_inverted_index.knowledge | 72 |
| abstract_inverted_index.leverages | 106 |
| abstract_inverted_index.parameter | 135 |
| abstract_inverted_index.promising | 171 |
| abstract_inverted_index.complexity | 10 |
| abstract_inverted_index.directions | 177 |
| abstract_inverted_index.extraction | 165 |
| abstract_inverted_index.inaccurate | 96 |
| abstract_inverted_index.innovative | 103 |
| abstract_inverted_index.meaningful | 76 |
| abstract_inverted_index.structured | 29 |
| abstract_inverted_index.suboptimal | 51 |
| abstract_inverted_index.techniques | 62, 81, 161 |
| abstract_inverted_index.widespread | 1 |
| abstract_inverted_index.evaluations | 155 |
| abstract_inverted_index.generation. | 168 |
| abstract_inverted_index.outperforms | 159 |
| abstract_inverted_index.researchers | 59 |
| abstract_inverted_index.descriptions | 44, 69, 139 |
| abstract_inverted_index.unstructured | 42 |
| abstract_inverted_index.specification | 148 |
| abstract_inverted_index.human-readable | 68 |
| abstract_inverted_index.specification, | 128 |
| abstract_inverted_index.specification. | 142 |
| abstract_inverted_index.specifications | 34 |
| abstract_inverted_index.specifications, | 47 |
| abstract_inverted_index.natural-language | 43, 138 |
| abstract_inverted_index.context-awareness | 111 |
| abstract_inverted_index.machine-interpretable | 130 |
| cited_by_percentile_year.max | 100 |
| cited_by_percentile_year.min | 96 |
| countries_distinct_count | 1 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile.value | 0.99007392 |
| citation_normalized_percentile.is_in_top_1_percent | True |
| citation_normalized_percentile.is_in_top_10_percent | True |