Optimizing Large Language Model Hyperparameters for Code Generation Article Swipe
Chetan Arora
,
Ahnaf Ibn Sayeed
,
Sherlock A. Licorish
,
Fanyu Wang
,
Christoph Treude
·
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2408.10577
YOU?
·
· 2024
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2408.10577
Large Language Models (LLMs), such as GPT models, are increasingly used in software engineering for various tasks, such as code generation, requirements management, and debugging. While automating these tasks has garnered significant attention, a systematic study on the impact of varying hyperparameters on code generation outcomes remains unexplored. This study aims to assess LLMs' code generation performance by exhaustively exploring the impact of various hyperparameters. Hyperparameters for LLMs are adjustable settings that affect the model's behaviour and performance. Specifically, we investigated how changes to the hyperparameters: temperature, top probability (top_p), frequency penalty, and presence penalty affect code generation outcomes. We systematically adjusted all hyperparameters together, exploring every possible combination by making small increments to each hyperparameter at a time. This exhaustive approach was applied to 13 Python code generation tasks, yielding one of four outcomes for each hyperparameter combination: no output from the LLM, non executable code, code that fails unit tests, or correct and functional code. We analysed these outcomes for a total of 14,742 generated Python code segments, focusing on correctness, to determine how the hyperparameters influence the LLM to arrive at each outcome. Using correlation coefficient and regression tree analyses, we ascertained which hyperparameters influence which aspect of the LLM. Our results indicate that optimal performance is achieved with a temperature below 0.5, top probability below 0.75, frequency penalty above -1 and below 1.5, and presence penalty above -1. We make our dataset and results available to facilitate replication.
Related Topics
Concepts
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- http://arxiv.org/abs/2408.10577
- https://arxiv.org/pdf/2408.10577
- OA Status
- green
- Cited By
- 3
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4403007121
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4403007121Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2408.10577Digital Object Identifier
- Title
-
Optimizing Large Language Model Hyperparameters for Code GenerationWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-08-20Full publication date if available
- Authors
-
Chetan Arora, Ahnaf Ibn Sayeed, Sherlock A. Licorish, Fanyu Wang, Christoph TreudeList of authors in order
- Landing page
-
https://arxiv.org/abs/2408.10577Publisher landing page
- PDF URL
-
https://arxiv.org/pdf/2408.10577Direct link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://arxiv.org/pdf/2408.10577Direct OA link when available
- Concepts
-
Hyperparameter, Code (set theory), Computer science, Language model, Programming language, Natural language processing, Artificial intelligence, Set (abstract data type)Top concepts (fields/topics) attached by OpenAlex
- Cited by
-
3Total citation count in OpenAlex
- Citations by year (recent)
-
2025: 3Per-year citation counts (last 5 years)
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4403007121 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2408.10577 |
| ids.doi | https://doi.org/10.48550/arxiv.2408.10577 |
| ids.openalex | https://openalex.org/W4403007121 |
| fwci | |
| type | preprint |
| title | Optimizing Large Language Model Hyperparameters for Code Generation |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10181 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9965000152587891 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1702 |
| topics[0].subfield.display_name | Artificial Intelligence |
| topics[0].display_name | Natural Language Processing Techniques |
| topics[1].id | https://openalex.org/T10028 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9625999927520752 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1702 |
| topics[1].subfield.display_name | Artificial Intelligence |
| topics[1].display_name | Topic Modeling |
| topics[2].id | https://openalex.org/T12031 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.9531000256538391 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1702 |
| topics[2].subfield.display_name | Artificial Intelligence |
| topics[2].display_name | Speech and dialogue systems |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C8642999 |
| concepts[0].level | 2 |
| concepts[0].score | 0.6759831309318542 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q4171168 |
| concepts[0].display_name | Hyperparameter |
| concepts[1].id | https://openalex.org/C2776760102 |
| concepts[1].level | 3 |
| concepts[1].score | 0.5958221554756165 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q5139990 |
| concepts[1].display_name | Code (set theory) |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.594634473323822 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C137293760 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5010919570922852 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q3621696 |
| concepts[3].display_name | Language model |
| concepts[4].id | https://openalex.org/C199360897 |
| concepts[4].level | 1 |
| concepts[4].score | 0.4656817317008972 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[4].display_name | Programming language |
| concepts[5].id | https://openalex.org/C204321447 |
| concepts[5].level | 1 |
| concepts[5].score | 0.3415507674217224 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q30642 |
| concepts[5].display_name | Natural language processing |
| concepts[6].id | https://openalex.org/C154945302 |
| concepts[6].level | 1 |
| concepts[6].score | 0.2751818597316742 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[6].display_name | Artificial intelligence |
| concepts[7].id | https://openalex.org/C177264268 |
| concepts[7].level | 2 |
| concepts[7].score | 0.0 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q1514741 |
| concepts[7].display_name | Set (abstract data type) |
| keywords[0].id | https://openalex.org/keywords/hyperparameter |
| keywords[0].score | 0.6759831309318542 |
| keywords[0].display_name | Hyperparameter |
| keywords[1].id | https://openalex.org/keywords/code |
| keywords[1].score | 0.5958221554756165 |
| keywords[1].display_name | Code (set theory) |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.594634473323822 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/language-model |
| keywords[3].score | 0.5010919570922852 |
| keywords[3].display_name | Language model |
| keywords[4].id | https://openalex.org/keywords/programming-language |
| keywords[4].score | 0.4656817317008972 |
| keywords[4].display_name | Programming language |
| keywords[5].id | https://openalex.org/keywords/natural-language-processing |
| keywords[5].score | 0.3415507674217224 |
| keywords[5].display_name | Natural language processing |
| keywords[6].id | https://openalex.org/keywords/artificial-intelligence |
| keywords[6].score | 0.2751818597316742 |
| keywords[6].display_name | Artificial intelligence |
| language | en |
| locations[0].id | pmh:oai:arXiv.org:2408.10577 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | https://arxiv.org/pdf/2408.10577 |
| locations[0].version | submittedVersion |
| locations[0].raw_type | text |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | http://arxiv.org/abs/2408.10577 |
| locations[1].id | doi:10.48550/arxiv.2408.10577 |
| locations[1].is_oa | True |
| locations[1].source.id | https://openalex.org/S4306400194 |
| locations[1].source.issn | |
| locations[1].source.type | repository |
| locations[1].source.is_oa | True |
| locations[1].source.issn_l | |
| locations[1].source.is_core | False |
| locations[1].source.is_in_doaj | False |
| locations[1].source.display_name | arXiv (Cornell University) |
| locations[1].source.host_organization | https://openalex.org/I205783295 |
| locations[1].source.host_organization_name | Cornell University |
| locations[1].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[1].license | cc-by |
| locations[1].pdf_url | |
| locations[1].version | |
| locations[1].raw_type | article |
| locations[1].license_id | https://openalex.org/licenses/cc-by |
| locations[1].is_accepted | False |
| locations[1].is_published | |
| locations[1].raw_source_name | |
| locations[1].landing_page_url | https://doi.org/10.48550/arxiv.2408.10577 |
| indexed_in | arxiv, datacite |
| authorships[0].author.id | https://openalex.org/A5019739552 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-1466-7386 |
| authorships[0].author.display_name | Chetan Arora |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Arora, Chetan |
| authorships[0].is_corresponding | False |
| authorships[1].author.id | https://openalex.org/A5107672041 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Ahnaf Ibn Sayeed |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Sayeed, Ahnaf Ibn |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | https://openalex.org/A5011391821 |
| authorships[2].author.orcid | https://orcid.org/0000-0001-7318-2421 |
| authorships[2].author.display_name | Sherlock A. Licorish |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Licorish, Sherlock |
| authorships[2].is_corresponding | False |
| authorships[3].author.id | https://openalex.org/A5108050054 |
| authorships[3].author.orcid | https://orcid.org/0000-0002-9937-8534 |
| authorships[3].author.display_name | Fanyu Wang |
| authorships[3].author_position | middle |
| authorships[3].raw_author_name | Wang, Fanyu |
| authorships[3].is_corresponding | False |
| authorships[4].author.id | https://openalex.org/A5077658936 |
| authorships[4].author.orcid | https://orcid.org/0000-0002-6919-2149 |
| authorships[4].author.display_name | Christoph Treude |
| authorships[4].author_position | last |
| authorships[4].raw_author_name | Treude, Christoph |
| authorships[4].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://arxiv.org/pdf/2408.10577 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Optimizing Large Language Model Hyperparameters for Code Generation |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T06:51:31.235846 |
| primary_topic.id | https://openalex.org/T10181 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9965000152587891 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1702 |
| primary_topic.subfield.display_name | Artificial Intelligence |
| primary_topic.display_name | Natural Language Processing Techniques |
| related_works | https://openalex.org/W2140186469, https://openalex.org/W4390421286, https://openalex.org/W4280563792, https://openalex.org/W4389724018, https://openalex.org/W4318719684, https://openalex.org/W3183136280, https://openalex.org/W4318559728, https://openalex.org/W2775233965, https://openalex.org/W4360995913, https://openalex.org/W4312193868 |
| cited_by_count | 3 |
| counts_by_year[0].year | 2025 |
| counts_by_year[0].cited_by_count | 3 |
| locations_count | 2 |
| best_oa_location.id | pmh:oai:arXiv.org:2408.10577 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | https://arxiv.org/pdf/2408.10577 |
| best_oa_location.version | submittedVersion |
| best_oa_location.raw_type | text |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | http://arxiv.org/abs/2408.10577 |
| primary_location.id | pmh:oai:arXiv.org:2408.10577 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | https://arxiv.org/pdf/2408.10577 |
| primary_location.version | submittedVersion |
| primary_location.raw_type | text |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | http://arxiv.org/abs/2408.10577 |
| publication_date | 2024-08-20 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 33, 117, 162, 212 |
| abstract_inverted_index.-1 | 223 |
| abstract_inverted_index.13 | 125 |
| abstract_inverted_index.We | 99, 157, 232 |
| abstract_inverted_index.as | 5, 18 |
| abstract_inverted_index.at | 116, 183 |
| abstract_inverted_index.by | 57, 109 |
| abstract_inverted_index.in | 11 |
| abstract_inverted_index.is | 209 |
| abstract_inverted_index.no | 139 |
| abstract_inverted_index.of | 39, 62, 132, 164, 200 |
| abstract_inverted_index.on | 36, 42, 171 |
| abstract_inverted_index.or | 152 |
| abstract_inverted_index.to | 51, 83, 113, 124, 173, 181, 239 |
| abstract_inverted_index.we | 79, 193 |
| abstract_inverted_index.-1. | 231 |
| abstract_inverted_index.GPT | 6 |
| abstract_inverted_index.LLM | 180 |
| abstract_inverted_index.Our | 203 |
| abstract_inverted_index.all | 102 |
| abstract_inverted_index.and | 23, 76, 92, 154, 189, 224, 227, 236 |
| abstract_inverted_index.are | 8, 68 |
| abstract_inverted_index.for | 14, 66, 135, 161 |
| abstract_inverted_index.has | 29 |
| abstract_inverted_index.how | 81, 175 |
| abstract_inverted_index.non | 144 |
| abstract_inverted_index.one | 131 |
| abstract_inverted_index.our | 234 |
| abstract_inverted_index.the | 37, 60, 73, 84, 142, 176, 179, 201 |
| abstract_inverted_index.top | 87, 216 |
| abstract_inverted_index.was | 122 |
| abstract_inverted_index.0.5, | 215 |
| abstract_inverted_index.1.5, | 226 |
| abstract_inverted_index.LLM, | 143 |
| abstract_inverted_index.LLM. | 202 |
| abstract_inverted_index.LLMs | 67 |
| abstract_inverted_index.This | 48, 119 |
| abstract_inverted_index.aims | 50 |
| abstract_inverted_index.code | 19, 43, 54, 96, 127, 147, 168 |
| abstract_inverted_index.each | 114, 136, 184 |
| abstract_inverted_index.four | 133 |
| abstract_inverted_index.from | 141 |
| abstract_inverted_index.make | 233 |
| abstract_inverted_index.such | 4, 17 |
| abstract_inverted_index.that | 71, 148, 206 |
| abstract_inverted_index.tree | 191 |
| abstract_inverted_index.unit | 150 |
| abstract_inverted_index.used | 10 |
| abstract_inverted_index.with | 211 |
| abstract_inverted_index.0.75, | 219 |
| abstract_inverted_index.LLMs' | 53 |
| abstract_inverted_index.Large | 0 |
| abstract_inverted_index.Using | 186 |
| abstract_inverted_index.While | 25 |
| abstract_inverted_index.above | 222, 230 |
| abstract_inverted_index.below | 214, 218, 225 |
| abstract_inverted_index.code, | 146 |
| abstract_inverted_index.code. | 156 |
| abstract_inverted_index.every | 106 |
| abstract_inverted_index.fails | 149 |
| abstract_inverted_index.small | 111 |
| abstract_inverted_index.study | 35, 49 |
| abstract_inverted_index.tasks | 28 |
| abstract_inverted_index.these | 27, 159 |
| abstract_inverted_index.time. | 118 |
| abstract_inverted_index.total | 163 |
| abstract_inverted_index.which | 195, 198 |
| abstract_inverted_index.14,742 | 165 |
| abstract_inverted_index.Models | 2 |
| abstract_inverted_index.Python | 126, 167 |
| abstract_inverted_index.affect | 72, 95 |
| abstract_inverted_index.arrive | 182 |
| abstract_inverted_index.aspect | 199 |
| abstract_inverted_index.assess | 52 |
| abstract_inverted_index.impact | 38, 61 |
| abstract_inverted_index.making | 110 |
| abstract_inverted_index.output | 140 |
| abstract_inverted_index.tasks, | 16, 129 |
| abstract_inverted_index.tests, | 151 |
| abstract_inverted_index.(LLMs), | 3 |
| abstract_inverted_index.applied | 123 |
| abstract_inverted_index.changes | 82 |
| abstract_inverted_index.correct | 153 |
| abstract_inverted_index.dataset | 235 |
| abstract_inverted_index.model's | 74 |
| abstract_inverted_index.models, | 7 |
| abstract_inverted_index.optimal | 207 |
| abstract_inverted_index.penalty | 94, 221, 229 |
| abstract_inverted_index.remains | 46 |
| abstract_inverted_index.results | 204, 237 |
| abstract_inverted_index.various | 15, 63 |
| abstract_inverted_index.varying | 40 |
| abstract_inverted_index.(top_p), | 89 |
| abstract_inverted_index.Language | 1 |
| abstract_inverted_index.achieved | 210 |
| abstract_inverted_index.adjusted | 101 |
| abstract_inverted_index.analysed | 158 |
| abstract_inverted_index.approach | 121 |
| abstract_inverted_index.focusing | 170 |
| abstract_inverted_index.garnered | 30 |
| abstract_inverted_index.indicate | 205 |
| abstract_inverted_index.outcome. | 185 |
| abstract_inverted_index.outcomes | 45, 134, 160 |
| abstract_inverted_index.penalty, | 91 |
| abstract_inverted_index.possible | 107 |
| abstract_inverted_index.presence | 93, 228 |
| abstract_inverted_index.settings | 70 |
| abstract_inverted_index.software | 12 |
| abstract_inverted_index.yielding | 130 |
| abstract_inverted_index.analyses, | 192 |
| abstract_inverted_index.available | 238 |
| abstract_inverted_index.behaviour | 75 |
| abstract_inverted_index.determine | 174 |
| abstract_inverted_index.exploring | 59, 105 |
| abstract_inverted_index.frequency | 90, 220 |
| abstract_inverted_index.generated | 166 |
| abstract_inverted_index.influence | 178, 197 |
| abstract_inverted_index.outcomes. | 98 |
| abstract_inverted_index.segments, | 169 |
| abstract_inverted_index.together, | 104 |
| abstract_inverted_index.adjustable | 69 |
| abstract_inverted_index.attention, | 32 |
| abstract_inverted_index.automating | 26 |
| abstract_inverted_index.debugging. | 24 |
| abstract_inverted_index.executable | 145 |
| abstract_inverted_index.exhaustive | 120 |
| abstract_inverted_index.facilitate | 240 |
| abstract_inverted_index.functional | 155 |
| abstract_inverted_index.generation | 44, 55, 97, 128 |
| abstract_inverted_index.increments | 112 |
| abstract_inverted_index.regression | 190 |
| abstract_inverted_index.systematic | 34 |
| abstract_inverted_index.ascertained | 194 |
| abstract_inverted_index.coefficient | 188 |
| abstract_inverted_index.combination | 108 |
| abstract_inverted_index.correlation | 187 |
| abstract_inverted_index.engineering | 13 |
| abstract_inverted_index.generation, | 20 |
| abstract_inverted_index.management, | 22 |
| abstract_inverted_index.performance | 56, 208 |
| abstract_inverted_index.probability | 88, 217 |
| abstract_inverted_index.significant | 31 |
| abstract_inverted_index.temperature | 213 |
| abstract_inverted_index.unexplored. | 47 |
| abstract_inverted_index.combination: | 138 |
| abstract_inverted_index.correctness, | 172 |
| abstract_inverted_index.exhaustively | 58 |
| abstract_inverted_index.increasingly | 9 |
| abstract_inverted_index.investigated | 80 |
| abstract_inverted_index.performance. | 77 |
| abstract_inverted_index.replication. | 241 |
| abstract_inverted_index.requirements | 21 |
| abstract_inverted_index.temperature, | 86 |
| abstract_inverted_index.Specifically, | 78 |
| abstract_inverted_index.hyperparameter | 115, 137 |
| abstract_inverted_index.systematically | 100 |
| abstract_inverted_index.Hyperparameters | 65 |
| abstract_inverted_index.hyperparameters | 41, 103, 177, 196 |
| abstract_inverted_index.hyperparameters. | 64 |
| abstract_inverted_index.hyperparameters: | 85 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 5 |
| citation_normalized_percentile |