Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency Article Swipe
Kurzynski, Marco
,
Aga, Shaizeen
,
Wu, Di
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2512.08242
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.48550/arxiv.2512.08242
Training large language models (LLMs) efficiently requires a deep understanding of how modern GPU systems behave under real-world distributed training workloads. While prior work has focused primarily on kernel-level performance or single-GPU microbenchmarks, the complex interaction between communication, computation, memory behavior, and power management in multi-GPU LLM training remains poorly characterized. In this work, we introduce Chopper, a profiling and analysis framework that collects, aligns, and visualizes GPU kernel traces and hardware performance counters across multiple granularities (i.e., from individual kernels to operations, layers, phases, iterations, and GPUs). Using Chopper, we perform a comprehensive end-to-end characterization of Llama 3 8B training under fully sharded data parallelism (FSDP) on an eight-GPU AMD InstinctTM MI300X node. Our analysis reveals several previously underexplored bottlenecks and behaviors, such as memory determinism enabling higher, more stable GPU and memory frequencies. We identify several sources of inefficiencies, with frequency overhead (DVFS effects) being the single largest contributor to the gap between theoretical and observed performance, exceeding the impact of MFMA utilization loss, communication/computation overlap, and kernel launch overheads. Overall, Chopper provides the first holistic, multi-granularity characterization of LLM training on AMD InstinctTM MI300X GPUs, yielding actionable insights for optimizing training frameworks, improving power-management strategies, and guiding future GPU architecture and system design.
Related Topics
Concepts
Computer science
Profiling (computer programming)
Kernel (algebra)
Overhead (engineering)
Computer architecture
Inefficiency
Architecture
Artificial intelligence
Training (meteorology)
Parallel computing
Computer engineering
General-purpose computing on graphics processing units
Characterization (materials science)
Training set
Distributed computing
Embedded system
Identification (biology)
Machine learning
Data structure
Memory bandwidth
Visualization
Memory management
Metadata
- Type
- preprint
- Landing Page
- https://doi.org/10.48550/arxiv.2512.08242
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W7114788792
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W7114788792Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.48550/arxiv.2512.08242Digital Object Identifier
- Title
-
Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training InefficiencyWork title
- Type
-
preprintOpenAlex work type
- Publication year
-
2025Year of publication
- Publication date
-
2025-12-09Full publication date if available
- Authors
-
Kurzynski, Marco, Aga, Shaizeen, Wu, DiList of authors in order
- Landing page
-
https://doi.org/10.48550/arxiv.2512.08242Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://doi.org/10.48550/arxiv.2512.08242Direct OA link when available
- Concepts
-
Computer science, Profiling (computer programming), Kernel (algebra), Overhead (engineering), Computer architecture, Inefficiency, Architecture, Artificial intelligence, Training (meteorology), Parallel computing, Computer engineering, General-purpose computing on graphics processing units, Characterization (materials science), Training set, Distributed computing, Embedded system, Identification (biology), Machine learning, Data structure, Memory bandwidth, Visualization, Memory managementTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W7114788792 |
|---|---|
| doi | https://doi.org/10.48550/arxiv.2512.08242 |
| ids.doi | https://doi.org/10.48550/arxiv.2512.08242 |
| ids.openalex | https://openalex.org/W7114788792 |
| fwci | |
| type | preprint |
| title | Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C41008148 |
| concepts[0].level | 0 |
| concepts[0].score | 0.8315686583518982 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[0].display_name | Computer science |
| concepts[1].id | https://openalex.org/C187191949 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5436477661132812 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q1138496 |
| concepts[1].display_name | Profiling (computer programming) |
| concepts[2].id | https://openalex.org/C74193536 |
| concepts[2].level | 2 |
| concepts[2].score | 0.5054957866668701 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q574844 |
| concepts[2].display_name | Kernel (algebra) |
| concepts[3].id | https://openalex.org/C2779960059 |
| concepts[3].level | 2 |
| concepts[3].score | 0.4984370470046997 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q7113681 |
| concepts[3].display_name | Overhead (engineering) |
| concepts[4].id | https://openalex.org/C118524514 |
| concepts[4].level | 1 |
| concepts[4].score | 0.3855784237384796 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q173212 |
| concepts[4].display_name | Computer architecture |
| concepts[5].id | https://openalex.org/C2778869765 |
| concepts[5].level | 2 |
| concepts[5].score | 0.3821324408054352 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q6028363 |
| concepts[5].display_name | Inefficiency |
| concepts[6].id | https://openalex.org/C123657996 |
| concepts[6].level | 2 |
| concepts[6].score | 0.3572549521923065 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q12271 |
| concepts[6].display_name | Architecture |
| concepts[7].id | https://openalex.org/C154945302 |
| concepts[7].level | 1 |
| concepts[7].score | 0.3435896933078766 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[7].display_name | Artificial intelligence |
| concepts[8].id | https://openalex.org/C2777211547 |
| concepts[8].level | 2 |
| concepts[8].score | 0.33451583981513977 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q17141490 |
| concepts[8].display_name | Training (meteorology) |
| concepts[9].id | https://openalex.org/C173608175 |
| concepts[9].level | 1 |
| concepts[9].score | 0.3276750445365906 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q232661 |
| concepts[9].display_name | Parallel computing |
| concepts[10].id | https://openalex.org/C113775141 |
| concepts[10].level | 1 |
| concepts[10].score | 0.32120853662490845 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q428691 |
| concepts[10].display_name | Computer engineering |
| concepts[11].id | https://openalex.org/C50630238 |
| concepts[11].level | 3 |
| concepts[11].score | 0.3175252676010132 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q971505 |
| concepts[11].display_name | General-purpose computing on graphics processing units |
| concepts[12].id | https://openalex.org/C2780841128 |
| concepts[12].level | 2 |
| concepts[12].score | 0.31723564863204956 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q5073781 |
| concepts[12].display_name | Characterization (materials science) |
| concepts[13].id | https://openalex.org/C51632099 |
| concepts[13].level | 2 |
| concepts[13].score | 0.3136138617992401 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q3985153 |
| concepts[13].display_name | Training set |
| concepts[14].id | https://openalex.org/C120314980 |
| concepts[14].level | 1 |
| concepts[14].score | 0.31070569157600403 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q180634 |
| concepts[14].display_name | Distributed computing |
| concepts[15].id | https://openalex.org/C149635348 |
| concepts[15].level | 1 |
| concepts[15].score | 0.30127495527267456 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q193040 |
| concepts[15].display_name | Embedded system |
| concepts[16].id | https://openalex.org/C116834253 |
| concepts[16].level | 2 |
| concepts[16].score | 0.2951914966106415 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q2039217 |
| concepts[16].display_name | Identification (biology) |
| concepts[17].id | https://openalex.org/C119857082 |
| concepts[17].level | 1 |
| concepts[17].score | 0.2896936237812042 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[17].display_name | Machine learning |
| concepts[18].id | https://openalex.org/C162319229 |
| concepts[18].level | 2 |
| concepts[18].score | 0.28153085708618164 |
| concepts[18].wikidata | https://www.wikidata.org/wiki/Q175263 |
| concepts[18].display_name | Data structure |
| concepts[19].id | https://openalex.org/C188045654 |
| concepts[19].level | 2 |
| concepts[19].score | 0.28037041425704956 |
| concepts[19].wikidata | https://www.wikidata.org/wiki/Q17148339 |
| concepts[19].display_name | Memory bandwidth |
| concepts[20].id | https://openalex.org/C36464697 |
| concepts[20].level | 2 |
| concepts[20].score | 0.2595088481903076 |
| concepts[20].wikidata | https://www.wikidata.org/wiki/Q451553 |
| concepts[20].display_name | Visualization |
| concepts[21].id | https://openalex.org/C176649486 |
| concepts[21].level | 3 |
| concepts[21].score | 0.25766023993492126 |
| concepts[21].wikidata | https://www.wikidata.org/wiki/Q2308807 |
| concepts[21].display_name | Memory management |
| keywords[0].id | https://openalex.org/keywords/profiling |
| keywords[0].score | 0.5436477661132812 |
| keywords[0].display_name | Profiling (computer programming) |
| keywords[1].id | https://openalex.org/keywords/kernel |
| keywords[1].score | 0.5054957866668701 |
| keywords[1].display_name | Kernel (algebra) |
| keywords[2].id | https://openalex.org/keywords/overhead |
| keywords[2].score | 0.4984370470046997 |
| keywords[2].display_name | Overhead (engineering) |
| keywords[3].id | https://openalex.org/keywords/inefficiency |
| keywords[3].score | 0.3821324408054352 |
| keywords[3].display_name | Inefficiency |
| keywords[4].id | https://openalex.org/keywords/architecture |
| keywords[4].score | 0.3572549521923065 |
| keywords[4].display_name | Architecture |
| keywords[5].id | https://openalex.org/keywords/training |
| keywords[5].score | 0.33451583981513977 |
| keywords[5].display_name | Training (meteorology) |
| keywords[6].id | https://openalex.org/keywords/general-purpose-computing-on-graphics-processing-units |
| keywords[6].score | 0.3175252676010132 |
| keywords[6].display_name | General-purpose computing on graphics processing units |
| language | |
| locations[0].id | doi:10.48550/arxiv.2512.08242 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400194 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | arXiv (Cornell University) |
| locations[0].source.host_organization | https://openalex.org/I205783295 |
| locations[0].source.host_organization_name | Cornell University |
| locations[0].source.host_organization_lineage | https://openalex.org/I205783295 |
| locations[0].license | |
| locations[0].pdf_url | |
| locations[0].version | |
| locations[0].raw_type | article |
| locations[0].license_id | |
| locations[0].is_accepted | False |
| locations[0].is_published | |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://doi.org/10.48550/arxiv.2512.08242 |
| indexed_in | datacite |
| authorships[0].author.id | |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Kurzynski, Marco |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Kurzynski, Marco |
| authorships[0].is_corresponding | True |
| authorships[1].author.id | |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | Aga, Shaizeen |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Aga, Shaizeen |
| authorships[1].is_corresponding | False |
| authorships[2].author.id | |
| authorships[2].author.orcid | |
| authorships[2].author.display_name | Wu, Di |
| authorships[2].author_position | last |
| authorships[2].raw_author_name | Wu, Di |
| authorships[2].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://doi.org/10.48550/arxiv.2512.08242 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-12-11T00:00:00 |
| display_name | Chopper: A Multi-Level GPU Characterization Tool & Derived Insights Into LLM Training Inefficiency |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-12-11T23:13:37.075516 |
| primary_topic | |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.48550/arxiv.2512.08242 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400194 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | arXiv (Cornell University) |
| best_oa_location.source.host_organization | https://openalex.org/I205783295 |
| best_oa_location.source.host_organization_name | Cornell University |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| best_oa_location.license | |
| best_oa_location.pdf_url | |
| best_oa_location.version | |
| best_oa_location.raw_type | article |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.48550/arxiv.2512.08242 |
| primary_location.id | doi:10.48550/arxiv.2512.08242 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400194 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | arXiv (Cornell University) |
| primary_location.source.host_organization | https://openalex.org/I205783295 |
| primary_location.source.host_organization_name | Cornell University |
| primary_location.source.host_organization_lineage | https://openalex.org/I205783295 |
| primary_location.license | |
| primary_location.pdf_url | |
| primary_location.version | |
| primary_location.raw_type | article |
| primary_location.license_id | |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://doi.org/10.48550/arxiv.2512.08242 |
| publication_date | 2025-12-09 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.3 | 98 |
| abstract_inverted_index.a | 7, 57, 92 |
| abstract_inverted_index.8B | 99 |
| abstract_inverted_index.In | 51 |
| abstract_inverted_index.We | 135 |
| abstract_inverted_index.an | 108 |
| abstract_inverted_index.as | 124 |
| abstract_inverted_index.in | 44 |
| abstract_inverted_index.of | 10, 96, 139, 162, 180 |
| abstract_inverted_index.on | 27, 107, 183 |
| abstract_inverted_index.or | 30 |
| abstract_inverted_index.to | 81, 151 |
| abstract_inverted_index.we | 54, 90 |
| abstract_inverted_index.AMD | 110, 184 |
| abstract_inverted_index.GPU | 13, 67, 131, 201 |
| abstract_inverted_index.LLM | 46, 181 |
| abstract_inverted_index.Our | 114 |
| abstract_inverted_index.and | 41, 59, 65, 70, 86, 121, 132, 156, 168, 198, 203 |
| abstract_inverted_index.for | 191 |
| abstract_inverted_index.gap | 153 |
| abstract_inverted_index.has | 24 |
| abstract_inverted_index.how | 11 |
| abstract_inverted_index.the | 33, 147, 152, 160, 175 |
| abstract_inverted_index.MFMA | 163 |
| abstract_inverted_index.data | 104 |
| abstract_inverted_index.deep | 8 |
| abstract_inverted_index.from | 78 |
| abstract_inverted_index.more | 129 |
| abstract_inverted_index.such | 123 |
| abstract_inverted_index.that | 62 |
| abstract_inverted_index.this | 52 |
| abstract_inverted_index.with | 141 |
| abstract_inverted_index.work | 23 |
| abstract_inverted_index.(DVFS | 144 |
| abstract_inverted_index.GPUs, | 187 |
| abstract_inverted_index.Llama | 97 |
| abstract_inverted_index.Using | 88 |
| abstract_inverted_index.While | 21 |
| abstract_inverted_index.being | 146 |
| abstract_inverted_index.first | 176 |
| abstract_inverted_index.fully | 102 |
| abstract_inverted_index.large | 1 |
| abstract_inverted_index.loss, | 165 |
| abstract_inverted_index.node. | 113 |
| abstract_inverted_index.power | 42 |
| abstract_inverted_index.prior | 22 |
| abstract_inverted_index.under | 16, 101 |
| abstract_inverted_index.work, | 53 |
| abstract_inverted_index.(FSDP) | 106 |
| abstract_inverted_index.(LLMs) | 4 |
| abstract_inverted_index.(i.e., | 77 |
| abstract_inverted_index.GPUs). | 87 |
| abstract_inverted_index.MI300X | 112, 186 |
| abstract_inverted_index.across | 74 |
| abstract_inverted_index.behave | 15 |
| abstract_inverted_index.future | 200 |
| abstract_inverted_index.impact | 161 |
| abstract_inverted_index.kernel | 68, 169 |
| abstract_inverted_index.launch | 170 |
| abstract_inverted_index.memory | 39, 125, 133 |
| abstract_inverted_index.models | 3 |
| abstract_inverted_index.modern | 12 |
| abstract_inverted_index.poorly | 49 |
| abstract_inverted_index.single | 148 |
| abstract_inverted_index.stable | 130 |
| abstract_inverted_index.system | 204 |
| abstract_inverted_index.traces | 69 |
| abstract_inverted_index.Chopper | 173 |
| abstract_inverted_index.aligns, | 64 |
| abstract_inverted_index.between | 36, 154 |
| abstract_inverted_index.complex | 34 |
| abstract_inverted_index.design. | 205 |
| abstract_inverted_index.focused | 25 |
| abstract_inverted_index.guiding | 199 |
| abstract_inverted_index.higher, | 128 |
| abstract_inverted_index.kernels | 80 |
| abstract_inverted_index.largest | 149 |
| abstract_inverted_index.layers, | 83 |
| abstract_inverted_index.perform | 91 |
| abstract_inverted_index.phases, | 84 |
| abstract_inverted_index.remains | 48 |
| abstract_inverted_index.reveals | 116 |
| abstract_inverted_index.several | 117, 137 |
| abstract_inverted_index.sharded | 103 |
| abstract_inverted_index.sources | 138 |
| abstract_inverted_index.systems | 14 |
| abstract_inverted_index.Chopper, | 56, 89 |
| abstract_inverted_index.Overall, | 172 |
| abstract_inverted_index.Training | 0 |
| abstract_inverted_index.analysis | 60, 115 |
| abstract_inverted_index.counters | 73 |
| abstract_inverted_index.effects) | 145 |
| abstract_inverted_index.enabling | 127 |
| abstract_inverted_index.hardware | 71 |
| abstract_inverted_index.identify | 136 |
| abstract_inverted_index.insights | 190 |
| abstract_inverted_index.language | 2 |
| abstract_inverted_index.multiple | 75 |
| abstract_inverted_index.observed | 157 |
| abstract_inverted_index.overhead | 143 |
| abstract_inverted_index.overlap, | 167 |
| abstract_inverted_index.provides | 174 |
| abstract_inverted_index.requires | 6 |
| abstract_inverted_index.training | 19, 47, 100, 182, 193 |
| abstract_inverted_index.yielding | 188 |
| abstract_inverted_index.behavior, | 40 |
| abstract_inverted_index.collects, | 63 |
| abstract_inverted_index.eight-GPU | 109 |
| abstract_inverted_index.exceeding | 159 |
| abstract_inverted_index.framework | 61 |
| abstract_inverted_index.frequency | 142 |
| abstract_inverted_index.holistic, | 177 |
| abstract_inverted_index.improving | 195 |
| abstract_inverted_index.introduce | 55 |
| abstract_inverted_index.multi-GPU | 45 |
| abstract_inverted_index.primarily | 26 |
| abstract_inverted_index.profiling | 58 |
| abstract_inverted_index.InstinctTM | 111, 185 |
| abstract_inverted_index.actionable | 189 |
| abstract_inverted_index.behaviors, | 122 |
| abstract_inverted_index.end-to-end | 94 |
| abstract_inverted_index.individual | 79 |
| abstract_inverted_index.management | 43 |
| abstract_inverted_index.optimizing | 192 |
| abstract_inverted_index.overheads. | 171 |
| abstract_inverted_index.previously | 118 |
| abstract_inverted_index.real-world | 17 |
| abstract_inverted_index.single-GPU | 31 |
| abstract_inverted_index.visualizes | 66 |
| abstract_inverted_index.workloads. | 20 |
| abstract_inverted_index.bottlenecks | 120 |
| abstract_inverted_index.contributor | 150 |
| abstract_inverted_index.determinism | 126 |
| abstract_inverted_index.distributed | 18 |
| abstract_inverted_index.efficiently | 5 |
| abstract_inverted_index.frameworks, | 194 |
| abstract_inverted_index.interaction | 35 |
| abstract_inverted_index.iterations, | 85 |
| abstract_inverted_index.operations, | 82 |
| abstract_inverted_index.parallelism | 105 |
| abstract_inverted_index.performance | 29, 72 |
| abstract_inverted_index.strategies, | 197 |
| abstract_inverted_index.theoretical | 155 |
| abstract_inverted_index.utilization | 164 |
| abstract_inverted_index.architecture | 202 |
| abstract_inverted_index.computation, | 38 |
| abstract_inverted_index.frequencies. | 134 |
| abstract_inverted_index.kernel-level | 28 |
| abstract_inverted_index.performance, | 158 |
| abstract_inverted_index.comprehensive | 93 |
| abstract_inverted_index.granularities | 76 |
| abstract_inverted_index.underexplored | 119 |
| abstract_inverted_index.understanding | 9 |
| abstract_inverted_index.characterized. | 50 |
| abstract_inverted_index.communication, | 37 |
| abstract_inverted_index.inefficiencies, | 140 |
| abstract_inverted_index.characterization | 95, 179 |
| abstract_inverted_index.microbenchmarks, | 32 |
| abstract_inverted_index.power-management | 196 |
| abstract_inverted_index.multi-granularity | 178 |
| abstract_inverted_index.communication/computation | 166 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 3 |
| citation_normalized_percentile |