Scalable AI Through Hybrid Parallelism: Balancing Data and Model Distribution for Enhanced Performance Article Swipe
Revista, Zen
,
IA, 10
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.5281/zenodo.17819420
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.5281/zenodo.17819420
This paper investigates the challenges and opportunities in scaling Artificial Intelligence (AI) through hybrid parallelism, specifically focusing on balancing data and model distribution for enhanced performance. As AI models grow in size and complexity, the computational demands for training and inference become increasingly prohibitive. Hybrid parallelism, which combines data parallelism and model parallelism, offers a promising approach to address these scalability challenges. This study explores various strategies for implementing hybrid parallelism, including optimal data partitioning, efficient model sharding, and communication-reducing techniques. The performance of these strategies is evaluated across different AI models and hardware architectures, highlighting the trade-offs between computational efficiency, memory usage, and communication overhead. Furthermore, the paper presents a novel framework for dynamically adapting the degree of data and model parallelism based on the characteristics of the AI model, the size of the dataset, and the available computational resources. Experimental results demonstrate that this adaptive hybrid parallelism framework can significantly improve the scalability and performance of AI models compared to traditional data or model parallelism alone, enabling the training and deployment of larger and more complex AI models with reduced time and cost. The implications of these findings for the future of AI and the development of scalable AI systems are also discussed.
Related Topics
Concepts
Scalability
Computer science
Data parallelism
Parallelism (grammar)
Inference
Distributed computing
Software deployment
Artificial intelligence
Machine learning
Data modeling
Computational model
Hybrid system
Task parallelism
Applications of artificial intelligence
Big data
Load balancing (electrical power)
Parallel computing
Training set
Massively parallel
Scale (ratio)
Scaling
Parallel processing
Metadata
- Type
- article
- Landing Page
- https://doi.org/10.5281/zenodo.17819420
- OA Status
- green
- OpenAlex ID
- https://openalex.org/W7108711514
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W7108711514Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.5281/zenodo.17819420Digital Object Identifier
- Title
-
Scalable AI Through Hybrid Parallelism: Balancing Data and Model Distribution for Enhanced PerformanceWork title
- Type
-
articleOpenAlex work type
- Publication year
-
2025Year of publication
- Publication date
-
2025-12-04Full publication date if available
- Authors
-
Revista, Zen, IA, 10List of authors in order
- Landing page
-
https://doi.org/10.5281/zenodo.17819420Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
greenOpen access status per OpenAlex
- OA URL
-
https://doi.org/10.5281/zenodo.17819420Direct OA link when available
- Concepts
-
Scalability, Computer science, Data parallelism, Parallelism (grammar), Inference, Distributed computing, Software deployment, Artificial intelligence, Machine learning, Data modeling, Computational model, Hybrid system, Task parallelism, Applications of artificial intelligence, Big data, Load balancing (electrical power), Parallel computing, Training set, Massively parallel, Scale (ratio), Scaling, Parallel processingTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
Full payload
| id | https://openalex.org/W7108711514 |
|---|---|
| doi | https://doi.org/10.5281/zenodo.17819420 |
| ids.doi | https://doi.org/10.5281/zenodo.17819420 |
| ids.openalex | https://openalex.org/W7108711514 |
| fwci | 0.0 |
| type | article |
| title | Scalable AI Through Hybrid Parallelism: Balancing Data and Model Distribution for Enhanced Performance |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10036 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.4857456684112549 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1707 |
| topics[0].subfield.display_name | Computer Vision and Pattern Recognition |
| topics[0].display_name | Advanced Neural Network Applications |
| topics[1].id | https://openalex.org/T14347 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.1188260167837143 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1710 |
| topics[1].subfield.display_name | Information Systems |
| topics[1].display_name | Big Data and Digital Economy |
| topics[2].id | https://openalex.org/T10054 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.024957431480288506 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1708 |
| topics[2].subfield.display_name | Hardware and Architecture |
| topics[2].display_name | Parallel Computing and Optimization Techniques |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C48044578 |
| concepts[0].level | 2 |
| concepts[0].score | 0.8293203115463257 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q727490 |
| concepts[0].display_name | Scalability |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.8247496485710144 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C61483411 |
| concepts[2].level | 3 |
| concepts[2].score | 0.5751669406890869 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q3124522 |
| concepts[2].display_name | Data parallelism |
| concepts[3].id | https://openalex.org/C2781172179 |
| concepts[3].level | 2 |
| concepts[3].score | 0.528877317905426 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q853109 |
| concepts[3].display_name | Parallelism (grammar) |
| concepts[4].id | https://openalex.org/C2776214188 |
| concepts[4].level | 2 |
| concepts[4].score | 0.5211313366889954 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q408386 |
| concepts[4].display_name | Inference |
| concepts[5].id | https://openalex.org/C120314980 |
| concepts[5].level | 1 |
| concepts[5].score | 0.45957157015800476 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q180634 |
| concepts[5].display_name | Distributed computing |
| concepts[6].id | https://openalex.org/C105339364 |
| concepts[6].level | 2 |
| concepts[6].score | 0.4513451159000397 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q2297740 |
| concepts[6].display_name | Software deployment |
| concepts[7].id | https://openalex.org/C154945302 |
| concepts[7].level | 1 |
| concepts[7].score | 0.41886141896247864 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11660 |
| concepts[7].display_name | Artificial intelligence |
| concepts[8].id | https://openalex.org/C119857082 |
| concepts[8].level | 1 |
| concepts[8].score | 0.36036112904548645 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q2539 |
| concepts[8].display_name | Machine learning |
| concepts[9].id | https://openalex.org/C67186912 |
| concepts[9].level | 2 |
| concepts[9].score | 0.35226014256477356 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q367664 |
| concepts[9].display_name | Data modeling |
| concepts[10].id | https://openalex.org/C66024118 |
| concepts[10].level | 2 |
| concepts[10].score | 0.35218092799186707 |
| concepts[10].wikidata | https://www.wikidata.org/wiki/Q1122506 |
| concepts[10].display_name | Computational model |
| concepts[11].id | https://openalex.org/C50897621 |
| concepts[11].level | 2 |
| concepts[11].score | 0.338762104511261 |
| concepts[11].wikidata | https://www.wikidata.org/wiki/Q2665508 |
| concepts[11].display_name | Hybrid system |
| concepts[12].id | https://openalex.org/C42992933 |
| concepts[12].level | 3 |
| concepts[12].score | 0.3334060609340668 |
| concepts[12].wikidata | https://www.wikidata.org/wiki/Q691169 |
| concepts[12].display_name | Task parallelism |
| concepts[13].id | https://openalex.org/C157170001 |
| concepts[13].level | 2 |
| concepts[13].score | 0.31953224539756775 |
| concepts[13].wikidata | https://www.wikidata.org/wiki/Q4781507 |
| concepts[13].display_name | Applications of artificial intelligence |
| concepts[14].id | https://openalex.org/C75684735 |
| concepts[14].level | 2 |
| concepts[14].score | 0.30380627512931824 |
| concepts[14].wikidata | https://www.wikidata.org/wiki/Q858810 |
| concepts[14].display_name | Big data |
| concepts[15].id | https://openalex.org/C138959212 |
| concepts[15].level | 3 |
| concepts[15].score | 0.2963809370994568 |
| concepts[15].wikidata | https://www.wikidata.org/wiki/Q1806783 |
| concepts[15].display_name | Load balancing (electrical power) |
| concepts[16].id | https://openalex.org/C173608175 |
| concepts[16].level | 1 |
| concepts[16].score | 0.27669626474380493 |
| concepts[16].wikidata | https://www.wikidata.org/wiki/Q232661 |
| concepts[16].display_name | Parallel computing |
| concepts[17].id | https://openalex.org/C51632099 |
| concepts[17].level | 2 |
| concepts[17].score | 0.26432308554649353 |
| concepts[17].wikidata | https://www.wikidata.org/wiki/Q3985153 |
| concepts[17].display_name | Training set |
| concepts[18].id | https://openalex.org/C190475519 |
| concepts[18].level | 2 |
| concepts[18].score | 0.26259925961494446 |
| concepts[18].wikidata | https://www.wikidata.org/wiki/Q544384 |
| concepts[18].display_name | Massively parallel |
| concepts[19].id | https://openalex.org/C2778755073 |
| concepts[19].level | 2 |
| concepts[19].score | 0.2604476809501648 |
| concepts[19].wikidata | https://www.wikidata.org/wiki/Q10858537 |
| concepts[19].display_name | Scale (ratio) |
| concepts[20].id | https://openalex.org/C99844830 |
| concepts[20].level | 2 |
| concepts[20].score | 0.25833073258399963 |
| concepts[20].wikidata | https://www.wikidata.org/wiki/Q102441924 |
| concepts[20].display_name | Scaling |
| concepts[21].id | https://openalex.org/C106515295 |
| concepts[21].level | 2 |
| concepts[21].score | 0.25071778893470764 |
| concepts[21].wikidata | https://www.wikidata.org/wiki/Q26806595 |
| concepts[21].display_name | Parallel processing |
| keywords[0].id | https://openalex.org/keywords/scalability |
| keywords[0].score | 0.8293203115463257 |
| keywords[0].display_name | Scalability |
| keywords[1].id | https://openalex.org/keywords/data-parallelism |
| keywords[1].score | 0.5751669406890869 |
| keywords[1].display_name | Data parallelism |
| keywords[2].id | https://openalex.org/keywords/parallelism |
| keywords[2].score | 0.528877317905426 |
| keywords[2].display_name | Parallelism (grammar) |
| keywords[3].id | https://openalex.org/keywords/inference |
| keywords[3].score | 0.5211313366889954 |
| keywords[3].display_name | Inference |
| keywords[4].id | https://openalex.org/keywords/software-deployment |
| keywords[4].score | 0.4513451159000397 |
| keywords[4].display_name | Software deployment |
| keywords[5].id | https://openalex.org/keywords/data-modeling |
| keywords[5].score | 0.35226014256477356 |
| keywords[5].display_name | Data modeling |
| keywords[6].id | https://openalex.org/keywords/computational-model |
| keywords[6].score | 0.35218092799186707 |
| keywords[6].display_name | Computational model |
| keywords[7].id | https://openalex.org/keywords/hybrid-system |
| keywords[7].score | 0.338762104511261 |
| keywords[7].display_name | Hybrid system |
| language | |
| locations[0].id | doi:10.5281/zenodo.17819420 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S4306400562 |
| locations[0].source.issn | |
| locations[0].source.type | repository |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | Zenodo (CERN European Organization for Nuclear Research) |
| locations[0].source.host_organization | https://openalex.org/I67311998 |
| locations[0].source.host_organization_name | European Organization for Nuclear Research |
| locations[0].source.host_organization_lineage | https://openalex.org/I67311998 |
| locations[0].license | cc-by |
| locations[0].pdf_url | |
| locations[0].version | |
| locations[0].raw_type | article-journal |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | False |
| locations[0].is_published | |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://doi.org/10.5281/zenodo.17819420 |
| indexed_in | datacite |
| authorships[0].author.id | |
| authorships[0].author.orcid | |
| authorships[0].author.display_name | Revista, Zen |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Revista, Zen |
| authorships[0].is_corresponding | True |
| authorships[1].author.id | |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | IA, 10 |
| authorships[1].author_position | last |
| authorships[1].raw_author_name | IA, 10 |
| authorships[1].is_corresponding | False |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://doi.org/10.5281/zenodo.17819420 |
| open_access.oa_status | green |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-12-05T00:00:00 |
| display_name | Scalable AI Through Hybrid Parallelism: Balancing Data and Model Distribution for Enhanced Performance |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-12-05T23:25:22.460635 |
| primary_topic.id | https://openalex.org/T10036 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.4857456684112549 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1707 |
| primary_topic.subfield.display_name | Computer Vision and Pattern Recognition |
| primary_topic.display_name | Advanced Neural Network Applications |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.5281/zenodo.17819420 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S4306400562 |
| best_oa_location.source.issn | |
| best_oa_location.source.type | repository |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | Zenodo (CERN European Organization for Nuclear Research) |
| best_oa_location.source.host_organization | https://openalex.org/I67311998 |
| best_oa_location.source.host_organization_name | European Organization for Nuclear Research |
| best_oa_location.source.host_organization_lineage | https://openalex.org/I67311998 |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | |
| best_oa_location.version | |
| best_oa_location.raw_type | article-journal |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | False |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.5281/zenodo.17819420 |
| primary_location.id | doi:10.5281/zenodo.17819420 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S4306400562 |
| primary_location.source.issn | |
| primary_location.source.type | repository |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | Zenodo (CERN European Organization for Nuclear Research) |
| primary_location.source.host_organization | https://openalex.org/I67311998 |
| primary_location.source.host_organization_name | European Organization for Nuclear Research |
| primary_location.source.host_organization_lineage | https://openalex.org/I67311998 |
| primary_location.license | cc-by |
| primary_location.pdf_url | |
| primary_location.version | |
| primary_location.raw_type | article-journal |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | False |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://doi.org/10.5281/zenodo.17819420 |
| publication_date | 2025-12-04 |
| publication_year | 2025 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 54, 110 |
| abstract_inverted_index.AI | 27, 90, 129, 158, 178, 194, 200 |
| abstract_inverted_index.As | 26 |
| abstract_inverted_index.in | 7, 30 |
| abstract_inverted_index.is | 86 |
| abstract_inverted_index.of | 83, 118, 127, 133, 157, 173, 187, 193, 198 |
| abstract_inverted_index.on | 17, 124 |
| abstract_inverted_index.or | 164 |
| abstract_inverted_index.to | 57, 161 |
| abstract_inverted_index.The | 81, 185 |
| abstract_inverted_index.and | 5, 20, 32, 39, 50, 78, 92, 103, 120, 136, 155, 171, 175, 183, 195 |
| abstract_inverted_index.are | 202 |
| abstract_inverted_index.can | 150 |
| abstract_inverted_index.for | 23, 37, 67, 113, 190 |
| abstract_inverted_index.the | 3, 34, 96, 107, 116, 125, 128, 131, 134, 137, 153, 169, 191, 196 |
| abstract_inverted_index.(AI) | 11 |
| abstract_inverted_index.This | 0, 62 |
| abstract_inverted_index.also | 203 |
| abstract_inverted_index.data | 19, 48, 73, 119, 163 |
| abstract_inverted_index.grow | 29 |
| abstract_inverted_index.more | 176 |
| abstract_inverted_index.size | 31, 132 |
| abstract_inverted_index.that | 144 |
| abstract_inverted_index.this | 145 |
| abstract_inverted_index.time | 182 |
| abstract_inverted_index.with | 180 |
| abstract_inverted_index.based | 123 |
| abstract_inverted_index.cost. | 184 |
| abstract_inverted_index.model | 21, 51, 76, 121, 165 |
| abstract_inverted_index.novel | 111 |
| abstract_inverted_index.paper | 1, 108 |
| abstract_inverted_index.study | 63 |
| abstract_inverted_index.these | 59, 84, 188 |
| abstract_inverted_index.which | 46 |
| abstract_inverted_index.Hybrid | 44 |
| abstract_inverted_index.across | 88 |
| abstract_inverted_index.alone, | 167 |
| abstract_inverted_index.become | 41 |
| abstract_inverted_index.degree | 117 |
| abstract_inverted_index.future | 192 |
| abstract_inverted_index.hybrid | 13, 69, 147 |
| abstract_inverted_index.larger | 174 |
| abstract_inverted_index.memory | 101 |
| abstract_inverted_index.model, | 130 |
| abstract_inverted_index.models | 28, 91, 159, 179 |
| abstract_inverted_index.offers | 53 |
| abstract_inverted_index.usage, | 102 |
| abstract_inverted_index.address | 58 |
| abstract_inverted_index.between | 98 |
| abstract_inverted_index.complex | 177 |
| abstract_inverted_index.demands | 36 |
| abstract_inverted_index.improve | 152 |
| abstract_inverted_index.optimal | 72 |
| abstract_inverted_index.reduced | 181 |
| abstract_inverted_index.results | 142 |
| abstract_inverted_index.scaling | 8 |
| abstract_inverted_index.systems | 201 |
| abstract_inverted_index.through | 12 |
| abstract_inverted_index.various | 65 |
| abstract_inverted_index.adapting | 115 |
| abstract_inverted_index.adaptive | 146 |
| abstract_inverted_index.approach | 56 |
| abstract_inverted_index.combines | 47 |
| abstract_inverted_index.compared | 160 |
| abstract_inverted_index.dataset, | 135 |
| abstract_inverted_index.enabling | 168 |
| abstract_inverted_index.enhanced | 24 |
| abstract_inverted_index.explores | 64 |
| abstract_inverted_index.findings | 189 |
| abstract_inverted_index.focusing | 16 |
| abstract_inverted_index.hardware | 93 |
| abstract_inverted_index.presents | 109 |
| abstract_inverted_index.scalable | 199 |
| abstract_inverted_index.training | 38, 170 |
| abstract_inverted_index.available | 138 |
| abstract_inverted_index.balancing | 18 |
| abstract_inverted_index.different | 89 |
| abstract_inverted_index.efficient | 75 |
| abstract_inverted_index.evaluated | 87 |
| abstract_inverted_index.framework | 112, 149 |
| abstract_inverted_index.including | 71 |
| abstract_inverted_index.inference | 40 |
| abstract_inverted_index.overhead. | 105 |
| abstract_inverted_index.promising | 55 |
| abstract_inverted_index.sharding, | 77 |
| abstract_inverted_index.Artificial | 9 |
| abstract_inverted_index.challenges | 4 |
| abstract_inverted_index.deployment | 172 |
| abstract_inverted_index.discussed. | 204 |
| abstract_inverted_index.resources. | 140 |
| abstract_inverted_index.strategies | 66, 85 |
| abstract_inverted_index.trade-offs | 97 |
| abstract_inverted_index.challenges. | 61 |
| abstract_inverted_index.complexity, | 33 |
| abstract_inverted_index.demonstrate | 143 |
| abstract_inverted_index.development | 197 |
| abstract_inverted_index.dynamically | 114 |
| abstract_inverted_index.efficiency, | 100 |
| abstract_inverted_index.parallelism | 49, 122, 148, 166 |
| abstract_inverted_index.performance | 82, 156 |
| abstract_inverted_index.scalability | 60, 154 |
| abstract_inverted_index.techniques. | 80 |
| abstract_inverted_index.traditional | 162 |
| abstract_inverted_index.Experimental | 141 |
| abstract_inverted_index.Furthermore, | 106 |
| abstract_inverted_index.Intelligence | 10 |
| abstract_inverted_index.distribution | 22 |
| abstract_inverted_index.highlighting | 95 |
| abstract_inverted_index.implementing | 68 |
| abstract_inverted_index.implications | 186 |
| abstract_inverted_index.increasingly | 42 |
| abstract_inverted_index.investigates | 2 |
| abstract_inverted_index.parallelism, | 14, 45, 52, 70 |
| abstract_inverted_index.performance. | 25 |
| abstract_inverted_index.prohibitive. | 43 |
| abstract_inverted_index.specifically | 15 |
| abstract_inverted_index.communication | 104 |
| abstract_inverted_index.computational | 35, 99, 139 |
| abstract_inverted_index.opportunities | 6 |
| abstract_inverted_index.partitioning, | 74 |
| abstract_inverted_index.significantly | 151 |
| abstract_inverted_index.architectures, | 94 |
| abstract_inverted_index.characteristics | 126 |
| abstract_inverted_index.communication-reducing | 79 |
| cited_by_percentile_year | |
| countries_distinct_count | 0 |
| institutions_distinct_count | 2 |
| citation_normalized_percentile.value | 0.81632598 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |