Adaptive Strategies for Geo Distributed Data Processing Across Heterogeneous Internet Landscapes Article Swipe
Vasanth Kumar
,
P. Sundaravadivel
,
Augustian Isaac R
,
K Premnath
·
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.21203/rs.3.rs-6696635/v1
YOU?
·
· 2025
· Open Access
·
· DOI: https://doi.org/10.21203/rs.3.rs-6696635/v1
The complexity of geo-distributed data processing is difficult for traditional Map Reduce frameworks to handle because they were initially created for single-cluster environments. Despite efforts to address these issues, extensions like Hierarchical Map Reduce and Geo-Hadoop are still ineffective at managing resource heterogeneity and inter-cluster communication, especially when dealing with uneven bandwidths and processing capacities among geographically separated clusters. Moreover, there is a substantial overhead and complexity introduced when relying on a single global reducer for result aggregation. In order to address these issues, we present Extended Cross-Map Reduce (ECMR), a novel framework that explicitly takes network dynamics and resource heterogeneity into account while adaptively optimizing geo-distributed data processing. In order to avoid data transfer bottlenecks, ECMR employs numerous global reducers, overlaps communication with computation, and intelligently decides the minimal amount of data needed for computation of the final result. In order to dynamically choose the ideal number and location of global reducers across clusters, it extends the Gale-Shapley method and presents a bipartite graph model. Extensive experimental tests on an actual testbed show that ECMR performs noticeably better than current methods. With overall make span reductions of up to 81% and 85%, respectively, ECMR outperforms Hierarchical and Geo-Hadoop systems, demonstrating its efficacy in improving performance and efficiency in diverse, Internet-scale data processing contexts.
Related Topics
Metadata
- Type
- preprint
- Language
- en
- Landing Page
- https://doi.org/10.21203/rs.3.rs-6696635/v1
- https://www.researchsquare.com/article/rs-6696635/latest.pdf
- OA Status
- gold
- References
- 18
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4411326554
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4411326554Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.21203/rs.3.rs-6696635/v1Digital Object Identifier
- Title
-
Adaptive Strategies for Geo Distributed Data Processing Across Heterogeneous Internet LandscapesWork title
- Type
-
preprintOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2025Year of publication
- Publication date
-
2025-06-16Full publication date if available
- Authors
-
Vasanth Kumar, P. Sundaravadivel, Augustian Isaac R, K PremnathList of authors in order
- Landing page
-
https://doi.org/10.21203/rs.3.rs-6696635/v1Publisher landing page
- PDF URL
-
https://www.researchsquare.com/article/rs-6696635/latest.pdfDirect link to full text PDF
- Open access
-
YesWhether a free full text is available
- OA status
-
goldOpen access status per OpenAlex
- OA URL
-
https://www.researchsquare.com/article/rs-6696635/latest.pdfDirect OA link when available
- Concepts
-
The Internet, Computer science, Distributed computing, Data science, World Wide WebTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- References (count)
-
18Number of works referenced by this work
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4411326554 |
|---|---|
| doi | https://doi.org/10.21203/rs.3.rs-6696635/v1 |
| ids.doi | https://doi.org/10.21203/rs.3.rs-6696635/v1 |
| ids.openalex | https://openalex.org/W4411326554 |
| fwci | 0.0 |
| type | preprint |
| title | Adaptive Strategies for Geo Distributed Data Processing Across Heterogeneous Internet Landscapes |
| biblio.issue | |
| biblio.volume | |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10101 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.9995999932289124 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1710 |
| topics[0].subfield.display_name | Information Systems |
| topics[0].display_name | Cloud Computing and Resource Management |
| topics[1].id | https://openalex.org/T10715 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.9984999895095825 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1705 |
| topics[1].subfield.display_name | Computer Networks and Communications |
| topics[1].display_name | Distributed and Parallel Computing Systems |
| topics[2].id | https://openalex.org/T10742 |
| topics[2].field.id | https://openalex.org/fields/17 |
| topics[2].field.display_name | Computer Science |
| topics[2].score | 0.994700014591217 |
| topics[2].domain.id | https://openalex.org/domains/3 |
| topics[2].domain.display_name | Physical Sciences |
| topics[2].subfield.id | https://openalex.org/subfields/1705 |
| topics[2].subfield.display_name | Computer Networks and Communications |
| topics[2].display_name | Peer-to-Peer Network Technologies |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C110875604 |
| concepts[0].level | 2 |
| concepts[0].score | 0.5781199932098389 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q75 |
| concepts[0].display_name | The Internet |
| concepts[1].id | https://openalex.org/C41008148 |
| concepts[1].level | 0 |
| concepts[1].score | 0.5656378269195557 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[1].display_name | Computer science |
| concepts[2].id | https://openalex.org/C120314980 |
| concepts[2].level | 1 |
| concepts[2].score | 0.3765799403190613 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q180634 |
| concepts[2].display_name | Distributed computing |
| concepts[3].id | https://openalex.org/C2522767166 |
| concepts[3].level | 1 |
| concepts[3].score | 0.3571435213088989 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q2374463 |
| concepts[3].display_name | Data science |
| concepts[4].id | https://openalex.org/C136764020 |
| concepts[4].level | 1 |
| concepts[4].score | 0.2394745647907257 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q466 |
| concepts[4].display_name | World Wide Web |
| keywords[0].id | https://openalex.org/keywords/the-internet |
| keywords[0].score | 0.5781199932098389 |
| keywords[0].display_name | The Internet |
| keywords[1].id | https://openalex.org/keywords/computer-science |
| keywords[1].score | 0.5656378269195557 |
| keywords[1].display_name | Computer science |
| keywords[2].id | https://openalex.org/keywords/distributed-computing |
| keywords[2].score | 0.3765799403190613 |
| keywords[2].display_name | Distributed computing |
| keywords[3].id | https://openalex.org/keywords/data-science |
| keywords[3].score | 0.3571435213088989 |
| keywords[3].display_name | Data science |
| keywords[4].id | https://openalex.org/keywords/world-wide-web |
| keywords[4].score | 0.2394745647907257 |
| keywords[4].display_name | World Wide Web |
| language | en |
| locations[0].id | doi:10.21203/rs.3.rs-6696635/v1 |
| locations[0].is_oa | True |
| locations[0].source | |
| locations[0].license | cc-by |
| locations[0].pdf_url | https://www.researchsquare.com/article/rs-6696635/latest.pdf |
| locations[0].version | acceptedVersion |
| locations[0].raw_type | posted-content |
| locations[0].license_id | https://openalex.org/licenses/cc-by |
| locations[0].is_accepted | True |
| locations[0].is_published | False |
| locations[0].raw_source_name | |
| locations[0].landing_page_url | https://doi.org/10.21203/rs.3.rs-6696635/v1 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5018240226 |
| authorships[0].author.orcid | https://orcid.org/0000-0003-2960-1216 |
| authorships[0].author.display_name | Vasanth Kumar |
| authorships[0].countries | IN |
| authorships[0].affiliations[0].institution_ids | https://openalex.org/I85461943 |
| authorships[0].affiliations[0].raw_affiliation_string | Saveetha Engineering Colege |
| authorships[0].institutions[0].id | https://openalex.org/I85461943 |
| authorships[0].institutions[0].ror | https://ror.org/0034me914 |
| authorships[0].institutions[0].type | education |
| authorships[0].institutions[0].lineage | https://openalex.org/I85461943 |
| authorships[0].institutions[0].country_code | IN |
| authorships[0].institutions[0].display_name | Saveetha University |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Vasanth Kumar CH |
| authorships[0].is_corresponding | False |
| authorships[0].raw_affiliation_strings | Saveetha Engineering Colege |
| authorships[1].author.id | https://openalex.org/A5003876519 |
| authorships[1].author.orcid | |
| authorships[1].author.display_name | P. Sundaravadivel |
| authorships[1].countries | IN |
| authorships[1].affiliations[0].institution_ids | https://openalex.org/I85461943 |
| authorships[1].affiliations[0].raw_affiliation_string | Saveetha Engineering Colege |
| authorships[1].institutions[0].id | https://openalex.org/I85461943 |
| authorships[1].institutions[0].ror | https://ror.org/0034me914 |
| authorships[1].institutions[0].type | education |
| authorships[1].institutions[0].lineage | https://openalex.org/I85461943 |
| authorships[1].institutions[0].country_code | IN |
| authorships[1].institutions[0].display_name | Saveetha University |
| authorships[1].author_position | middle |
| authorships[1].raw_author_name | Sundaravadivel P |
| authorships[1].is_corresponding | False |
| authorships[1].raw_affiliation_strings | Saveetha Engineering Colege |
| authorships[2].author.id | https://openalex.org/A5022935773 |
| authorships[2].author.orcid | https://orcid.org/0000-0002-7640-4111 |
| authorships[2].author.display_name | Augustian Isaac R |
| authorships[2].countries | IN |
| authorships[2].affiliations[0].institution_ids | https://openalex.org/I85461943 |
| authorships[2].affiliations[0].raw_affiliation_string | Saveetha Engineering Colege |
| authorships[2].institutions[0].id | https://openalex.org/I85461943 |
| authorships[2].institutions[0].ror | https://ror.org/0034me914 |
| authorships[2].institutions[0].type | education |
| authorships[2].institutions[0].lineage | https://openalex.org/I85461943 |
| authorships[2].institutions[0].country_code | IN |
| authorships[2].institutions[0].display_name | Saveetha University |
| authorships[2].author_position | middle |
| authorships[2].raw_author_name | Augustian Isaac R |
| authorships[2].is_corresponding | False |
| authorships[2].raw_affiliation_strings | Saveetha Engineering Colege |
| authorships[3].author.id | https://openalex.org/A5050829274 |
| authorships[3].author.orcid | |
| authorships[3].author.display_name | K Premnath |
| authorships[3].countries | IN |
| authorships[3].affiliations[0].institution_ids | https://openalex.org/I85461943 |
| authorships[3].affiliations[0].raw_affiliation_string | Saveetha Engineering Colege |
| authorships[3].institutions[0].id | https://openalex.org/I85461943 |
| authorships[3].institutions[0].ror | https://ror.org/0034me914 |
| authorships[3].institutions[0].type | education |
| authorships[3].institutions[0].lineage | https://openalex.org/I85461943 |
| authorships[3].institutions[0].country_code | IN |
| authorships[3].institutions[0].display_name | Saveetha University |
| authorships[3].author_position | last |
| authorships[3].raw_author_name | Premnath K |
| authorships[3].is_corresponding | False |
| authorships[3].raw_affiliation_strings | Saveetha Engineering Colege |
| has_content.pdf | True |
| has_content.grobid_xml | True |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://www.researchsquare.com/article/rs-6696635/latest.pdf |
| open_access.oa_status | gold |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-10-10T00:00:00 |
| display_name | Adaptive Strategies for Geo Distributed Data Processing Across Heterogeneous Internet Landscapes |
| has_fulltext | True |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10101 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.9995999932289124 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1710 |
| primary_topic.subfield.display_name | Information Systems |
| primary_topic.display_name | Cloud Computing and Resource Management |
| related_works | https://openalex.org/W4391375266, https://openalex.org/W2899084033, https://openalex.org/W2748952813, https://openalex.org/W2390279801, https://openalex.org/W4391913857, https://openalex.org/W2358668433, https://openalex.org/W4396701345, https://openalex.org/W2376932109, https://openalex.org/W2001405890, https://openalex.org/W4396696052 |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.21203/rs.3.rs-6696635/v1 |
| best_oa_location.is_oa | True |
| best_oa_location.source | |
| best_oa_location.license | cc-by |
| best_oa_location.pdf_url | https://www.researchsquare.com/article/rs-6696635/latest.pdf |
| best_oa_location.version | acceptedVersion |
| best_oa_location.raw_type | posted-content |
| best_oa_location.license_id | https://openalex.org/licenses/cc-by |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | False |
| best_oa_location.raw_source_name | |
| best_oa_location.landing_page_url | https://doi.org/10.21203/rs.3.rs-6696635/v1 |
| primary_location.id | doi:10.21203/rs.3.rs-6696635/v1 |
| primary_location.is_oa | True |
| primary_location.source | |
| primary_location.license | cc-by |
| primary_location.pdf_url | https://www.researchsquare.com/article/rs-6696635/latest.pdf |
| primary_location.version | acceptedVersion |
| primary_location.raw_type | posted-content |
| primary_location.license_id | https://openalex.org/licenses/cc-by |
| primary_location.is_accepted | True |
| primary_location.is_published | False |
| primary_location.raw_source_name | |
| primary_location.landing_page_url | https://doi.org/10.21203/rs.3.rs-6696635/v1 |
| publication_date | 2025-06-16 |
| publication_year | 2025 |
| referenced_works | https://openalex.org/W2259941770, https://openalex.org/W3006691480, https://openalex.org/W2731964388, https://openalex.org/W4387764728, https://openalex.org/W2068115726, https://openalex.org/W1998051997, https://openalex.org/W2329835557, https://openalex.org/W2086392024, https://openalex.org/W2556111255, https://openalex.org/W3085985816, https://openalex.org/W1859126261, https://openalex.org/W4226314337, https://openalex.org/W2064349969, https://openalex.org/W2038043779, https://openalex.org/W2325704703, https://openalex.org/W4391515519, https://openalex.org/W4297571605, https://openalex.org/W3106036258 |
| referenced_works_count | 18 |
| abstract_inverted_index.a | 63, 72, 91, 163 |
| abstract_inverted_index.In | 79, 110, 141 |
| abstract_inverted_index.an | 171 |
| abstract_inverted_index.at | 40 |
| abstract_inverted_index.in | 204, 209 |
| abstract_inverted_index.is | 7, 62 |
| abstract_inverted_index.it | 156 |
| abstract_inverted_index.of | 3, 132, 137, 151, 188 |
| abstract_inverted_index.on | 71, 170 |
| abstract_inverted_index.to | 14, 26, 81, 112, 143, 190 |
| abstract_inverted_index.up | 189 |
| abstract_inverted_index.we | 85 |
| abstract_inverted_index.81% | 191 |
| abstract_inverted_index.Map | 11, 33 |
| abstract_inverted_index.The | 1 |
| abstract_inverted_index.and | 35, 44, 53, 66, 99, 126, 149, 161, 192, 198, 207 |
| abstract_inverted_index.are | 37 |
| abstract_inverted_index.for | 9, 21, 76, 135 |
| abstract_inverted_index.its | 202 |
| abstract_inverted_index.the | 129, 138, 146, 158 |
| abstract_inverted_index.85%, | 193 |
| abstract_inverted_index.ECMR | 117, 176, 195 |
| abstract_inverted_index.With | 183 |
| abstract_inverted_index.data | 5, 108, 114, 133, 212 |
| abstract_inverted_index.into | 102 |
| abstract_inverted_index.like | 31 |
| abstract_inverted_index.make | 185 |
| abstract_inverted_index.show | 174 |
| abstract_inverted_index.span | 186 |
| abstract_inverted_index.than | 180 |
| abstract_inverted_index.that | 94, 175 |
| abstract_inverted_index.they | 17 |
| abstract_inverted_index.were | 18 |
| abstract_inverted_index.when | 48, 69 |
| abstract_inverted_index.with | 50, 124 |
| abstract_inverted_index.among | 56 |
| abstract_inverted_index.avoid | 113 |
| abstract_inverted_index.final | 139 |
| abstract_inverted_index.graph | 165 |
| abstract_inverted_index.ideal | 147 |
| abstract_inverted_index.novel | 92 |
| abstract_inverted_index.order | 80, 111, 142 |
| abstract_inverted_index.still | 38 |
| abstract_inverted_index.takes | 96 |
| abstract_inverted_index.tests | 169 |
| abstract_inverted_index.there | 61 |
| abstract_inverted_index.these | 28, 83 |
| abstract_inverted_index.while | 104 |
| abstract_inverted_index.Reduce | 12, 34, 89 |
| abstract_inverted_index.across | 154 |
| abstract_inverted_index.actual | 172 |
| abstract_inverted_index.amount | 131 |
| abstract_inverted_index.better | 179 |
| abstract_inverted_index.choose | 145 |
| abstract_inverted_index.global | 74, 120, 152 |
| abstract_inverted_index.handle | 15 |
| abstract_inverted_index.method | 160 |
| abstract_inverted_index.model. | 166 |
| abstract_inverted_index.needed | 134 |
| abstract_inverted_index.number | 148 |
| abstract_inverted_index.result | 77 |
| abstract_inverted_index.single | 73 |
| abstract_inverted_index.uneven | 51 |
| abstract_inverted_index.(ECMR), | 90 |
| abstract_inverted_index.Despite | 24 |
| abstract_inverted_index.account | 103 |
| abstract_inverted_index.address | 27, 82 |
| abstract_inverted_index.because | 16 |
| abstract_inverted_index.created | 20 |
| abstract_inverted_index.current | 181 |
| abstract_inverted_index.dealing | 49 |
| abstract_inverted_index.decides | 128 |
| abstract_inverted_index.efforts | 25 |
| abstract_inverted_index.employs | 118 |
| abstract_inverted_index.extends | 157 |
| abstract_inverted_index.issues, | 29, 84 |
| abstract_inverted_index.minimal | 130 |
| abstract_inverted_index.network | 97 |
| abstract_inverted_index.overall | 184 |
| abstract_inverted_index.present | 86 |
| abstract_inverted_index.reducer | 75 |
| abstract_inverted_index.relying | 70 |
| abstract_inverted_index.result. | 140 |
| abstract_inverted_index.testbed | 173 |
| abstract_inverted_index.Extended | 87 |
| abstract_inverted_index.diverse, | 210 |
| abstract_inverted_index.dynamics | 98 |
| abstract_inverted_index.efficacy | 203 |
| abstract_inverted_index.location | 150 |
| abstract_inverted_index.managing | 41 |
| abstract_inverted_index.methods. | 182 |
| abstract_inverted_index.numerous | 119 |
| abstract_inverted_index.overhead | 65 |
| abstract_inverted_index.overlaps | 122 |
| abstract_inverted_index.performs | 177 |
| abstract_inverted_index.presents | 162 |
| abstract_inverted_index.reducers | 153 |
| abstract_inverted_index.resource | 42, 100 |
| abstract_inverted_index.systems, | 200 |
| abstract_inverted_index.transfer | 115 |
| abstract_inverted_index.Cross-Map | 88 |
| abstract_inverted_index.Extensive | 167 |
| abstract_inverted_index.Moreover, | 60 |
| abstract_inverted_index.bipartite | 164 |
| abstract_inverted_index.clusters, | 155 |
| abstract_inverted_index.clusters. | 59 |
| abstract_inverted_index.contexts. | 214 |
| abstract_inverted_index.difficult | 8 |
| abstract_inverted_index.framework | 93 |
| abstract_inverted_index.improving | 205 |
| abstract_inverted_index.initially | 19 |
| abstract_inverted_index.reducers, | 121 |
| abstract_inverted_index.separated | 58 |
| abstract_inverted_index.Geo-Hadoop | 36, 199 |
| abstract_inverted_index.adaptively | 105 |
| abstract_inverted_index.bandwidths | 52 |
| abstract_inverted_index.capacities | 55 |
| abstract_inverted_index.complexity | 2, 67 |
| abstract_inverted_index.efficiency | 208 |
| abstract_inverted_index.especially | 47 |
| abstract_inverted_index.explicitly | 95 |
| abstract_inverted_index.extensions | 30 |
| abstract_inverted_index.frameworks | 13 |
| abstract_inverted_index.introduced | 68 |
| abstract_inverted_index.noticeably | 178 |
| abstract_inverted_index.optimizing | 106 |
| abstract_inverted_index.processing | 6, 54, 213 |
| abstract_inverted_index.reductions | 187 |
| abstract_inverted_index.computation | 136 |
| abstract_inverted_index.dynamically | 144 |
| abstract_inverted_index.ineffective | 39 |
| abstract_inverted_index.outperforms | 196 |
| abstract_inverted_index.performance | 206 |
| abstract_inverted_index.processing. | 109 |
| abstract_inverted_index.substantial | 64 |
| abstract_inverted_index.traditional | 10 |
| abstract_inverted_index.Gale-Shapley | 159 |
| abstract_inverted_index.Hierarchical | 32, 197 |
| abstract_inverted_index.aggregation. | 78 |
| abstract_inverted_index.bottlenecks, | 116 |
| abstract_inverted_index.computation, | 125 |
| abstract_inverted_index.experimental | 168 |
| abstract_inverted_index.communication | 123 |
| abstract_inverted_index.demonstrating | 201 |
| abstract_inverted_index.environments. | 23 |
| abstract_inverted_index.heterogeneity | 43, 101 |
| abstract_inverted_index.intelligently | 127 |
| abstract_inverted_index.inter-cluster | 45 |
| abstract_inverted_index.respectively, | 194 |
| abstract_inverted_index.Internet-scale | 211 |
| abstract_inverted_index.communication, | 46 |
| abstract_inverted_index.geographically | 57 |
| abstract_inverted_index.single-cluster | 22 |
| abstract_inverted_index.geo-distributed | 4, 107 |
| abstract_inverted_index.<title>Abstract</title> | 0 |
| cited_by_percentile_year | |
| countries_distinct_count | 1 |
| institutions_distinct_count | 4 |
| citation_normalized_percentile.value | 0.29438482 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | True |