Real-time Data Pipeline Optimization using AI-driven Resource Allocation Article Swipe
This paper discusses an AI approach to improve how data is processed in real time by adjusting the way resources are given to distributed systems. Our framework adapts and changes the amount of CPU, available memory, and network use to hit performance goals due to machine learning and reinforcement learning. The system has shown improvement in latency, throughput, and using resources more efficiently during evaluation with many workloads. To tackle the challenges produced by intense and fast data, we make use of predictive analytics and scheduling that can adjust automatically. Findings point out that using AI for resource efficiency is a flexible and dependable way to run modern data engineering tasks both in clouds and at edges.
Related Topics
Concepts
Metadata
- Type
- article
- Language
- en
- Landing Page
- https://doi.org/10.15662/ijareeie.2024.1308012
- OA Status
- diamond
- Related Works
- 10
- OpenAlex ID
- https://openalex.org/W4412186799
All OpenAlex metadata
Raw OpenAlex JSON
- OpenAlex ID
-
https://openalex.org/W4412186799Canonical identifier for this work in OpenAlex
- DOI
-
https://doi.org/10.15662/ijareeie.2024.1308012Digital Object Identifier
- Title
-
Real-time Data Pipeline Optimization using AI-driven Resource AllocationWork title
- Type
-
articleOpenAlex work type
- Language
-
enPrimary language
- Publication year
-
2024Year of publication
- Publication date
-
2024-08-30Full publication date if available
- Authors
-
Manoj K. PalList of authors in order
- Landing page
-
https://doi.org/10.15662/ijareeie.2024.1308012Publisher landing page
- Open access
-
YesWhether a free full text is available
- OA status
-
diamondOpen access status per OpenAlex
- OA URL
-
https://doi.org/10.15662/ijareeie.2024.1308012Direct OA link when available
- Concepts
-
Pipeline (software), Resource allocation, Computer science, Resource (disambiguation), Operations research, Data mining, Real-time computing, Engineering, Computer network, Programming languageTop concepts (fields/topics) attached by OpenAlex
- Cited by
-
0Total citation count in OpenAlex
- Related works (count)
-
10Other works algorithmically related by OpenAlex
Full payload
| id | https://openalex.org/W4412186799 |
|---|---|
| doi | https://doi.org/10.15662/ijareeie.2024.1308012 |
| ids.doi | https://doi.org/10.15662/ijareeie.2024.1308012 |
| ids.openalex | https://openalex.org/W4412186799 |
| fwci | 0.0 |
| type | article |
| title | Real-time Data Pipeline Optimization using AI-driven Resource Allocation |
| biblio.issue | 08 |
| biblio.volume | 13 |
| biblio.last_page | |
| biblio.first_page | |
| topics[0].id | https://openalex.org/T10715 |
| topics[0].field.id | https://openalex.org/fields/17 |
| topics[0].field.display_name | Computer Science |
| topics[0].score | 0.630299985408783 |
| topics[0].domain.id | https://openalex.org/domains/3 |
| topics[0].domain.display_name | Physical Sciences |
| topics[0].subfield.id | https://openalex.org/subfields/1705 |
| topics[0].subfield.display_name | Computer Networks and Communications |
| topics[0].display_name | Distributed and Parallel Computing Systems |
| topics[1].id | https://openalex.org/T10101 |
| topics[1].field.id | https://openalex.org/fields/17 |
| topics[1].field.display_name | Computer Science |
| topics[1].score | 0.5945000052452087 |
| topics[1].domain.id | https://openalex.org/domains/3 |
| topics[1].domain.display_name | Physical Sciences |
| topics[1].subfield.id | https://openalex.org/subfields/1710 |
| topics[1].subfield.display_name | Information Systems |
| topics[1].display_name | Cloud Computing and Resource Management |
| is_xpac | False |
| apc_list | |
| apc_paid | |
| concepts[0].id | https://openalex.org/C43521106 |
| concepts[0].level | 2 |
| concepts[0].score | 0.7203757762908936 |
| concepts[0].wikidata | https://www.wikidata.org/wiki/Q2165493 |
| concepts[0].display_name | Pipeline (software) |
| concepts[1].id | https://openalex.org/C29202148 |
| concepts[1].level | 2 |
| concepts[1].score | 0.5873928666114807 |
| concepts[1].wikidata | https://www.wikidata.org/wiki/Q287260 |
| concepts[1].display_name | Resource allocation |
| concepts[2].id | https://openalex.org/C41008148 |
| concepts[2].level | 0 |
| concepts[2].score | 0.5767917633056641 |
| concepts[2].wikidata | https://www.wikidata.org/wiki/Q21198 |
| concepts[2].display_name | Computer science |
| concepts[3].id | https://openalex.org/C206345919 |
| concepts[3].level | 2 |
| concepts[3].score | 0.5341799259185791 |
| concepts[3].wikidata | https://www.wikidata.org/wiki/Q20380951 |
| concepts[3].display_name | Resource (disambiguation) |
| concepts[4].id | https://openalex.org/C42475967 |
| concepts[4].level | 1 |
| concepts[4].score | 0.4767906665802002 |
| concepts[4].wikidata | https://www.wikidata.org/wiki/Q194292 |
| concepts[4].display_name | Operations research |
| concepts[5].id | https://openalex.org/C124101348 |
| concepts[5].level | 1 |
| concepts[5].score | 0.42911022901535034 |
| concepts[5].wikidata | https://www.wikidata.org/wiki/Q172491 |
| concepts[5].display_name | Data mining |
| concepts[6].id | https://openalex.org/C79403827 |
| concepts[6].level | 1 |
| concepts[6].score | 0.3304870128631592 |
| concepts[6].wikidata | https://www.wikidata.org/wiki/Q3988 |
| concepts[6].display_name | Real-time computing |
| concepts[7].id | https://openalex.org/C127413603 |
| concepts[7].level | 0 |
| concepts[7].score | 0.23096448183059692 |
| concepts[7].wikidata | https://www.wikidata.org/wiki/Q11023 |
| concepts[7].display_name | Engineering |
| concepts[8].id | https://openalex.org/C31258907 |
| concepts[8].level | 1 |
| concepts[8].score | 0.08906108140945435 |
| concepts[8].wikidata | https://www.wikidata.org/wiki/Q1301371 |
| concepts[8].display_name | Computer network |
| concepts[9].id | https://openalex.org/C199360897 |
| concepts[9].level | 1 |
| concepts[9].score | 0.0 |
| concepts[9].wikidata | https://www.wikidata.org/wiki/Q9143 |
| concepts[9].display_name | Programming language |
| keywords[0].id | https://openalex.org/keywords/pipeline |
| keywords[0].score | 0.7203757762908936 |
| keywords[0].display_name | Pipeline (software) |
| keywords[1].id | https://openalex.org/keywords/resource-allocation |
| keywords[1].score | 0.5873928666114807 |
| keywords[1].display_name | Resource allocation |
| keywords[2].id | https://openalex.org/keywords/computer-science |
| keywords[2].score | 0.5767917633056641 |
| keywords[2].display_name | Computer science |
| keywords[3].id | https://openalex.org/keywords/resource |
| keywords[3].score | 0.5341799259185791 |
| keywords[3].display_name | Resource (disambiguation) |
| keywords[4].id | https://openalex.org/keywords/operations-research |
| keywords[4].score | 0.4767906665802002 |
| keywords[4].display_name | Operations research |
| keywords[5].id | https://openalex.org/keywords/data-mining |
| keywords[5].score | 0.42911022901535034 |
| keywords[5].display_name | Data mining |
| keywords[6].id | https://openalex.org/keywords/real-time-computing |
| keywords[6].score | 0.3304870128631592 |
| keywords[6].display_name | Real-time computing |
| keywords[7].id | https://openalex.org/keywords/engineering |
| keywords[7].score | 0.23096448183059692 |
| keywords[7].display_name | Engineering |
| keywords[8].id | https://openalex.org/keywords/computer-network |
| keywords[8].score | 0.08906108140945435 |
| keywords[8].display_name | Computer network |
| language | en |
| locations[0].id | doi:10.15662/ijareeie.2024.1308012 |
| locations[0].is_oa | True |
| locations[0].source.id | https://openalex.org/S2764607494 |
| locations[0].source.issn | 2278-8875, 2320-3765 |
| locations[0].source.type | journal |
| locations[0].source.is_oa | True |
| locations[0].source.issn_l | 2278-8875 |
| locations[0].source.is_core | False |
| locations[0].source.is_in_doaj | False |
| locations[0].source.display_name | International Journal of Advanced Research in Electrical Electronics and Instrumentation Engineering |
| locations[0].source.host_organization | https://openalex.org/P4323238129 |
| locations[0].source.host_organization_name | Ess And Ess Research Publications |
| locations[0].source.host_organization_lineage | https://openalex.org/P4323238129 |
| locations[0].source.host_organization_lineage_names | Ess And Ess Research Publications |
| locations[0].license | |
| locations[0].pdf_url | |
| locations[0].version | publishedVersion |
| locations[0].raw_type | journal-article |
| locations[0].license_id | |
| locations[0].is_accepted | True |
| locations[0].is_published | True |
| locations[0].raw_source_name | International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering |
| locations[0].landing_page_url | https://doi.org/10.15662/ijareeie.2024.1308012 |
| indexed_in | crossref |
| authorships[0].author.id | https://openalex.org/A5065954633 |
| authorships[0].author.orcid | https://orcid.org/0000-0001-9152-1028 |
| authorships[0].author.display_name | Manoj K. Pal |
| authorships[0].author_position | first |
| authorships[0].raw_author_name | Manoj Pal |
| authorships[0].is_corresponding | True |
| has_content.pdf | False |
| has_content.grobid_xml | False |
| is_paratext | False |
| open_access.is_oa | True |
| open_access.oa_url | https://doi.org/10.15662/ijareeie.2024.1308012 |
| open_access.oa_status | diamond |
| open_access.any_repository_has_fulltext | False |
| created_date | 2025-07-11T00:00:00 |
| display_name | Real-time Data Pipeline Optimization using AI-driven Resource Allocation |
| has_fulltext | False |
| is_retracted | False |
| updated_date | 2025-11-06T03:46:38.306776 |
| primary_topic.id | https://openalex.org/T10715 |
| primary_topic.field.id | https://openalex.org/fields/17 |
| primary_topic.field.display_name | Computer Science |
| primary_topic.score | 0.630299985408783 |
| primary_topic.domain.id | https://openalex.org/domains/3 |
| primary_topic.domain.display_name | Physical Sciences |
| primary_topic.subfield.id | https://openalex.org/subfields/1705 |
| primary_topic.subfield.display_name | Computer Networks and Communications |
| primary_topic.display_name | Distributed and Parallel Computing Systems |
| related_works | https://openalex.org/W3037187668, https://openalex.org/W4234772502, https://openalex.org/W2380685755, https://openalex.org/W2252100032, https://openalex.org/W2963436428, https://openalex.org/W4400978025, https://openalex.org/W2918743509, https://openalex.org/W2734796617, https://openalex.org/W3083218341, https://openalex.org/W2033726889 |
| cited_by_count | 0 |
| locations_count | 1 |
| best_oa_location.id | doi:10.15662/ijareeie.2024.1308012 |
| best_oa_location.is_oa | True |
| best_oa_location.source.id | https://openalex.org/S2764607494 |
| best_oa_location.source.issn | 2278-8875, 2320-3765 |
| best_oa_location.source.type | journal |
| best_oa_location.source.is_oa | True |
| best_oa_location.source.issn_l | 2278-8875 |
| best_oa_location.source.is_core | False |
| best_oa_location.source.is_in_doaj | False |
| best_oa_location.source.display_name | International Journal of Advanced Research in Electrical Electronics and Instrumentation Engineering |
| best_oa_location.source.host_organization | https://openalex.org/P4323238129 |
| best_oa_location.source.host_organization_name | Ess And Ess Research Publications |
| best_oa_location.source.host_organization_lineage | https://openalex.org/P4323238129 |
| best_oa_location.source.host_organization_lineage_names | Ess And Ess Research Publications |
| best_oa_location.license | |
| best_oa_location.pdf_url | |
| best_oa_location.version | publishedVersion |
| best_oa_location.raw_type | journal-article |
| best_oa_location.license_id | |
| best_oa_location.is_accepted | True |
| best_oa_location.is_published | True |
| best_oa_location.raw_source_name | International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering |
| best_oa_location.landing_page_url | https://doi.org/10.15662/ijareeie.2024.1308012 |
| primary_location.id | doi:10.15662/ijareeie.2024.1308012 |
| primary_location.is_oa | True |
| primary_location.source.id | https://openalex.org/S2764607494 |
| primary_location.source.issn | 2278-8875, 2320-3765 |
| primary_location.source.type | journal |
| primary_location.source.is_oa | True |
| primary_location.source.issn_l | 2278-8875 |
| primary_location.source.is_core | False |
| primary_location.source.is_in_doaj | False |
| primary_location.source.display_name | International Journal of Advanced Research in Electrical Electronics and Instrumentation Engineering |
| primary_location.source.host_organization | https://openalex.org/P4323238129 |
| primary_location.source.host_organization_name | Ess And Ess Research Publications |
| primary_location.source.host_organization_lineage | https://openalex.org/P4323238129 |
| primary_location.source.host_organization_lineage_names | Ess And Ess Research Publications |
| primary_location.license | |
| primary_location.pdf_url | |
| primary_location.version | publishedVersion |
| primary_location.raw_type | journal-article |
| primary_location.license_id | |
| primary_location.is_accepted | True |
| primary_location.is_published | True |
| primary_location.raw_source_name | International Journal of Advanced Research in Electrical, Electronics and Instrumentation Engineering |
| primary_location.landing_page_url | https://doi.org/10.15662/ijareeie.2024.1308012 |
| publication_date | 2024-08-30 |
| publication_year | 2024 |
| referenced_works_count | 0 |
| abstract_inverted_index.a | 100 |
| abstract_inverted_index.AI | 4, 95 |
| abstract_inverted_index.To | 68 |
| abstract_inverted_index.an | 3 |
| abstract_inverted_index.at | 115 |
| abstract_inverted_index.by | 15, 73 |
| abstract_inverted_index.in | 12, 55, 112 |
| abstract_inverted_index.is | 10, 99 |
| abstract_inverted_index.of | 32, 81 |
| abstract_inverted_index.to | 6, 22, 39, 44, 105 |
| abstract_inverted_index.we | 78 |
| abstract_inverted_index.Our | 25 |
| abstract_inverted_index.The | 50 |
| abstract_inverted_index.and | 28, 36, 47, 58, 75, 84, 102, 114 |
| abstract_inverted_index.are | 20 |
| abstract_inverted_index.can | 87 |
| abstract_inverted_index.due | 43 |
| abstract_inverted_index.for | 96 |
| abstract_inverted_index.has | 52 |
| abstract_inverted_index.hit | 40 |
| abstract_inverted_index.how | 8 |
| abstract_inverted_index.out | 92 |
| abstract_inverted_index.run | 106 |
| abstract_inverted_index.the | 17, 30, 70 |
| abstract_inverted_index.use | 38, 80 |
| abstract_inverted_index.way | 18, 104 |
| abstract_inverted_index.CPU, | 33 |
| abstract_inverted_index.This | 0 |
| abstract_inverted_index.both | 111 |
| abstract_inverted_index.data | 9, 108 |
| abstract_inverted_index.fast | 76 |
| abstract_inverted_index.make | 79 |
| abstract_inverted_index.many | 66 |
| abstract_inverted_index.more | 61 |
| abstract_inverted_index.real | 13 |
| abstract_inverted_index.that | 86, 93 |
| abstract_inverted_index.time | 14 |
| abstract_inverted_index.with | 65 |
| abstract_inverted_index.data, | 77 |
| abstract_inverted_index.given | 21 |
| abstract_inverted_index.goals | 42 |
| abstract_inverted_index.paper | 1 |
| abstract_inverted_index.point | 91 |
| abstract_inverted_index.shown | 53 |
| abstract_inverted_index.tasks | 110 |
| abstract_inverted_index.using | 59, 94 |
| abstract_inverted_index.adapts | 27 |
| abstract_inverted_index.adjust | 88 |
| abstract_inverted_index.amount | 31 |
| abstract_inverted_index.clouds | 113 |
| abstract_inverted_index.during | 63 |
| abstract_inverted_index.edges. | 116 |
| abstract_inverted_index.modern | 107 |
| abstract_inverted_index.system | 51 |
| abstract_inverted_index.tackle | 69 |
| abstract_inverted_index.changes | 29 |
| abstract_inverted_index.improve | 7 |
| abstract_inverted_index.intense | 74 |
| abstract_inverted_index.machine | 45 |
| abstract_inverted_index.memory, | 35 |
| abstract_inverted_index.network | 37 |
| abstract_inverted_index.Findings | 90 |
| abstract_inverted_index.approach | 5 |
| abstract_inverted_index.flexible | 101 |
| abstract_inverted_index.latency, | 56 |
| abstract_inverted_index.learning | 46 |
| abstract_inverted_index.produced | 72 |
| abstract_inverted_index.resource | 97 |
| abstract_inverted_index.systems. | 24 |
| abstract_inverted_index.adjusting | 16 |
| abstract_inverted_index.analytics | 83 |
| abstract_inverted_index.available | 34 |
| abstract_inverted_index.discusses | 2 |
| abstract_inverted_index.framework | 26 |
| abstract_inverted_index.learning. | 49 |
| abstract_inverted_index.processed | 11 |
| abstract_inverted_index.resources | 19, 60 |
| abstract_inverted_index.challenges | 71 |
| abstract_inverted_index.dependable | 103 |
| abstract_inverted_index.efficiency | 98 |
| abstract_inverted_index.evaluation | 64 |
| abstract_inverted_index.predictive | 82 |
| abstract_inverted_index.scheduling | 85 |
| abstract_inverted_index.workloads. | 67 |
| abstract_inverted_index.distributed | 23 |
| abstract_inverted_index.efficiently | 62 |
| abstract_inverted_index.engineering | 109 |
| abstract_inverted_index.improvement | 54 |
| abstract_inverted_index.performance | 41 |
| abstract_inverted_index.throughput, | 57 |
| abstract_inverted_index.reinforcement | 48 |
| abstract_inverted_index.automatically. | 89 |
| cited_by_percentile_year | |
| corresponding_author_ids | https://openalex.org/A5065954633 |
| countries_distinct_count | 0 |
| institutions_distinct_count | 1 |
| citation_normalized_percentile.value | 0.42651185 |
| citation_normalized_percentile.is_in_top_1_percent | False |
| citation_normalized_percentile.is_in_top_10_percent | False |