SCALABLE MICROSERVICES FOR REAL-TIME BIG DATA PROCESSING IN THE CLOUD

Authors

  • Sai Manish Podduturi Fannie Mae, USA Author

Keywords:

Microservices Architecture, Real-time Data Processing, Cloud Scalability, Stream Processing, Fault Tolerance

Abstract

The use of scalable microservices for real-time large data processing in cloud systems is examined in this article. It looks at how modern businesses can handle large data volumes with little delay by using microservices architecture. The article illustrates how microservices outperform conventional batch processing techniques in stream processing scenarios by analyzing a number of components, such as data input, message broker integration, and processing layers. In order to facilitate effective data processing at scale, it also looks into resource management techniques, auto-scaling systems, and cloud integration capabilities. The article also provides a thorough framework for enterprises looking to use microservices for their large data processing requirements by outlining best practices for microservices implementation, including important topics like failure handling, monitoring, and data flow optimization.

References

David Reinsel, John Gantz, and John Rydning, "The Digitization of the World From Edge to Core ," IDC White Paper, Nov. 2018. [Online]. Available: https://www.seagate.com/files/www-content/our-story/trends/files/idc-seagate-dataage-whitepaper.pdf

Mayukh Nair, "How Netflix works: the (hugely simplified) complex stuff that happens every time you hit Play," Medium, Oct 17, 2017. [Online]. Available: https://medium.com/refraction-tech-everything/how-netflix-works-the-hugely-simplified-complex-stuff-that-happens-every-time-you-hit-play-3a40c9be254b

Safa Ben Atitallah, Maha Driss, and Henda Ben Ghzela, "Microservices for Data Analytics in IoT Applications: Current Solutions, Open Challenges, and Future Research Directions," Science Direct, Oct 19, 2022. [Online]. Available: https://www.sciencedirect.com/science/article/pii/S1877050922013503

Misha Epikhin, "Benchmarking Apache Kafka: performance-per-price," Double Cloud Engineering Blog, June 24, 2024. [Online]. Available: https://double.cloud/blog/posts/2024/06/benchmarking-apache-kafka-performance-per-price/

Daniel Greenberg, "Optimizing Data Pipelines: Understanding Batch Processing vs. Stream Processing," Rivery Blog, July 26, 2024. [Online]. Available: https://rivery.io/blog/batch-vs-stream-processing-pros-and-cons-2/

S. Kumar, "Scalability and Fault Tolerance in Event-Driven Microservices with Kafka," Medium - Platform Engineers, Nov 20, 2024. [Online]. Available: https://medium.com/@platform.engineers/scalability-and-fault-tolerance-in-event-driven-microservices-with-kafka-193cd4404e10

Oyekunle Oyeniran, Adebunmi Adewusi, Adams Gbolahan Adeleke, and Lucy Akwawa, "Microservices architecture in cloud-native applications: Design patterns and scalability," ResearchGate, Sep. 2024. [Online]. Available: https://www.researchgate.net/publication/383831564_Microservices_architecture_in_cloud-native_applications_Design_patterns_and_scalability

Isam Mashhour al Jawarneh, Paolo Bellavista, Filippo Bosi, and Luca Foschini, "Container Orchestration Engines: A Thorough Functional and Performance Comparison," ResearchGate, May 2019. [Online]. Available: https://www.researchgate.net/publication/334487211_Container_Orchestration_Engines_A_Thorough_Functional_and_Performance_Comparison

Sudip Sengupta, "15 Best Practices for Building a Microservices Architecture," BMC Blogs, Dec 23, 2022. [Online]. Available: https://www.bmc.com/blogs/microservices-best-practices/

Ankush Madaan, "Observability In Modern Microservices Architecture," SquareOps Blog, June 4, 2024. [Online]. Available: https://squareops.com/blog/observability-in-modern-microservices-architecture/

Published

2024-12-13

How to Cite

Sai Manish Podduturi. (2024). SCALABLE MICROSERVICES FOR REAL-TIME BIG DATA PROCESSING IN THE CLOUD. INTERNATIONAL JOURNAL OF RESEARCH IN COMPUTER APPLICATIONS AND INFORMATION TECHNOLOGY (IJRCAIT), 7(2), 2250-2258. https://ijrcait.com/index.php/home/article/view/IJRCAIT_07_02_163