Abstract

Real-time data streaming pipelines are immensely valuable in today's data-driven world since they enable continuous data processing and analytics. This research paper provides a comprehensive exploration of the architecture, development, and deployment of an advanced real-time data streaming pipeline. It utilizes Docker for containerization, Apache Kafka for distributed streaming, Apache Spark for dynamic data transformation, and Cassandra for efficient NoSQL storage. The study outlines the intricacies of integrating these technologies, examining the pipeline's components, functionalities, performance metrics, and potential applications. Through this case study, the paper showcases the efficacy of open-source tools in constructing highly scalable and resilient data streaming pipelines.