Browse Library

Real Time Spark Project for Beginners: Hadoop, Spark, Docker

Building Real Time Data Pipeline Using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexmonster on Docker

Instructed by Pari Margu

18 days left at this price!
$25 $30 17% off
30 days money back guarantee
It Includes
  • Get access to this course only
  • Lifetime Course Access
  • Play & Pause Videos
  • Get Certificate of Completion
  • High Quality Recorded Lectures
  • Learn Online from Mobile/PC/Tablet
  • Download Course for Offline Viewing
  • Inlcudes Real Projects
  • Free Instructor Support
  • Complete Development of Real Time Streaming Data Pipeline using Hadoop and Spark Cluster on Docker
  • Setting up Single Node Hadoop and Spark Cluster on Docker
  • Features of Spark Structured Streaming using Spark with Scala
  • Features of Spark Structured Streaming using Spark with Python(PySpark)
  • How to use PostgreSQL with Spark Structured Streaming
  • Basic understanding of Apache Kafka
  • How to build Data Visualisation using Django Web Framework and Flexmonster
  • Fundamentals of Docker and Containerization

In many data centers, different type of servers generate large amount of data(events, Event in this case is status of the server in the data center) in real-time. 
There is always a need to process these data in real-time and generate insights which will be used by the server/data center monitoring people and they have to track these server's status regularly and find the resolution in case of issues occurring, for better server stability. 
Since the data is huge and coming in real-time, we need to choose the right architecture with scalable storage and computation frameworks/technologies.
Hence we want to build the Real Time Data Pipeline Using Apache Kafka, Apache Spark, Hadoop, PostgreSQL, Django and Flexmonster on Docker to generate insights out of this data.
The Spark Project/Data Pipeline is built using Apache Spark with Scala and PySpark on Apache Hadoop Cluster which is on top of Docker.
Data Visualization is built using Django Web Framework and Flexmonster.

  • Windows 10 Operating System
  • Laptop or Desktop with 8 GB of RAM(minimum) and 100 GB of free disk space
  • Any Text Editor
  • Java 1.8
  • Basic understanding of Programming Language
  • Basic understanding of Apache Hadoop
  • Basic understanding of Apache Spark
View More...

Section 1 : Introduction

  • Lecture 1 :
  • Introduction to Apache Spark Preview
  • Lecture 2 :
  • Real Time Spark Project Overview | Building End to End Streaming Data Pipeline

Section 2 : Environment Setup

  • Lecture 1 :
  • Setting up Docker Environment
  • Lecture 2 :
  • Create Single Node Kafka Cluster on Docker
  • Lecture 3 :
  • Create Single Node Apache Hadoop and Spark Cluster on Docker
  • Lecture 4 :
  • Setting up IntelliJ IDEA Community Edition(IDE)
  • Lecture 5 :
  • Setting up PyCharm Community Edition(IDE)
  • Lecture 6 :
  • Setting up Django Web Framework

Section 3 : Development | Project Code Walk-through

  • Lecture 1 :
  • Building Streaming Data Pipeline using Scala | Spark Structured Streaming
  • Lecture 2 :
  • Building Streaming Data Pipeline using PySpark | Spark Structured Streaming
  • Lecture 3 :
  • Event Simulator using Python(Server Status Detail
  • Lecture 4 :
  • Setting up PostgreSQL Database(Events Database)
  • Lecture 5 :
  • Building Dashboard using Django Web Framework and Flexmonster | Visualization

Section 4 : Complete Project Demo

  • Lecture 1 :
  • Real Time Spark Project Demo
  • Lecture 2 :
  • Running Real Time Streaming Data Pipeline using Spark Cluster On Docker

Section 5 : Bonus Tutorial - Docker Tutorial for Beginners

  • Lecture 1 :
  • Introduction to Docker
  • Lecture 2 :
  • Install Docker on Ubuntu 18.04
  • Lecture 3 :
  • Docker Commands | Commonly Used
  • Lecture 4 :
  • Create First Docker Image and Container
  • Lecture 5 :
  • Create MySQL Docker Container
  • Lecture 6 :
  • Cassandra on Docker Container
  • Lecture 7 :
  • MongoDB on Docker Container
  • Lecture 8 :
  • Setting up Docker Compose
  • Lecture 9 :
  • How to create Docker Volume
  • How do i access the course after purchase?

    Once you purchase a course (Single course or Subscription), you will be able to access the courses instantly online by logging into your account. Use the user name & password that you created while signing up. Once logged in, you can go to the "My Courses" section to access your course.
  • Are these video based online self-learning courses?

    Yes. All of the courses comes with online video based lectures created by certified instructors. Instructors have crafted these courses with a blend of high quality interactive videos, lectures, quizzes & real world projects to give you an indepth knowledge about the topic.
  • Can i play & pause the course as per my convenience?

    Yes absolutely & thats one of the advantage of self-paced courses. You can anytime pause or resume the course & come back & forth from one lecture to another lecture, play the videos mulitple times & so on.
  • How do i contact the instructor for any doubts or questions?

    Most of these courses have general questions & answers already covered within the course lectures. However, if you need any further help from the instructor, you can use the inbuilt Chat with Instructor option to send a message to an instructor & they will reply you within 24 hours. You can ask as many questions as you want.
  • Do i need a pc to access the course or can i do it on mobile & tablet as well?

    Brilliant question? Isn't it? You can access the courses on any device like PC, Mobile, Tablet & even on a smart tv. For mobile & a tablet you can download the Learnfly android or an iOS app. If mobile app is not available in your country, you can access the course directly by visting our website, its fully mobile friendly.
  • Do i get any certification after completing the course?

    Yes. Once you succesfully complete any course on Learnfly marketplace, you get a certiifcate of course completion emailed to you within 24 hours with your name & the Learnfly badge. You can definately brag about it & share it on your social media or with friends as one of your achievement. Click here to view the sample certificate Click Here
  • For how long can i access my course after the purchase?

    If you buy a single course, that course is accessible to you for a lifetime. If you go for a premium subcription, you can access all the courses on Learnfly marketplace till your subscription is Active.
  • Whats the difference between Single Course Purchase & Go Premium option?

    With Single Course Purchase, you only get an access of one single course. Whereas, with premium monhtly or annual subscription, you can access all the existing or new courses on learnfly marketplace. You can decide what option suits you the best and accordingly you can make your purchase.
  • Is there any free trial?

    Currently, we don't have any free trial but it may be available in near future.
  • What is the refund policy?

    We would hate you to leave us. However, if you are not satisfied, you can ask for a full refund within 30 days & we will be happy to assist you further.

Pari Margu,

Data Engineer(Big Data/Hadoop, Apache Spark, Python) cum Freelance Consultant, YouTube Creator. Having 12+ years of experience in implementing solutions to the enterprise clients and having strong Framework skills to implement complex business solutions. Worked on, Web, Windows, Mobile and Hadoop/Big Data, Apache Spark applications. Having 6+ years of experience with Hadoop/Big Data, Apache Spark Framework. Worked on Hadoop distributions like Cloudera CDH, Apache Hadoop.
View More...

Robotic Process Automation RPA Lear...

By : Sayantan Tarafdar

Lecture 41


Docker Hands-on for beginners

By : Samarth Deyagond

Lecture 16


Kubernetes and Docker: The Containe...

By : Cerulean Canvas

Lecture 114


Ansible for the DevOps Beginners & ...

By : Shankar AR

Lecture 37


DevOps Methodology with DevOps Tool...

By : Abhishek Roshan .

Lecture 103

Sign up and start learning
By signing up, you agree to our Terms of Use and Privacy Policy
Reset Password Request
Enter your email and we will email you the reset link