Spark and Spark streaming with Python

This course is designed for developers, BI experts, analysts with python programming experience, working experience with datasets, including data analytics.

• Working experience in python programming
• Basic knowledge of machine learning
• Basic knowledge of SQL is helpful
• Prior knowledge of Hadoop is not required

Introduction to Hadoop and the Hadoop Ecosystem

Installation and implementation in developer’s environment

Spark Architecture:

·         Infrastructure layer

·         Persistence layer

·         Integration layer

·         Analytics layer

·         Engagement layer

 Working with Datasets:

·         RDD, DataFrames – Explanation

·         Read/Write Data from HIVE

·         Persisting data in CSV

·         Persisting data in JSON

·         Filtering the header

·         Missing values

·         Anomalous data

·         Split and parse a record

·         Lambda Functions

·         Filter records – matching a condition Apply a boolean function

·         Compute aggregates using the reduce operation

·         Computing Averages with reduceByKey

·         Transform each record to another record

·         merge Function

·         Merging Pair RDDs

·         Types of Joins

·         Exploring data using Spark SQL

·         Understanding the Spark SQL query optimizer

·         Loading and processing CSV files with Spark SQL

Introduction to Scala

Spark Mlib

Introduction to machine learning

Introduction to Mlib

·         linear regression problems

·         classification problems and decision tree algorithm

·         clustering problems and k-means algorithm

 Spark Streaming

·         Spark Streaming inner working

·         Kafka and Spark

·         Processing live data

·         Developing producers

·         Developing consumers

·         Developing a Spark Streaming consumer for Kafka

Monitoring And Performance Tuning