Pyspark Training Course

• Explaining Functions and Various Forms of Function Arguments
• Learning Variable Scope, Function Parameters, and Lambda Functions
• Sorting Using Python
• Exception Handling
• Package Installation
• Regular Expressions

• Using Class, Objects, and Attributes
• Developing Applications Based on OOP
• Learning About Classes, Objects and How They Function Together
• Explaining OOPs Concepts Including Inheritance, Encapsulation, and Polymorphism, Among Others

• Debugging Python Scripts Using pdb and IDE
• Classifying Errors and Developing Test Units
• Implementing Databases Using SQLite
• Performing CRUD Operations

• Introduction to PySpark
• Who uses PySpark?
• Why Python for Spark?
• Values, Types, Variables
• Operands and Expressions
• Conditional Statements
• Loops
• Numbers
• Python files I/O Functions
• Strings and associated operations
• Sets and associated operations
• Lists and associated operations
• Tuples and associated operations
• Dictionaries and associated operations

Hands-On:

• Demonstrating Loops and Conditional Statements
• Tuple – related operations, properties, list, etc.
• List – operations, related properties
• Set – properties, associated operations
• Dictionary – operations, related properties

• Functions
• Lambda Functions
• Global Variables, its Scope, and Returning Values
• Standard Libraries
• Object-Oriented Concepts
• Modules Used in Python
• The Import Statements
• Module Search Path
• Package Installation Ways

Hands-On:

• Lambda – Features, Options, Syntax, Compared with the Functions
• Functions – Syntax, Return Values, Arguments, and Keyword Arguments
• Errors and Exceptions – Issue Types, Remediation
• Packages and Modules – Import Options, Modules, sys Path

• Spark Components & its Architecture
• Spark Deployment Modes
• Spark Web UI
• Introduction to PySpark Shell
• Submitting PySpark Job
• Writing your first PySpark Job Using Jupyter Notebook
• What is Spark RDDs?
• Stopgaps in existing computing methodologies
• How RDD solve the problem?
• What are the ways to create RDD in PySpark?
• RDD persistence and caching
• General operations: Transformation, Actions, and Functions
• Concept of Key-Value pair in RDDs
• Other pair, two pair RDDs
• RDD Lineage
• RDD Persistence
• WordCount Program Using RDD Concepts
• RDD Partitioning & How it Helps Achieve Parallelization
• Passing Functions to Spark

Hands-On:

• Building and Running Spark Application
• Spark Application Web UI
• Loading data in RDDs
• Saving data through RDDs
• RDD Transformations
• RDD Actions and Functions
• RDD Partitions
• WordCount program using RDD’s in Python

• Need for Spark SQL
• What is Spark SQL
• Spark SQL Architecture
• SQL Context in Spark SQL
• User-Defined Functions
• Data Frames
• Interoperating with RDDs
• Loading Data through Different Sources
• Performance Tuning
• Spark-Hive Integration

Hands-On:

• Spark SQL – Creating data frames
• Loading and transforming data through different sources
• Spark-Hive Integration

• Why Kafka
• What is Kafka?
• Kafka Workflow
• Kafka Architecture
• Kafka Cluster Configuring
• Kafka Monitoring tools
• Basic operations
• What is Apache Flume?
• Integrating Apache Flume and Apache Kafka

Hands-On:

• Single Broker Kafka Cluster
• Multi-Broker Kafka Cluster
• Topic Operations
• Integrating Apache Flume and Apache Kafka

• Introduction to Spark Streaming
• Features of Spark Streaming
• Spark Streaming Workflow
• StreamingContext Initializing
• Discretized Streams (DStreams)
• Input DStreams, Receivers
• Transformations on DStreams
• DStreams Output Operations
• Describe Windowed Operators and Why it is Useful
• Stateful Operators
• Vital Windowed Operators
• Twitter Sentiment Analysis
• Streaming using Netcat server
• WordCount program using Kafka-Spark Streaming

Hands-On:

• Twitter Sentiment Analysis
• Streaming using Netcat server
• WordCount program using Kafka-Spark Streaming
• Spark-flume Integration

• Introduction to Machine Learning- What, Why and Where?
• Use Case
• Types of Machine Learning Techniques
• Why use Machine Learning for Spark?
• Applications of Machine Learning (general)
• Applications of Machine Learning with Spark
• Introduction to MLlib
• Features of MLlib and MLlib Tools
• Various ML algorithms supported by MLlib
• Supervised Learning Algorithms
• Unsupervised Learning Algorithms
• ML workflow utilities

Hands-On:

• K- Means Clustering
• Linear Regression
• Logistic Regression
• Decision Tree
• Random Forest