• Big Data Technologies Introduction

• Big Data & Hadoop – Core concepts and ecosystem overview.

  • NoSQL & Modern Data Tech – NoSQL basics and key Big Data technologies.

• Big Data and DataOps

• Basic Linux

  • Connecting Using Putty

  • Working With Directories, Files, File Contents

  • Control Operators

  • Shell Variables

  • I/O Redirection

  • Filters

  • File Permissions

• Git

  • Introduction to Source control

  • Git Branches, tags and release management

  • gitignore

  • pull requests (PR)

• Docker

  • What is Docker &containers

  • Docker main components and terminology

  • Docker architecture, hub, desktop, and commands

• CI/CD

  • CI/CD pipeline

  • Jenkins

  • Build machine

  • Artifact management

  • Tagging and releases

Python and PyData

• Basic Python

  • Working environments (Anaconda, Jupiter, PyCharm, etc.)

  • Data types (numbers, strings, Booleans, etc.)

  • Data collections (lists, dictionaries, etc.)

  • Flow control (if, for, while etc.)

  • Textual interface (input and formatting)

• Intermediate Python

  • Functions

    • User-defined functions

    • *args and **kwargs

    •  Built-in functions

    • Lambda expressions

  • Debugging and Error Handling

  • Text files

  • The standard library

    • import

    • datetime

• Pandas and Data Resources

  • The pandas library

    • Series and Index

    • DataFrame

    • GroupBy

    • Visualizations

  • General tools and DB

    • Intro to regular expressions (re)

    • JSON

    • DBs with SQLAlchemy package and Pandas

Generative AI For Data Engineering

• AI in the world of Data engineering

  • Role of Data Engineers in AI projects

  •  Data preparation – collection, cleaning, feature engineering

  • OCR for unstructured data (documents)

  • Multimodal AI – text, image

  • LangChain fundamentals and RAG process

  • Building enterprise chatbots with AI

  • Security & Governance in AI pipelines

  • Challenges – data quality, bias, cost

• AI Tools & Productivity for Data Engineers

  • GitHub Copilot / Copilot Workspace – coding & PR assist

  •  Cursor / JetBrains AI – completions & refactor

  •  Notion AI / Confluence AI – docs & runbooks

NoSQL

• MongoDB

  • MongoDB introduction and architecture, components, deployment, indexes

  • MongoDB CRUD and Administrative Commands

• Search Engines - ELK

  • Introduction to search engines, Elasticsearch and typical use-cases

  • Basic setup and configurations

  • Indexes (Lucene) and mapping configurations

  • Querying: APIs, queries and complex queries, aggregations

  • Introduction to Logstash

  • Introduction to Kibana

• Apache NiFi

  • Introduction

  • Core Components

  • Parameter Context

  • Architecture

  • Registry

  • Limitations

Spark

• Introduction to Spark

  • Spark architecture

  • Running modes

  • Parallelism concept

  • SparkContent & SQLContent & SparkSession

  • Common fileformats

• RDD – Low Level API

  • What is it RDD

  • Lazy execution

  • Transformations & Actions

  • Collecting data to driver

• Broadcast & Accumulators

• DataFrame Operations

  • Introduction to DataFrame

  • DataFrame optimizers

  • Working with Data Sources

  • DataFrame API

  • Spark functions

  • UDF

  • Analytic function

• Spark SQL API

• Spark Partitioning

• Spark Streaming

  • Spark Streaming introduction

  • Spark Streaming concepts

• Structured Streaming

  • Structured Streaming introduction

  • Structured Streaming with Kafka

  • Stateful Streaming Application

• Management & Monitoring

  • Spark UI Introduction

  • Read execution plan

  • Find bottlenecks

  • Access logs

  • Performance tuning

AWS

• Cloud Deployment Models

  • Service models:

  • Infrastructure as a service (Iaas)

  • Platform as a service (Paas)

  • Software as a service (Saas)

• Fundamentals Overview

  • Storage

  • Networking and security

  • Compute

  • Virtualization and Containers

  • Managed services

  • Databases – Relational and NoSQL

• Amazon and AWS History & Region

• AWS Management Console

• AWS Command Line Interface

• Amazon S3

  • What is a Big Data Data-Lake S3

  • Amazon S3 Integration

  • Data Lake on S3 Use-Cases

• Amazon RDS

  • What is Amazon RDS

  • Supported Database Engines

  • Features and Advantages

  • Creation Method

  • Database Engine Type & Instance Sizes

  • Amazon RDS Storage Types

  • Setting the Availability

  • RDS Parameter Groups

  • Amazon Aurora

• Amazon Redshift

  • What is Amazon Redshift

  • Amazon Redshift and S3 Integration

  • Architecture

  • Performance Optimization

  • Managing and Query Methods

  • Amazon Redshift SQL Editor

  • Settings and Configuration

  • Amazon Redshift Lab

• Amazon Athena

  • Features andAdvantages

  • Athena SQL Engine

  • Data model for Athena & Partitioning

  • Athena Table Definitions and Schema

  • Cost Considerations

  • Amazon Athena AWS Services Integrations

  • Amazon Athena Real World Projects

• Amazon Glue

  • What is AWS Glue

  • Features and Advantages

  • Main Concepts:

    • Supported data Sources

    • Supported data Targets

  • Main Components

    • Data Catalog

    • Crawlers and Classifiers

    • ETL Operations

    • Streaming ETL

    • Tables

    • Connections

    • Glue Use-Cases

• Lambda & API Gateway

  • Lambda Features and Advantages

  • Lambda Code Editor

  • Function Creation Options

  • Lambda Considerations

  • Exposure Rest API and handle events

  • Integration with sources and targets Lab

• Message Queues

  • Introduction to message queues

  • Simple Queue Service (SQS)

  • Kinesis Services

  • Kinesis Data Streams Use-Cases

  • Amazon Kinesis Pricing Method

Final Project