Course Description

In this course, the student will learn about data engineering as it relates to working with batch and real-time analytics solutions using Azure data platform technologies. Students will begin by learning the basic processing and storage technologies used to create an analytical solution. They will also learn how to interactively explore data stored in files in a data lake. They will learn about the various ingestion techniques that can be used to load data using the Apache Spark functionality included in Azure Synapse Analytics or Azure Databricks, or how to ingest using Azure Data Factory or Azure Synapse pipelines. Students will also learn about the different ways they can transform data using the same technologies used to ingest data. They will understand the importance of implementing security to ensure that data (at rest or in transit) is protected. After that, they will be explained how to create a real-time analytics system to create real-time analytics solutions.

Public Profile

The primary audience for this course is data professionals, data architects, and business intelligence professionals who want to learn about data engineering and building analytics solutions using data platform technologies found in Microsoft Azure. The secondary audience for this course is data analysts and data scientists working with analytics solutions based on Microsoft Azure.

Items in this collection

• Introduction to Azure Synapse Analytics (7 Units)
• Explore Azure Databricks (7 Units)
• Introduction to Azure Data Lake Storage (7 Units)
• Introduction to Azure Stream Analytics (7 Units)
• Using an Azure Synapse serverless SQL pool to query files in a data lake (7 Units)
• Using Azure Synapse Serverless SQL Pools to Transform Data in a Data Lake (7 Units)
• Create a lake database in Azure Synapse Analytics (8 Units)
• Data Protection and User Management in Azure Synapse Serverless SQL Pools (6 Units)
• Using Apache Spark on Azure Databricks (9 Units)
• Using Delta Lake on Azure Databricks (8 Units)
• Data analysis with Apache Spark in Azure Synapse Analytics (8 Units)
• Integration of SQL and Apache Spark groups in Azure Synapse Analytics (11 Units)
• Using best practices for loading data into Azure Synapse Analytics (11 Units)
• Petabyte-scale ingestion with Azure Data Factory or an Azure Synapse pipeline (9 Units)
• Integrate data with Azure Data Factory or Azure Synapse pipeline (13 Units)
• Perform no-code transformations at scale with Azure Data Factory or an Azure Synapse pipeline (10 Units)
• Orchestrate data movement and transformation in Azure Data Factory or Azure Synapse pipelines (9 Units)
• Planning hybrid analytical and transactional processing using Azure Synapse Analytics (5 Units)
• Implementation of Azure Synapse Link with Azure Cosmos DB (9 Units)
• Create a data warehouse in Azure Synapse Analytics (10 Units)
• Configuring and managing secrets in Azure Key Vault (6 Units)
• Implementation of compliance controls for confidential data (11 Units)
• Enabling reliable messaging for big data applications with Azure Event Hubs (8 Units)

Course outline

Module 1: Exploring compute and storage options for data engineering workloads

This module provides an overview of the Azure compute and storage technology options available to data engineers building analytical workloads. This module teaches you how to structure your data lake and optimize files for scanning, streaming, and batch workloads. The student will learn to organize the data lake into data refinement levels as they transform files through batch and stream processing. They will then learn how to create indexes on their data sets, such as CSV, JSON, and Parquet files, and use them for potential query and workload acceleration.

Lessons

• Get started with Azure Synapse Analytics

• Description of Azure Databricks

• Get started with Azure Data Lake Storage

• Delta Lake Architecture Description

• Work with data streams using Azure Stream Analytics

Lab: Exploring compute and storage options for data engineering workloads

• Combine batch and stream processing in a single pipeline

• Organize the data lake into file transformation tiers

• Index data lake storage for query and workload acceleration

After completing this module, students will be able to do the following:

• Describe Azure Synapse Analytics

• Description of Azure Databricks

• Describe Azure Data Lake Storage

• Describe the architecture of Delta Lake

• Describe Azure Stream Analytics

Module 2: Run interactive queries with Azure Synapse Analytics Serverless SQL Pools

In this module, students will learn how to work with files stored in the data lake and external file sources using T-SQL statements executed by a serverless SQL pool in Azure Synapse Analytics. They will query Parquet files stored in a data lake, as well as CSV files stored in an external data store. Next, they will create Azure Active Directory security groups and enforce access to data lake files through role-based access control (RBAC) and access control lists (ACL).

Lessons

• Exploring the capabilities of Azure Synapse Serverless SQL Pools

• Query data in the lake using Azure Synapse Serverless SQL Pools

• Create metadata objects in Azure Synapse serverless SQL pools

• Data protection and user management in Azure Synapse serverless SQL pools

Lab: Running Interactive Queries with Serverless SQL Pools

• Query Parquet Data with Serverless SQL Pools

• Create external tables for Parquet and CSV files

• Create views with serverless SQL pools

• Secure data access in a data lake when using serverless SQL pools

• Configure data lake security through role-based access control (RBAC) and access control lists (ACL)

After completing this module, students will be able to do the following:

• Describe the capabilities of Azure Synapse Serverless SQL Pools

• Query data in the lake using Azure Synapse Serverless SQL Pools

• Create metadata objects in Azure Synapse serverless SQL pools

• Data protection and user management in Azure Synapse serverless SQL pools

Module 3: Explore and transform data in Azure Databricks

This module teaches you how to use various Apache Spark DataFrame methods to explore and transform data in Azure Databricks. Students will learn to use standard DataFrame methods to explore and transform data. They will also learn to perform more advanced tasks, such as removing duplicate data, manipulating date and time values, renaming columns, and appending data.

Lessons

• Description of Azure Databricks

• Reading and writing data to Azure Databricks

• Work with DataFrame elements in Azure Databricks

• Work with advanced DataFrame methods in Azure Databricks

Lab: Perform data explorations and transformations in Azure Databricks

• Use DataFrames in Azure Databricks to explore and filter data

• Cache DataFrames for faster queries later

• Deduplication

• Manipulate date and time values

• Remove columns from DataFrame and rename them

• Add data stored in a DataFrame

After completing this module, students will be able to do the following:

• Description of Azure Databricks

• Reading and writing data to Azure Databricks

• Work with DataFrame elements in Azure Databricks

• Work with advanced DataFrame methods in Azure Databricks

Module 4: Exploring, transforming, and loading data into data warehouses with Apache Spark

This module teaches you how to explore data stored in a data lake, transform the data, and load it into a relational data warehouse. Students will explore Parquet and JSON files and use techniques to query and transform JSON files with hierarchical structures. They will then use Apache Spark to load data into the data warehouse and join data from Parquet in the data lake with data from the dedicated SQL pool.

Lessons

• Defining big data engineering with Apache Spark in Azure Synapse Analytics

• Ingest data with Apache Spark notebooks in Azure Synapse Analytics

• Transform data with Azure Synapse Analytics Apache Spark Pool DataFrame objects

• Integrate SQL pools and Apache Spark in Azure Synapse Analytics

Lab: Exploring, transforming, and loading data into data warehouses with Apache Spark

• Perform data explorations in Synapse Studio

• Ingest data with Spark notebooks in Azure Synapse Analytics

• Transform data with Azure Synapse Analytics Spark Pool DataFrame

• Integrate SQL and Spark pools into Azure Synapse Analytics

After completing this module, students will be able to do the following:

• Describe big data engineering with Apache Spark in Azure Synapse Analytics

• Ingest data with Apache Spark notebooks in Azure Synapse Analytics

• Transform data with Azure Synapse Analytics Apache Spark Pool DataFrame objects

• Integrate SQL pools and Apache Spark in Azure Synapse Analytics

Module 5: Ingesting and loading data into data stores

This module teaches students how to ingest data into the data warehouse using T-SQL scripts and Synapse Analytics integration pipelines. Students will learn how to load data into dedicated Synapse SQL pools with PolyBase and COPY using T-SQL. They will also learn how to use workload management along with a copy activity in an Azure Synapse pipeline for petabyte-scale data ingestion.

Lessons

• Using best practices for loading data into Azure Synapse Analytics

• Petabyte-scale ingestion with Azure Data Factory

Lab: Ingesting and loading data into data warehouses

• Perform petabyte-scale ingestions with Azure Synapse pipelines

• Import data with PolyBase and COPY using T-SQL

• Using best practices for loading data into Azure Synapse Analytics

After completing this module, students will be able to do the following:

• Using best practices for loading data into Azure Synapse Analytics

• Petabyte-scale ingestion with Azure Data Factory

Module 6: Transform data with Azure Data Factory or Azure Synapse pipelines

This module teaches students how to create data integration pipelines to ingest from multiple data sources, transform data using mapping data flows, and perform data moves to one or more data sinks.

Lessons

• Data integration with Azure Data Factory or Azure Synapse pipeline

• Perform code-free transformations at scale with Azure Data Factory or Azure Synapse pipelines

Lab: Transform data with Azure Data Factory or Azure Synapse pipelines

• Run transformations without code and at scale with Azure Synapse pipelines

• Create a data pipeline to import poorly formatted CSV files

• Create mapping data flows

After completing this module, students will be able to do the following:

• Perform data integrations with Azure Data Factory

• Perform codeless transformations at scale with Azure Data Factory

Module 7: Organize data moves and transformations in Azure Synapse pipelines

In this module, we will learn how to create linked services and organize data movement and transformation using notebooks in Azure Synapse pipelines.

Lessons

• Orchestrate data movements and transformations in Azure Data Factory

Lab: Orchestrate data moves and transformations in Azure Synapse pipelines

• Integrate Notebook data with Azure Data Factory or Azure Synapse pipelines

After completing this module, students will be able to do the following:

• Orchestrate data moves and transformations in Azure Synapse pipelines

Module 8: Comprehensive security with Azure Synapse Analytics

In this module, students will learn how to secure a Synapse Analytics workspace and its supporting infrastructure. They will analyze SQL Active Directory Manager, manage IP firewall rules, manage secrets with Azure Key Vault, and access those secrets through a Key Vault linked service and pipeline activities. They will also learn how to implement column-level and row-level security and dynamic data masking when using dedicated SQL pools.

Lessons

• Create a data warehouse in Azure Synapse Analytics

• Set up and manage secrets in Azure Key Vault

• Implementation of compliance controls for sensitive data

Lab: Comprehensive security with Azure Synapse Analytics

• Protect infrastructure behind Azure Synapse Analytics

• Secure your Azure Synapse Analytics workspace and managed services

• Protect your Azure Synapse Analytics workspace data

After completing this module, students will be able to do the following:

• Create a data warehouse in Azure Synapse Analytics

• Set up and manage secrets in Azure Key Vault

• Implementation of compliance controls for sensitive data

Module 9: Supporting hybrid transactional analytics processing with Azure Synapse Link

In this module, students will learn how Azure Synapse Link enables seamless connectivity between an Azure Cosmos DB account and a Synapse workspace. Students will see how to enable and configure Synapse Link, and then how to query the Azure Cosmos DB analytical store using Apache Spark and Serverless SQL.

Lessons

• Design hybrid transactional and analytical processing using Azure Synapse Analytics

• Configuring Azure Synapse Link with Azure Cosmos DB

• Azure Cosmos DB query with Apache Spark clusters

• Azure Cosmos DB Query with Serverless SQL Pools

Lab: Supporting hybrid transactional analytics processing with Azure Synapse Link

• Configuring Azure Synapse Link with Azure Cosmos DB

• Query Azure Cosmos DB with Apache Spark for Synapse Analytics

• Query Azure Cosmos DB with Serverless SQL Pools for Azure Synapse Analytics

After completing this module, students will be able to do the following:

• Design hybrid transactional and analytical processing using Azure Synapse Analytics

• Configuring Azure Synapse Link with Azure Cosmos DB

• Azure Cosmos DB query with Apache Spark for Azure Synapse Analytics

• Query Azure Cosmos DB with Serverless SQL for Azure Synapse Analytics

Module 10: Real-time stream processing with Stream Analytics

In this module, students will learn how to process stream data with Azure Stream Analytics. They will ingest vehicle telemetry data into Event Hubs and then process it in real time using various window-based functions in Azure Stream Analytics. They will send the data to Azure Synapse Analytics. Finally, students will learn how to scale Stream Analytics work to increase performance.

Lessons

• Enable reliable messaging for big data applications with Azure Event Hubs

• Work with data streams using Azure Stream Analytics

• Ingest data streams with Azure Stream Analytics

Lab: Real-time stream processing with Stream Analytics

• Use Stream Analytics to process data in real time from Event Hubs

• Use Stream Analytics window-based functions to create aggregates and send them to Synapse Analytics

• Scale Azure Stream Analytics jobs to increase performance through partitioning

• Re-partition stream input to optimize parallelization

After completing this module, students will be able to do the following:

• Enable reliable messaging for big data applications with Azure Event Hubs

• Work with data streams using Azure Stream Analytics

• Ingest data streams with Azure Stream Analytics

Module 11: Create a stream processing solution with Event Hubs and Azure Databricks

In this module, students will learn how to ingest and process stream data at scale with Event Hubs and Spark structured streaming in Azure Databricks. Students will learn the uses and key features of structured streaming. They will implement sliding windows to add data fragments and apply watermarks to remove obsolete data. Finally, students will connect to Event Hubs to read and write sequences.

Lessons

• Process streaming data with Azure Databricks Structured Streaming

Lab: Create a stream processing solution with Event Hubs and Azure Databricks

• Analyze the uses and key features of structured streaming.

• Stream data from a file and write it to a distributed file system

• Use sliding windows to add chunks of data instead of all data

• Apply watermarks to remove obsolete data

• Connect to Event Hubs read and write streams

After completing this module, students will be able to do the following:

• Process streaming data with Azure Databricks Structured Streaming

Previous requirements

Eligible students begin this course with knowledge of cloud computing and data fundamentals, and professional experience with data solutions.

Specifically carrying out:

• AZ-900: Azure Fundamentals

• DP-900: Data Fundamentals in Microsoft Azure

Language

• English course

• Labs: English