DP-3028: Implement Generative AI engineering with Azure Databricks

Course DP-3028: Implement Generative AI engineering with Azure Databricks

This course covers generative AI engineering in Azure Databricks, using Spark to explore, refine, evaluate, and integrate advanced language models. It teaches how to implement techniques such as Retrieval-Augmented Generation (RAG) and multi-stage reasoning, as well as how to fine-tune large language models for specific tasks and evaluate their performance. Students will also learn about responsible AI practices for implementing AI solutions and how to manage models in production using LLMOps (Large Language Model Operations) in Azure Databricks.

Level: Intermediate - Role: AI Engineer, Data Scientist - Product: Azure - Subject: Artificial Intelligence, Machine Learning

Course audience

This course is designed for data scientists, machine learning engineers, and other AI professionals who wish to build generative AI applications with Azure Databricks. It is aimed at professionals familiar with fundamental AI concepts and the Azure Databricks platform.

Objectives of the official DP-3028 course

• Introduction to Language Models (LLMs): Understand the fundamentals of generative AI, language models, and their application in Natural Language Processing (NLP) tasks.
• Implement RAG (Retrieval-Augmented Generation): Learn to prepare data, perform vector searches, and apply re-ranking techniques to improve the accuracy of generated responses.
• Develop multi-stage reasoning: Use frameworks like LangChain, LlamaIndex, Haystack, and DSPy to build complex reasoning flows.
• Fine-tuning Language Models: Prepare data and fine-tune Azure OpenAI models for specific tasks.
• Evaluate Language Models: Compare traditional evaluations with specific metrics for LLMs, including the "LLM-as-a-judge" approach.
• Apply responsible AI principles: Identify risks, mitigate problems, and apply security tools to protect AI systems.
• Implement LLMOps: Transition from MLOps to LLMOps, manage deployments with MLflow, and use Unity Catalog for model version control and security.

Content of the official Azure Databricks DP-3028 course

Module 1 Introduction to Language Models in Azure Databricks

• Introduction
• Overview of Generative Artificial Intelligence
• Understanding Large Language Models (LLMs)
• Identifying key components of LLM applications
• Using LLMs for Natural Language Processing (NLP) tasks
• Exercise: Exploring Language Models

Module 2 Implementing Retrieval-Augmented Generation (RAG) with Azure Databricks

• Introduction
• Exploring the main concepts of a RAG workflow
• Preparing data for RAG
• Searching for relevant data with vector search
• Re-ranking retrieved results
• Exercise: Setting up RAG

Module 3 Implementing Multi-stage Reasoning in Azure Databricks

• Introduction
• What are multi-stage reasoning systems?
• Exploring LangChain
• Exploring LlamaIndex
• Explore Haystack
• Explore the DSPy framework
• Exercise: Implementing multi-stage reasoning with LangChain

Module 4 Fine-tuning Language Models with Azure Databricks

• Introduction
• What is fine-tuning?
• Preparing data for fine-tuning
• Fine-tuning an Azure OpenAI model
• Exercise: Fine-tuning an Azure OpenAI model

Module 5 Evaluating Language Models with Azure Databricks

• Introduction
• Comparing LLM and traditional ML evaluations
• Evaluating virtual machines and AI systems
• Evaluating LLMs with standard metrics
• Overview of LLM-as-a-judge for evaluation
• Exercise: Evaluating an Azure OpenAI model

Module 6 Review of Responsible AI Principles for Language Models in Azure Databricks

• Introduction
• What is Responsible Artificial Intelligence?
• Identifying risks
• Mitigating problems
• Using key security tools to protect AI systems
• Exercise: Implementing Responsible AI

Module 7 Implementing LLMOps in Azure Databricks

• Introduction
• Transitioning from traditional MLOps to LLMOps
• Understanding model deployments
• Overview of MLflow deployment functionalities
• Using Unity Catalog to manage models
• Exercise: Implementing LLMOps

Prerequisites

Before starting this module, you should be familiar with the fundamental concepts of artificial intelligence and Azure Databricks.

Language

• Course: English / Spanish