DP-3028: Implement Generative AI engineering with Azure Databricks

DP-3028 Course: Implement Generative AI engineering with Azure Databricks

This course covers generative AI engineering on Azure Databricks, using Spark to explore, fine-tune, evaluate, and integrate advanced language models. It teaches how to implement techniques such as Retrieval-Augmented Generation (RAG) and multi-step reasoning, as well as how to fine-tune large language models for specific tasks and evaluate their performance. Students will also learn about responsible AI practices for implementing AI solutions and how to manage models in production using LLMOps (Large Language Model Operations) on Azure Databricks.

Level: Intermediate - Role: AI Engineer, Data Scientist - Product: Azure - Subject: Artificial Intelligence, Machine Learning

Target audience

This course is designed for data scientists, machine learning engineers, and other AI professionals who wish to create generative AI applications with Azure Databricks. It is intended for professionals familiar with fundamental AI concepts and the Azure Databricks platform.

Official DP-3028 course objectives

• Introduction to Large Language Models (LLMs): Understand the fundamentals of generative AI, language models, and their application in Natural Language Processing (NLP) tasks.
• Implement RAG (Retrieval-Augmented Generation): Learn to prepare data, perform vector searches, and apply re-ranking techniques to improve the accuracy of generated responses.
• Develop multi-step reasoning: Use frameworks like LangChain, LlamaIndex, Haystack, and DSPy to build complex reasoning flows.
• Fine-tune language models: Prepare data and perform fine-tuning on Azure OpenAI models for specific tasks.
• Evaluate language models: Compare traditional evaluations with metrics specific to LLMs, including the "LLM-as-a-judge" approach.
• Apply responsible AI principles: Identify risks, mitigate issues, and apply security tools to protect AI systems.
• Implement LLMOps: Transition from MLOps to LLMOps, manage deployments with MLflow, and use Unity Catalog for version control and model security.

Official Azure Databricks DP-3028 course content

Module 1: Introduction to language models in Azure Databricks

• Introduction
• Description of generative artificial intelligence
• Understanding Large Language Models (LLM)
• Identification of key components of LLM applications
• Use of LLMs for Natural Language Processing (NLP) tasks
• Exercise: Exploration of language models

Module 2: Implementing Retrieval-Augmented Generation (RAG) with Azure Databricks

• Introduction
• Exploration of the main concepts of a RAG workflow
• Data preparation for RAG
• Retrieving relevant data with vector search
• Re-ranking retrieved results
• Exercise: RAG configuration

Module 3: Implementing multi-step reasoning in Azure Databricks

• Introduction
• What are multi-step reasoning systems?
• Exploring LangChain
• Exploring LlamaIndex
• Exploring Haystack
• Exploring the DSPy framework
• Exercise: Implement multi-step reasoning with LangChain

Module 4: Fine-tuning language models with Azure Databricks

• Introduction
• What is fine-tuning?
• Data preparation for fine-tuning
• Fine-tuning an Azure OpenAI model
• Exercise: Fine-tuning an Azure OpenAI model

Module 5: Evaluating language models with Azure Databricks

• Introduction
• Comparison of LLM and traditional ML evaluations
• Evaluating VMs and artificial intelligence systems
• Evaluating LLMs with standard metrics
• Description of LLM-as-a-judge for evaluation
• Exercise: Evaluating an Azure OpenAI model

Module 6: Reviewing responsible AI principles for language models in Azure Databricks

• Introduction
• What is responsible artificial intelligence?
• Identifying risks
• Mitigating issues
• Using key security tools to protect artificial intelligence systems
• Exercise: Implementing responsible AI

Module 7: Implementing LLMOps in Azure Databricks

• Introduction
• Transitioning from traditional MLOps to LLMOps
• Understanding model deployments
• Description of MLflow deployment capabilities
• Using Unity Catalog to manage models
• Exercise: Implementing LLMOps

Prerequisites

Before starting this module, you should be familiar with the fundamental concepts of artificial intelligence and Azure Databricks.

Language

• Course: English / Spanish