Virtualization, Docker, and Kubernetes for Data Engineering

Virtualization, Docker, and Kubernetes for Data Engineering

This course is part of Applied Python Data Engineering Specialization

Instructors: Noah Gift

6,433 already enrolled

Included with

Learn more

4 modules

Gain insight into a topic and learn the fundamentals.

49 reviews

Advanced level

Recommended experience

3 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

4 modules

Gain insight into a topic and learn the fundamentals.

49 reviews

Advanced level

Recommended experience

3 weeks to complete

at 10 hours a week

Flexible schedule

Learn at your own pace

What you'll learn

Master virtualization, containerization, and Docker, including Dockerfile creation and multi-container orchestration with Compose and Airflow.
Develop expertise in Kubernetes core concepts, cluster architecture, and deployment using cloud environments, GitHub Codespaces, and AI-driven tools.
Navigate data scenarios mastering containerization, deploying apps, and addressing production issues with cloud orchestration and SRE practices.

Skills you'll gain

Tools you'll learn

Details to know

Shareable certificate

Add to your LinkedIn profile

Assessments

38 assignments

Taught in English

See how employees at top companies are mastering in-demand skills

Learn more about Coursera for Business

logos of Petrobras, TATA, Danone, Capgemini, P&G and L'Oreal

Build your subject-matter expertise

This course is part of the Applied Python Data Engineering Specialization

When you enroll in this course, you'll also be enrolled in this Specialization.

Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate

There are 4 modules in this course

Throughout this course, you'll explore virtualization, containerization, and Kubernetes, mastering the very tools that power data engineering in the industry. Each week presents a new set of tools and platforms that are indispensable in data engineering. From mastering Docker and Kubernetes to exploring advanced topics such as AI-driven coding with GitHub Copilot, efficient container image management with Azure and Amazon Elastic Container Registries, and Site Reliability Engineering (SRE) practices, you'll go beyond the basics and acquire the expertise needed to thrive in the dynamic and data-driven landscape of advanced data engineering. Whether you're a current student looking to expand your skills or a working professional aiming to take your expertise to the next level, this course is tailored to equip you with the advanced knowledge and hands-on experience necessary for success.

In this module, you will learn about the fundamentals of virtualization, exploring its various aspects such as hardware utilization and scaling applications. You will start by understanding what virtualization is and delve into the concept of virtual machines. Through the introduction of Virtual Box and a hands-on demo, you will gain a practical understanding of how virtual machines work and their benefits. Additionally, you will explore container concepts, focusing on Docker as a key containerization tool. Through an introduction to Docker and its architecture, you will learn how to scale applications using containers, providing a comprehensive overview of virtualization and its practical applications. To apply your newfound knowledge, you will be assessed through a series of hands-on exercises involving the creation and management of virtual machines and containers, demonstrating your ability to effectively utilize virtualization technologies.

What's included

8 videos9 readings8 assignments1 discussion prompt

8 videos Total 26 minutes

Virtualization 3 minutes
Scaling Applications 2 minutes
Hardware Utilization 1 minute
Introduction to Virtual Machines 2 minutes
Virtual Box Demo 10 minutes
Container Concepts 2 minutes
Introduction to Docker 5 minutes
Docker Architecture 1 minute

9 readings Total 90 minutes

Welcome to Kubernetes for Data Engineering with Python! 10 minutes
Meet your Instructors: Noah Gift and Kennedy Behrman 10 minutes
Tools and Platforms 10 minutes
Report a problem with the course 10 minutes
What is Virtualization? 10 minutes
What is a Virtual Machine? 10 minutes
Introduction to Containers 10 minutes
Docker: The Container Platform 10 minutes
Spin up a local Docker container 10 minutes

8 assignments Total 240 minutes

Virtualization 30 minutes
Scaling Applications 30 minutes
Introduction to Virtual Machines 30 minutes
Virtual Box 30 minutes
Containers 30 minutes
Introduction to Docker 30 minutes
Docker Architecture 30 minutes
Virtualization 30 minutes

1 discussion prompt Total 10 minutes

Meet and Greet (optional) 10 minutes

In this module, you will learn how to effectively work with the Docker client, create volumes, and run databases in containers, gaining hands-on experience in managing containerized applications. You will also explore how to use the Docker command line for tasks such as building images and working with Dockerfiles, enabling you to package your software efficiently. You'll get a chance to study real-life Dockerfile examples and consult the Dockerfile reference for best practices. Furthermore, you will dive into orchestration with Docker Compose, learning how to manage multi-container applications using Compose. As an extension to this, you will be introduced to Airflow, a workflow management platform, and learn how to integrate it with Docker Compose for a seamless automation experience.

What's included

9 videos6 readings8 assignments

9 videos Total 40 minutes

Docker Client 5 minutes
Creating a Volume 5 minutes
Running a Database in a Container 6 minutes
Building an Image 5 minutes
Dockerfiles 2 minutes
Dockerfile Examples 5 minutes
Orchestration with Docker Compose 3 minutes
Introduction to Airflow 4 minutes
Airflow Demonstration using Compose 5 minutes

6 readings Total 60 minutes

Use the Docker Command Line 10 minutes
Creating a Docker Image (Step-by-Step) 10 minutes
Getting Started with Docker Compose 10 minutes
Getting Started with Apache Airflow 10 minutes
Docker vs. Kubernetes: A Primer 10 minutes
Use Docker to Spin Up Airflow 10 minutes

8 assignments Total 240 minutes

Docker Client 30 minutes
Volumes 30 minutes
Running a Database in a Container 30 minutes
Building an Image 30 minutes
Dockerfiles 30 minutes
Compose 30 minutes
Airflow 30 minutes
Docker 30 minutes

In this module, you will embark on a comprehensive journey into Kubernetes, the cornerstone of modern container orchestration. You'll begin by grasping Kubernetes key concepts, cluster architecture, and service deployments. The advantages of cloud development environments, exemplified by GitHub Codespaces, will become more clear as you explore GitHub's ecosystem and harness AI-driven coding with GitHub Copilot and OpenAI Codewhisper. The module culminates in hands-on experience as you deploy Kubernetes using Minikube within GitHub Codespaces. Gain a solid foundation in Kubernetes essentials and the power of cloud-based development, setting the stage for successful containerized application management and collaborative coding in the modern era.

What's included

14 videos7 readings8 assignments

14 videos Total 52 minutes

Kubernetes Key Concepts 2 minutes
Kubernetes Clusters 1 minute
Kubernetes Nodes 1 minute
Kubernetes Service Deployments 1 minute
Cloud Developer Workspace Advantage 4 minutes
Key Concepts in the GitHub Ecosystem 4 minutes
Using GitHub Templates 2 minutes
Using GitHub Codespaces 6 minutes
Using OpenAI Codewhisper 1 minute
Fine-Tuning a Model with Hugging Face 3 minutes
Using GitHub Copilot 8 minutes
GitHub Actions 4 minutes
Running Minikube in GitHub Codespaces 7 minutes
Deploying a Service with Minikube 7 minutes

7 readings Total 70 minutes

What is Kubernetes? 10 minutes
Virtualization, Containerization, and Elasticity 10 minutes
Fine-Tune a Pretrained Model 10 minutes
Getting Started with GitHub Copilot 10 minutes
Hello Minikube 10 minutes
Minikube + Kubernetes: A Recap 10 minutes
Deploying FastAPI to AWS with ECR and App Runner 10 minutes

8 assignments Total 240 minutes

Kubernetes Key Concepts 30 minutes
Kubernetes Clusters 30 minutes
Kubernetes Nodes 30 minutes
Kubernetes Service Deployments 30 minutes
Key Concepts in the GitHub Ecosystem 30 minutes
Running Minikube in GitHub Codespaces 30 minutes
Deploying a Service with Minikube 30 minutes
Kubernetes, GitHub, and Minikube 30 minutes

This module immerses you in the hands-on world of Kubernetes solutions. You'll start by mastering containerization, constructing FastAPI microservices, and deploying containerized applications using Azure Container Registry and Amazon Elastic Container Registry. Next, explore options for cloud-based container orchestration, featuring Google Cloud Run and AWS Copilot, and expand your coding horizons in AWS Cloud9. Finally, address critical production issues as you delve into load testing, monitoring systems, the SRE mindset for MLOps, and the art of operationalizing microservices. This module offers a comprehensive toolkit to navigate Kubernetes in real-world scenarios, combining theory and practice to prepare you for Kubernetes success.

What's included

13 videos8 readings14 assignments

13 videos Total 67 minutes

Build a Tiny Bash Container using GitHub Codespaces 9 minutes
Build FastAPI Microservice in Cloud9 in Python 6 minutes
Deploy a FastAPI PyTorch Containerized Application to AWS App Runner 8 minutes
Options for Container Orchestration 2 minutes
GCP Cloud Run 5 minutes
Build Microservice in Cloud9 in C# 7 minutes
AWS Copilot - Command Line Interface for Containerized Applications 9 minutes
Load-Testing with Locust 3 minutes
Monitoring Systems 1 minute
SRE Mindset for MLOps 5 minutes
Operationalize Microservices 2 minutes
CI for Microservices 7 minutes
What is Continuous Delivery? 3 minutes

8 readings Total 80 minutes

Using Container Registries with Kubernetes: Azure Container Registry and Amazon Elastic Container Registry (ECR) 10 minutes
Kubernetes and Google Cloud 10 minutes
Deploying Containerized Applications and Kubernetes in the Cloud with AWS 10 minutes
Getting Started with Site Reliability Engineering (SRE) 10 minutes
Continuous Delivery of FastAPI App to AWS App Runner 10 minutes
Final Project Explained 10 minutes
Next Steps 10 minutes
Share your learning experience 10 minutes

14 assignments Total 420 minutes

Build a Tiny Bash Container using GitHub Codespaces 30 minutes
Build FastAPI Microservice in Cloud9 in Python 30 minutes
Deploying a FastAPI PyTorch Containerized Application to AWS App Runner 30 minutes
Options for Container Orchestration 30 minutes
GCP Cloud Run 30 minutes
Build Microservice in Cloud9 in C# 30 minutes
AWS Copilot 30 minutes
Load-Testing with Locust 30 minutes
Monitoring Systems 30 minutes
SRE Mindset for MLOps 30 minutes
Operationalize Microservices 30 minutes
CI for Microservices 30 minutes
Continuous Delivery 30 minutes
Kubernetes Data Engineering Solutions 30 minutes

Earn a career certificate

Add this credential to your LinkedIn profile, resume, or CV. Share it on social media and in your performance review.

Instructors

Instructor ratings

(7 ratings)

Noah Gift

Duke University

40 Courses 265,419 learners

Offered by

Duke University

Explore more from Machine Learning

Coursera
Build & Publish Versioned Docker Images
Course
KodeKloud
Data Engineering Essentials
Course
Duke University
Cloud Data Engineering
Course
Duke University
Advanced Data Engineering
Course

Why people choose Coursera for their career

Felipe M.

Learner since 2018

"To be able to take courses at my own pace and rhythm has been an amazing experience. I can learn whenever it fits my schedule and mood."

Jennifer J.

Learner since 2020

"I directly applied the concepts and skills I learned from my courses to an exciting new project at work."

Larry W.

Learner since 2021

"When I need courses on topics that my university doesn't offer, Coursera is one of the best places to go."

Chaitanya A.

"Learning isn't just about being better at your job: it's so much more than that. Coursera allows me to learn without limits."

Learner reviews

5 stars
51.02%
4 stars
12.24%
3 stars
12.24%
2 stars
10.20%
1 star
14.28%

Showing 3 of 49

Reviewed on Feb 18, 2024

It was great but the last module of week 4 felt isolated from the rest of the course.

Reviewed on Mar 19, 2025

For the first time I really understood what "virtualization" is and how things sort of expand for larger and larger units. Really solid stuff on CI/CD stuff also.

View more reviews

Open new doors with Coursera Plus

Unlimited access to 10,000+ world-class courses, hands-on projects, and job-ready certificate programs - all included in your subscription

Learn more

Advance your career with an online degree

Earn a degree from world-class universities - 100% online

Explore degrees

Join over 3,400 global companies that choose Coursera for Business

Upskill your employees to excel in the digital economy

Learn more

Frequently asked questions

To access the course materials, assignments and to earn a Certificate, you will need to purchase the Certificate experience when you enroll in a course. You can try a Free Trial instead, or apply for Financial Aid. The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.

When you enroll in the course, you get access to all of the courses in the Specialization, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile.

Yes. In select learning programs, you can apply for financial aid or a scholarship if you can’t afford the enrollment fee. If fin aid or scholarship is available for your learning program selection, you’ll find a link to apply on the description page.