When you enroll in this course, you'll also be enrolled in this Professional Certificate.
Learn new concepts from industry experts
Gain a foundational understanding of a subject or tool
Develop job-relevant skills with hands-on projects
Earn a shareable career certificate from Coursera
There are 7 modules in this course
This long course develops skills for operational analytics, secure data practices, and governance essential to building trustworthy, auditable agentic systems. You will aggregate and analyze operational metrics, design A/B experiments and statistical tests to validate agent improvements, and craft clear visualizations and alerting rules for stakeholders. The course covers end-to-end data hygiene: cleaning, schema validation, reproducible notebooks with data versioning, and trade-offs between sample size and noise for experimental design. It also addresses security and governance: securing API endpoints per OWASP ASVS, dependency vulnerability analysis, secret-management trade-offs (on-prem vs managed), and threat modeling (STRIDE). Practical tasks include building DBT models for telemetry, configuring alerts, producing reproducible analytic notebooks, and creating STRIDE diagrams with documented mitigations to reduce operational and supply-chain risk.
This module trains data analysts, ML engineers, and developers to optimize AI agents built with frameworks like LangChain and Autogen and learn to prove the effectiveness of the agents. You will transform raw logs into actionable KPIs using SQL and dbt, design and execute A/B tests to compare agent versions, and apply statistical methods like the Chi-square test to validate your results. This course equips you to make objective, evidence-based recommendations for deploying agent enhancements, moving from correlation to causation and ensuring your improvements are statistically significant.
What's included
5 videos2 readings4 assignments1 ungraded lab
Show info about module content
5 videos•Total 27 minutes
Defining Agent Success: From Vanity Metrics to Actionable KPIs•6 minutes
The Modern Data Stack for AI•6 minutes
Correlation is not Causation•5 minutes
Running a Chi-square Test•5 minutes
Non-Parametric Tests•6 minutes
2 readings•Total 15 minutes
Advanced Time-Series Aggregation: Windows, Bucketing, and Operational Definitions•7 minutes
Principles of A/B Testing•8 minutes
4 assignments•Total 70 minutes
Agent Performance Analysis Report•30 minutes
Build an Agent Performance Data Model•20 minutes
Knowledge Check: Data Transformation for Business Intelligence•10 minutes
Knowledge Check: Statistical Significance in Agent Experiments•10 minutes
1 ungraded lab•Total 25 minutes
Analyze a Controlled Experiment•25 minutes
Visualize and Alert AI Performance KPIs
Module 2•2 hours to complete
Module details
This module is for training data analysts, ML engineers, and product managers to monitor the operational health of AI systems by focusing on cost, latency, and impact. You will master data storytelling, transforming complex performance data into clear, compelling visualizations that drive decisions. Through hands-on labs, you will learn to build proactive monitoring systems by defining critical KPIs, setting precise thresholds, and configuring automated alerts. By the end, you can create dashboards that empower leadership and build automated defenses to protect your AI systems from budget overruns and performance degradation, ensuring real-world success.
What's included
4 videos4 readings4 assignments1 ungraded lab
Show info about module content
4 videos•Total 23 minutes
Dashboard Failure: The Cost of Clutter•6 minutes
Choosing the Right Visualization Type•5 minutes
The High Cost of Unmonitored AI•8 minutes
How to Configure an Alert in a BI Tool•4 minutes
4 readings•Total 29 minutes
What Makes a Visualization Effective?•10 minutes
How to Redesign a Cluttered Chart•7 minutes
What is an Effective Alerting System?•6 minutes
Best Practices for Alerting•6 minutes
4 assignments•Total 68 minutes
Visualizing and Alerting on AI KPIs•30 minutes
Knowledge Check: Data Visualization Best Practices•10 minutes
Hands-On Learning: Designing a Cost Management Alerting Plan•18 minutes
Knowledge Check: Proactive Alerting for AI Cost and Performance Management
•10 minutes
1 ungraded lab•Total 25 minutes
Redesigning a Performance Visualization•25 minutes
Clean, Analyze, and Visualize Your Data
Module 3•2 hours to complete
Module details
This module, designed for aspiring AI and data professionals, provides hands-on experience in data preparation and exploration. You will learn to build world-class models on high-quality data by implementing systematic cleaning and validation routines with tools like Pandera. In guided Jupyter labs, you will master statistical visualization and dimensionality reduction techniques, such as t-SNE, to transform complex data into clear, interpretable plots. Uncover hidden patterns, diagnose issues, and derive key insights. You'll move beyond just cleaning data to truly understanding it, ensuring your AI development is built on a solid foundation.
What's included
3 videos2 readings3 assignments2 ungraded labs
Show info about module content
3 videos•Total 13 minutes
How to Build a Validation Schema with Pandera•4 minutes
Seeing the Unseen: Finding a Hidden Error Cluster•5 minutes
How to Create and Interpret a t-SNE Plot•5 minutes
2 readings•Total 18 minutes
The Data Wrangler's Toolkit: Core Cleaning Concepts•8 minutes
Taming the Dimensions: An Introduction to t-SNE and PCA•10 minutes
3 assignments•Total 55 minutes
Report: From Data Cleaning to Visual Insight•30 minutes
Data Validation and Imputation: Quiz •15 minutes
Analyzing a New Visualization •10 minutes
2 ungraded labs•Total 40 minutes
Cleaning a Raw Customer Dataset•20 minutes
Visualizing Message Embeddings to Find Errors•20 minutes
Evaluate and Reproduce Data Findings Fast
Module 4•3 hours to complete
Module details
This module helps data scientists and analysts deliver efficient, trustworthy results. Tackle critical questions like, "Is our data sufficient?" and "Are our findings replicable?" Learn statistical power analysis to optimize sample sizes, preventing wasted resources. You will master reproducible workflows by parameterizing Jupyter notebooks with Papermill and versioning data with DVC. Move beyond simple scripts to build robust, automated analytical projects that accelerate innovation and foster a culture of trust, ensuring your findings can be validated by peers and stakeholders.
What's included
3 videos2 readings4 assignments1 ungraded lab
Show info about module content
3 videos•Total 17 minutes
The Trade-Off Triangle: Sample Size, Noise, and Confidence•6 minutes
Why Reproducibility Matters?•4 minutes
How to Build a Reproducible Notebook?•7 minutes
2 readings•Total 14 minutes
The Point of Diminishing Returns•7 minutes
The Reproducibility Toolkit: Papermill and DVC•7 minutes
4 assignments•Total 85 minutes
Reproducible Data Analysis Project•30 minutes
Hands-On Learning: Analyzing Sample Size and Diminishing Returns•25 minutes
This module transforms developers into defenders, teaching you to build secure, production-grade AI. Learn to harden API endpoints using OWASP guidelines by implementing JWT authentication, input validation, and rate limiting. Adopt an attacker’s mindset, using DAST tools like OWASP ZAP to verify your defenses. You'll master software supply chain security by analyzing vulnerabilities, prioritizing threats with the CVSS framework, and creating hotfix and rollback plans. Through hands-on labs simulating real security incidents, you will be prepared to build and deploy resilient AI services against modern threats.
What's included
4 videos4 readings5 assignments
Show info about module content
4 videos•Total 17 minutes
JWT: Authentication and Access Control in AI Services•4 minutes
The Tester's Mindset: From Coder to Attacker•4 minutes
CVSS Explained: Technical Severity vs. Contextual Risk•5 minutes
Hotfix Strategy: Compatibility and Rollback Planning•4 minutes
4 readings•Total 27 minutes
Securing the Gates: The OWASP API Security Top 10•5 minutes
Input Validation: The Primary Defense Against Injection•7 minutes
The Log4j Case Study: Anatomy of a Supply Chain Crisis•7 minutes
The CVSS Framework: A Deeper Dive•8 minutes
5 assignments•Total 60 minutes
Security Portfolio and SecOps Defense•15 minutes
Hands-On Learning: Implement Authentication and Validation Guards•10 minutes
Hands-On Learning: Verification with Dynamic Security Testing (DAST)•15 minutes
Response: Defending Against the Next Attack•10 minutes
Hands-On Learning: Scan Report Analysis: Spotting the Critical CVE in urllib3•10 minutes
Secure Your AI: Threat Modeling
Module 6•2 hours to complete
Module details
This module teaches architects and engineers to build resilience directly into AI system designs. You'll master secret management by comparing self-hosted (Vault) and cloud (AWS Secrets Manager) solutions, using Total Cost of Ownership (TCO) analysis to make a justifiable recommendation. Learn to proactively hunt for vulnerabilities by deconstructing architecture with Data Flow Diagrams and applying the STRIDE framework to mitigate threats. Through hands-on projects, you will draft professional security documents, defend your decisions, and gain the skills to design, build, and maintain secure AI systems from the ground up.
What's included
4 videos5 readings6 assignments
Show info about module content
4 videos•Total 19 minutes
TCO and Compliance: A Cost-Benefit Deep Dive•5 minutes
Architect's Choice: Documenting Your Recommendation•6 minutes
DFDs and Trust Boundaries: Decomposing AI Architecture•5 minutes
STRIDE in Practice: Identifying Spoofing and Information Disclosure•3 minutes
5 readings•Total 30 minutes
Cloud vs. On-Prem: The Secret Management Trade-off•7 minutes
Integration and Latency: Prototyping Your Connection•6 minutes
The Power of Proactivity: Threat Modeling in DevSecOps•6 minutes
STRIDE: Your Framework for Systematic Threat Identification•6 minutes
Targeted Mitigations: Countering Spoofing and Info Disclosure•5 minutes
6 assignments•Total 76 minutes
Architectural Review and Mitigation Proposal•16 minutes
Hands-On Learning: Prototype and Compare Solutions•15 minutes
Hands-On Learning: Draft the Technical Recommendation•10 minutes
Justification of Secret Management Decision•10 minutes
Hands-On Learning: Scan Report Analysis: Diagramming the Chat-Agent•10 minutes
Hands-On Learning: STRIDE Analysis and Mitigation Plan•15 minutes
Governance, Alerts and Analytics
Module 7•2 hours to complete
Module details
In this hands-on module, you'll master governance, alerting, and analytics by building a complete, reproducible telemetry-to-alert pipeline. Using automated notebooks, you will construct a workflow that ingests raw system data and generates critical, real-time alerts. To embed security directly into your design, you will apply the industry-standard STRIDE framework to develop a proactive threat model, identifying and mitigating vulnerabilities before they are exploited. This module will equip you with the skills to translate data into actionable intelligence, creating a robust, automated system for maintaining secure and reliable operations in a production environment.
What's included
2 readings1 assignment
Show info about module content
2 readings•Total 30 minutes
Why This Project Matters: Building Trust in Automated Systems •5 minutes
Your Project Blueprint: Requirement and Evaluation•25 minutes
1 assignment•Total 90 minutes
Project: Governance, Alerts and Analytics•90 minutes
Earn a career certificate
Add this credential to your LinkedIn profile, resume, or CV. Share it on social media and in your performance review.
Coursera brings together a diverse network of subject matter experts who have demonstrated their expertise through professional industry experience or strong academic backgrounds. These instructors design and teach courses that make practical, career-relevant skills accessible to learners worldwide.
Is Analyzing and Securing AI System Performance suitable for beginners?
This course assumes practical ML and engineering experience. Beginners should complete foundational ML and data-engineering courses first to gain the necessary background for the labs.
What hands-on projects are included in this course?
Labs include building telemetry-to-alert pipelines, creating DBT models and reproducible notebooks, configuring dashboards and alerts, and producing STRIDE threat models with mitigations suitable for a portfolio artifact.
What tools will I use in the course?
The curriculum references telemetry tooling, DBT, reproducible notebooks, and dependency scanners. Exact tool choices and versions will be confirmed by instructors and may vary by offering.
When will I have access to the lectures and assignments?
To access the course materials, assignments and to earn a Certificate, you will need to purchase the Certificate experience when you enroll in a course. You can try a Free Trial instead, or apply for Financial Aid. The course may offer 'Full Course, No Certificate' instead. This option lets you see all course materials, submit required assessments, and get a final grade. This also means that you will not be able to purchase a Certificate experience.
What will I get if I subscribe to this Certificate?
When you enroll in the course, you get access to all of the courses in the Certificate, and you earn a certificate when you complete the work. Your electronic Certificate will be added to your Accomplishments page - from there, you can print your Certificate or add it to your LinkedIn profile.