Production AgentOps
By Wanderson Leandro de Oliveira
Building an agent that works once in a notebook is the easy part. Production AgentOps is about everything that separates a demo from a system you can trust with real users and real money: how do you know it's actually good, how do you test something that gives a different answer every time you ask, how do you see inside it when it fails at 3am, how do you keep it from being turned against you, and how do you run it reliably while keeping cost and blast radius under control. This is the fourth and final course in the AI Agents Engineering track, and it assumes you already know how to build an agent — this course is about operating one. You will build evaluation pipelines with golden datasets and LLM-as-judge, write tests for non-deterministic systems with fake providers and chaos injection, instrument agents with structured logging and tracing, close a real authorization vulnerability in a production-shaped codebase, design deployment architecture with CI/CD and canary releases, and apply reliability patterns and cost controls used by teams running agents at scale. The course closes with a capstone that ties together everything from all four courses in the track into a single production-readiness design.
Course content
Evaluating Agents
- 🔒 What to measure: success criteria and golden datasets text
- 🔒 Offline and online evaluation text
- 🔒 LLM-as-a-judge text
Testing Agent Systems
- 🔒 Testing strategy for non-deterministic systems text
- 🔒 Mock tools, fake models and deterministic fixtures text
- 🔒 Chaos and error-injection testing text
Observability and Debugging
- 🔒 Structured logging, traces and spans text
- 🔒 Hands-on: instrumenting an agent text
- 🔒 Debugging non-deterministic failures text
Securing Production Agents
- 🔒 Threat modeling a production agent text
- 🔒 Guardrails and policy enforcement outside the model text
- 🔒 Hands-on: fixing a missing auth dependency text
Deploying and Operating Agents
- 🔒 Production architecture for agent systems text
- 🔒 Deploying with Docker and CI/CD text
- 🔒 AgentOps: release management and incidents text
Reliability, Cost, and Governance
- 🔒 Reliability patterns: retries, circuit breakers and degraded modes text
- 🔒 Controlling cost and unit economics text
- 🔒 Capstone: ship a production-ready agent text