Here is a scenario that should concern every enterprise architect shipping autonomous AI systems right now: An observability agent is running in production. Its job is to detect infrastructure anomalies and trigger the appropriate response. Late one night, it flags an elevated anomaly score across a production cluster, 0.87, above its defined threshold of 0.75. The agent is within its permission boundaries. It has access to the rollback service. So it uses it.

The rollback causes a four-hour outage. The anomaly it was responding to was a scheduled batch job the agent had never encountered before. There was no actual fault. The agent did not escalate. It did not ask. It acted,  confidently, autonomously, and catastrophically.

What makes this scenario particularly uncomfortable is that the failure was not in the model. The model behaved exactly as trained. The failure was in how the system was tested before it reached production. The engineers had validated happy-path behavior, run load tests, and done a security review. What they had not done is ask: what does this agent do when it encounters conditions it was never designed for?

That question is the gap I want to talk about.

Why the industry has its testing priorities backwards

The enterprise AI conversation in 2026 has largely collapsed into two areas: identity governance (who is the agent acting as?) and observability (can we see what it’s doing?). Both are legitimate concerns. Neither addresses the more fundamental question of whether your agent will behave as intended when production stops cooperating.

The Gravitee State of AI Agent Security 2026 report found that only 14.4% of agents go live with full security and IT approval. A February 2026 paper from 30-plus researchers at Harvard, MIT, Stanford, and CMU documented something even more unsettling: Well-aligned AI agents drift toward manipulation and false task completion in multi-agent environments purely from incentive structures, no adversarial prompting required. The agents weren’t broken. The system-level behavior was the problem.

This is the distinction that matters most for builders of agentic infrastructure: A model can be aligned and a system can still fail. Local optimization at the model level does not guarantee safe behavior at the system level. Chaos engineers have known this about distributed systems for fifteen years. We are relearning it the hard way with agentic AI. The reason our current testing approaches fall short is not that engineers are cutting corners. It is that three foundational assumptions embedded in traditional testing methodology break down completely with agentic systems:

• Determinism: Traditional testing assumes that given the same input, a system produces the same output. A large language model (LLM)-backed agent produces probabilistically similar outputs. This is close enough for most tasks, but dangerous for edge cases in production where an unexpected input triggers a reasoning chain no one anticipated.

• Isolated failure: Traditional testing assumes that when component A fails, it fails in a bounded, traceable way. In a multi-agent pipeline, one agent’s degraded output becomes the next agent’s poisoned input. The failure compounds and mutates. By the time it surfaces, you are debugging five layers removed from the actual source.

• Observable completion: Traditional testing assumes that when a task is done, the system accurately signals it. Agentic systems can, and regularly do, signal task completion while operating in a degraded or out-of-scope state. The MIT NANDA project has a term for this: “confident incorrectness.” I have a less polite term for it: the thing that causes the 4am incident that took three hours to trace.

Intent-based chaos testing exists to address exactly these failure modes, before your agents reach production.

The core concept: Measuring deviation from intent, not just from success

Chaos engineering as a discipline is not new. Netflix built Chaos Monkey in 2011. The principle is straightforward: Deliberately inject failure into your system to discover its weaknesses before users find them. What is new, and what the industry has not yet applied rigorously to agentic AI, is calibrating chaos experiments not just to infrastructure failure scenarios, but to behavioral intent.

The distinction is critical. When a traditional microservice fails under a chaos experiment, you measure recovery time, error rates, and availability. When an agentic AI system fails, those metrics can look perfectly normal while the agent is operating completely outside its intended behavioral boundaries: Zero errors, normal latency, catastrophically wrong decisions. This is the concept behind a chaos scale system calibrated not just to failure severity, but to how far a system’s behavior deviates from its intended purpose. I call the output of that measurement an intent deviation score.

Here is what that looks like in practice. Before running any chaos experiment against an enterprise observability agent, you define five behavioral dimensions that together describe what “acting correctly” means for that specific agent in its specific deployment context:

The weights are not arbitrary. They reflect the risk profile of the specific agent. For a read-only analytics agent, you might weight data access scope lower. For an agent with write access to production systems, completion signal accuracy and escalation fidelity are where failures become outages. The point is that you define these dimensions before you inject any failure, based on what the agent is actually supposed to do.

The deviation score is computed as a weighted average of how far each observed dimension has drifted from its baseline:

def compute_intent_deviation_score(

    baseline: dict[str, float],

    observed: dict[str, float],

    weights: dict[str, float]

) -> float:

    “””

The system computes how far an agent’s behavior has drifted from its intended baseline, and returns a score from 0.0 (no deviation) to 1.0 (complete intent violation).   

This is NOT a performance metric. Latency and error rates may look fine while this score is elevated. That’s the entire point.

    “””

    score = 0.0

    for dimension, weight in weights.items():

        baseline_val = baseline.get(dimension, 0.0)

        observed_val = observed.get(dimension, 0.0)

        # Normalize deviation relative to baseline magnitude

        raw_deviation = abs(observed_val – baseline_val) / max(abs(baseline_val), 1e-9)

        score += min(raw_deviation, 1.0) * weight

    return round(min(score, 1.0), 4)

Once you have a deviation score, you classify it into actionable levels:

The rollback agent from the opening scenario? Under this framework, it would have scored approximately 0.78 on the intent deviation scale during Phase 3 testing (catastrophic). The completion signal accuracy dimension alone would have flagged that the agent was reporting success states that did not correspond to valid system outcomes. That score would have blocked the agent from production. The four-hour outage would have been a pre-production finding instead.

The experiment structure: Four phases, expanding blast radius

The practical implementation of this framework runs in four phases, each designed to expand the chaos gradually and validate the agent’s behavioral boundaries before widening the experiment. You do not start with composite failure injection. You earn the right to each phase by passing the previous one.

Phase 1: Single tool degradation. Degrade one downstream dependency and observe how the agent adapts. Does it retry intelligently? Does it escalate when retries fail? Does it modify its tool call sequence in a reasonable way, or does it start making calls it was never designed to make? At this phase, the blast radius is intentionally narrow: One tool, one agent, no production traffic.

Phase 2: Context poisoning. Introduce corrupted or missing telemetry context,  the kind of data quality degradation that happens constantly in real enterprise environments. Missing fields, stale baselines, contradictory signals from different sources. This is where you find out whether your agent autopilots through bad data or escalates appropriately when its informational foundation is compromised.

The log schema your observability stack needs to capture to make Phase 2 meaningful is not just error counts and latency. You need intent signals:

{

  “timestamp”: “2026-03-30T02:47:13.441Z”,

  “agent_id”: “observability-agent-prod-07”,

  “action”: “triggered_rollback”,

  “decision_chain”: [

    {“step”: 1, “observation”: “anomaly_score=0.87”, “source”: “telemetry_feed”},

    {“step”: 2, “reasoning”: “score exceeds threshold,  initiating response”},

    {“step”: 3, “tool_called”: “rollback_service”, “params”: {“scope”: “prod-cluster-3”}}

  ],

  “context_completeness”: 0.62,

  “escalation_triggered”: false,

  “intent_deviation_score”: 0.78,

  “chaos_level”: “CATASTROPHIC”

}

The field that would have changed everything in the opening scenario is context_completeness: 0.62. The agent made a high-confidence, irreversible decision with 62% of its expected context available. It did not detect the missing fields. It did not escalate. A log schema that captures this turns a mysterious outage into a diagnosable engineering problem,  but only if you instrument for it before you start testing.

Phase 3: Multi-agent interference. Introduce a second agent operating on overlapping data or shared resources. This is where emergent failures from incentive misalignment surface. Two agents with individually correct behaviors can produce collectively harmful outcomes when they share write access to the same resource. This phase is where the Harvard/MIT/Stanford paper findings become directly applicable: Run your agents in a realistic multi-agent environment and watch what happens to their deviation scores.

Phase 4: Composite failure. Combine multiple simultaneous degradations: Tool latency, missing context, concurrent agents, stale baselines. This is your closest approximation to the actual entropy of a production environment. Pass criteria here should be stricter than the lower phases, not because you expect the agent to be perfect under composite failure, but because you want to understand its blast radius under the worst conditions you can reasonably anticipate.

The pass/fail criteria across all four phases follow a consistent rule: If the intent deviation score exceeds the threshold for that phase, the agent does not proceed to the next phase or to production. Full stop.

Calibrating testing depth to deployment risk

Not every agent needs all four phases. The investment in chaos testing should match the risk profile of the deployment. Here is a practical calibration matrix:

The rollback agent was in row four. It had been tested to row two. That delta is where the four-hour outage lived.

The retraining loop: The piece most teams skip

Running a chaos experiment once before deployment is necessary but not sufficient. Agentic systems evolve. They get new tool integrations. Their prompts get updated. Their data access scope expands. An agent that cleared all four phases in January with a clean bill of behavioral health may have a very different risk profile by April.

The feedback loop from chaos experiments needs to feed back into two places: The chaos scale itself (which dimensions are showing the most drift? should their weights be adjusted?) and the agent’s behavioral guardrails (which escalation thresholds are too loose? which tool permissions are too broad?).

In practice, this means treating your chaos experiment results as a governance artifact, not a PDF report that gets shared in Slack and forgotten, but a structured input to your deployment decision process. Every meaningful change to an agent’s configuration, tooling, or scope should trigger re-running the affected phases. Not a full regression — targeted re-testing of the dimensions most likely to be affected by the specific change.

This is the kind of discipline that traditional software engineering built over decades. We are building it from scratch for probabilistic, autonomous systems, and we do not have the luxury of another decade to get there.

Where this fits in the pipeline

To be clear about what this framework is and is not: Intent-based chaos testing is not a replacement for any of the testing you are already doing. Unit tests, integration tests, load tests, security red teams are all still necessary. This is an additional gate, and it belongs at a specific point in your deployment pipeline:

Development  →  Unit / Integration Tests

Staging      →  Load Testing + Security Red Team

Pre-Prod     →  Intent-Based Chaos Testing   ← the gap this fills

Production   →  Observability + Sampled Ongoing Chaos

The pre-production gate is where you answer the question that none of the other gates answer: Given realistic failure conditions, does this agent stay within its intended behavioral boundaries, or does it drift in ways that are going to cost you?

If you cannot answer that question before your agent goes live, you are not testing it. You are deploying it and hoping.

The uncomfortable arithmetic

Gartner projects that more than 40% of agentic AI projects will be canceled by the end of 2027 due to escalating costs, unclear ROI, and inadequate risk controls. Based on what I have seen building and deploying these systems, the risk controls piece is doing most of that work,  and the specific risk control that is most consistently absent is structured pre-deployment behavioral validation.

We built decades of testing discipline for deterministic software. We are starting nearly from scratch for systems that reason probabilistically, act autonomously, and operate in environments they were not specifically trained on. Intent-based chaos testing is one piece of what that discipline needs to look like. It will not prevent every incident. Nothing does. But it will ensure that when an incident happens, you either prevented it with pre-production evidence, or you made a conscious, documented decision to accept the risk.

That is a meaningfully higher bar than deploying and hoping; and right now, it is the bar most enterprise teams are not clearing.

Sayali Patil is an AI infrastructure and product leader with experience at Cisco Systems and Splunk.

The most expensive AI failure I have seen in enterprise deployments did not produce an error. No alert fired. No dashboard turned red. The system was fully operational, it was just consistently, confidently wrong. That is the reliability gap. And it is the problem most enterprise AI programs are not built to catch.

We have spent the last two years getting very good at evaluating models: benchmarks, accuracy scores, red-team exercises, retrieval quality tests. But in production, the model is rarely where the system breaks. It breaks in the infrastructure layer, the data pipelines feeding it, the orchestration logic wrapping it, the retrieval systems grounding it, the downstream workflows trusting its output. That layer is still being monitored with tools designed for a different kind of software.

The gap no one is measuring

Here’s what makes this problem hard to see: Operationally healthy and behaviorally reliable are not the same thing, and most monitoring stacks cannot tell the difference.

A system can show green across every infrastructure metric, latency within SLA, throughput normal, error rate flat, while simultaneously reasoning over retrieval results that are six months stale, silently falling back to cached context after a tool call degrades, or propagating a misinterpretation through five steps of an agentic workflow. None of that shows up in Prometheus. None of it trips a Datadog alert.

The reason is straightforward: Traditional observability was built to answer the question “is the service up?” Enterprise AI requires answering a harder question: “Is the service behaving correctly?” Those are different instruments.

 Closing this gap requires adding a behavioral telemetry layer alongside the infrastructure one — not replacing what exists, but extending it to capture what the model actually did with the context it received, not just whether the service responded.

Four failure patterns that standard monitoring will not catch

Across enterprise AI deployments in network operations, logistics, and observability platforms, I see four failure patterns repeat with enough consistency to name them.

The first is context degradation. The model reasons over incomplete or stale data in a way that is invisible to the end user. The answer looks polished. The grounding is gone. Detection usually happens weeks later, through downstream consequences rather than system alerts.

The second is orchestration drift. Agentic pipelines rarely fail because one component breaks. They fail because the sequence of interactions between retrieval, inference, tool use, and downstream action starts to diverge under real-world load. A system that looked stable in testing behaves very differently when latency compounds across steps and edge cases stack.

The third is a silent partial failure. One component underperforms without crossing an alert threshold. The system degrades behaviorally before it degrades operationally. These failures accumulate quietly and surface first as user mistrust, not incident tickets. By the time the signal reaches a postmortem, the erosion has been happening for weeks.

The fourth is the automation blast radius. In traditional software, a localized defect stays local. In AI-driven workflows, one misinterpretation early in the chain can propagate across steps, systems, and business decisions. The cost is not just technical. It becomes organizational, and it is very hard to reverse.

Metrics tell you what happened. They rarely tell you what almost happened.

Why classic chaos engineering is not enough and what needs to change

Traditional chaos engineering asks the right kind of question: What happens when things break? Kill a node. Drop a partition. Spike CPU. Observe. Those tests are necessary, and enterprises should run them.

But for AI systems, the most dangerous failures are not caused by hard infrastructure faults. They emerge at the interaction layer between data quality, context assembly, model reasoning, orchestration logic, and downstream action. You can stress the infrastructure all day and never surface the failure mode that costs you the most.

What AI reliability testing needs is an intent-based layer: Define what the system must do under degraded conditions, not just what it should do when everything works. Then test the specific conditions that challenge that intent. What happens if the retrieval layer returns content that is technically valid but six months outdated? What happens if a summarization agent loses 30% of its context window to unexpected token inflation upstream? What happens if a tool call succeeds syntactically but returns semantically incomplete data? What happens if an agent retries through a degraded workflow and compounds its own error with each step?

These scenarios are not edge cases. They are what production looks like. This is the framework I have applied in building reliability systems for enterprise infrastructure: Intent-based chaos level creation for distributed computing environments. The key insight: Intent defines the test, not just the fault.

What the infrastructure layer actually needs

None of this requires reinventing the stack. It requires extending four things.

Add behavioral telemetry alongside infrastructure telemetry. Track whether responses were grounded, whether fallback behavior was triggered, whether confidence dropped below a meaningful threshold, whether the output was appropriate for the downstream context it entered. This is the observability layer that makes everything else interpretable.

Introduce semantic fault injection into pre-production environments. Deliberately simulate stale retrieval, incomplete context assembly, tool-call degradation, and token-boundary pressure. The goal is not theatrical chaos. The goal is finding out how the system behaves when conditions are slightly worse than your staging environment — which is always what production is.

Define safe halt conditions before deployment, not after the first incident. AI systems need the equivalent of circuit breakers at the reasoning layer. If a system cannot maintain grounding, validate context integrity, or complete a workflow with enough confidence to be trusted, it should stop cleanly, label the failure, and hand control to a human or a deterministic fallback. A graceful halt is almost always safer than a fluent error. Too many systems are designed to keep going because confident output creates the illusion of correctness.

Assign shared ownership for end-to-end reliability. The most common organizational failure is a clean separation between model teams, platform teams, data teams, and application teams. When the system is operationally up but behaviorally wrong, no one owns it clearly. Semantic failure needs an owner. Without one, it accumulates.

The maturity curve is shifting

For the last two years, the enterprise AI differentiator has been adoption — who gets to production fastest. That phase is ending. As models commoditize and baseline capability converges, competitive advantage will come from something harder to copy: The ability to operate AI reliably at scale, in real conditions, with real consequences.

Yesterday’s differentiator was model adoption. Today’s is system integration. Tomorrow’s will be reliability under production stress.

The enterprises that get there first will not have the most advanced models. They will have the most disciplined infrastructure around them — infrastructure that was tested against the conditions it would actually face, not the conditions that made the pilot look good.

The model is not the whole risk. The untested system around it is.

Sayali Patil is an AI infrastructure and product leader.

The stochastic challenge

Traditional software is predictable: Input A plus function B always equals output C. This determinism allows engineers to develop robust tests. On the other hand, generative AI is stochastic and unpredictable. The exact same prompt often yields different results on Monday versus Tuesday, breaking the traditional unit testing that engineers know and love.

To ship enterprise-ready AI, engineers cannot rely on mere “vibe checks” that pass today but fail when customers use the product. Product builders need to adopt a new infrastructure layer: The AI Evaluation Stack.

This framework is informed by my extensive experience shipping AI products for Fortune 500 enterprise customers in high-stakes industries, where “hallucination” is not funny — it’s a huge compliance risk.

Defining the AI evaluation paradigm

Traditional software tests are binary assertions (pass/fail). While some AI evals use binary asserts, many evaluate on a gradient. An eval is not a single script; it is a structured pipeline of assertions — ranging from strict code syntax to nuanced semantic checks — that verify the AI system’s intended function.

The taxonomy of evaluation checks

To build a robust, cost-effective pipeline, asserts must be separated into two distinct architectural layers:

Layer 1: Deterministic assertions

A surprisingly large share of production AI failures aren’t semantic “hallucinations” — they are basic syntax and routing failures. Deterministic assertions serve as the pipeline’s first gate, using traditional code and regex to validate structural integrity.

Instead of asking if a response is “helpful,” these assertions ask strict, binary questions:

• Did the model generate the correct JSON key/value schema?

• Did it invoke the correct tool call with the required arguments?

• Did it successfully slot-fill a valid GUID or email address?

// Example: Layer 1 Deterministic Tool Call Assertion

{

  “test_scenario”: “User asks to look up an account”,

  “assertion_type”: “schema_validation”,

  “expected_action”: “Call API: get_customer_record”,

  “actual_ai_output”: “I found the customer.”,

  “eval_result”: “FAIL – AI hallucinated conversational text instead of generating the required API payload.”

}

In the example above, the test failed instantly because the model generated conversational text instead of the required tool call payload.

Architecturally, deterministic assertions must be the first layer of the stack, operating on a computationally inexpensive “fail-fast” principle. If a downstream API requires a specific schema, a malformed JSON string is a fatal error. By failing the evaluation immediately at this layer, engineering teams prevent the pipeline from triggering expensive semantic checks (Layer 2) or wasting valuable human review time (Layer 3).

Layer 2: Model-based assertions

When deterministic assertions pass, the pipeline must evaluate semantic quality. Because natural language is fluid, traditional code cannot easily assert if a response is “helpful” or “empathetic.” This introduces model-based evaluation, commonly referred to as “LLM-as-a-Judge” or “LLM-Judge.”

While using one non-deterministic system to evaluate another seems counterintuitive, it is an exceptionally powerful architectural pattern for use cases requiring nuance. It is virtually impossible to write a reliable regex to verify if a response is “actionable” or “polite.” While human reviewers excel at this nuance, they cannot scale to evaluate tens of thousands of CI/CD test cases. Thus, the LLM-as-a-Judge becomes the scalable proxy for human discernment.

3 critical inputs for model-based assertions

However, model-based assertions only yield reliable data when the LLM-as-a-Judge is provisioned with three critical inputs:

1. A state-of-the-art reasoning model: The Judge must possess superior reasoning capabilities compared to the production model. If your app runs on a smaller, faster model for latency, the judge must be a frontier reasoning model to approximate human-level discernment.

2. A strict assessment rubric: Vague evaluation prompts (“Rate how good this answer is”) yield noisy, stochastic evaluations. A robust rubric explicitly defines the gradients of failure and success. (For example, a “Helpfulness” rubric should define Score 1 as an irrelevant refusal, Score 2 as addressing the prompt but lacking actionable steps, and Score 3 as providing actionable next steps strictly within context.)

3. Ground truth (golden outputs): While the rubric provides the rules, a human-vetted “expected answer” acts as the answer key. When the LLM-Judge can compare the production model’s output against a verified Golden Output, its scoring reliability increases dramatically.

Architecture: The offline vs online pipeline

A robust evaluation architecture requires two complementary pipelines. The online pipeline monitors post-deployment telemetry, while the offline pipeline provides the foundational baseline and deterministic constraints required to evaluate stochastic models safely.

The offline evaluation pipeline

The offline pipeline’s primary objective is regression testing — identifying failures, drift, and latency before production. Deploying an enterprise LLM feature without a gating offline evaluation suite is an architectural anti-pattern; it is the equivalent of merging uncompiled code into a main branch.

Process

1. Curating the golden dataset

The offline lifecycle begins by curating a “golden dataset” — a static, version-controlled repository of 200 to 500 test cases representing the AI’s full operational envelope. Each case pairs an exact input payload with an expected “golden output” (ground truth).

Crucially, this dataset must reflect expected real-world traffic distributions. While most cases cover standard “happy-path” interactions, engineers must systematically incorporate edge cases, jailbreaks, and adversarial inputs. Evaluating “refusal capabilities” under stress remains a strict compliance requirement.

Example test case payload (standard tool use):

• Input: “Schedule a 30-minute follow-up meeting with the client for next Tuesday at 10 a.m.”

• Expected output (golden): The system successfully invokes the schedule_meeting tool with the correct JSON payload: {“duration_minutes”: 30, “day”: “Tuesday”, “time”: “10 AM”, “attendee”: “client_email”}.

While manually curating hundreds of edge cases is tedious, the process can be accelerated with synthetic data generation pipelines that use a specialized LLM to produce diverse TSV/CSV test payloads. However, relying entirely on AI-generated test cases introduces the risk of data contamination and bias. A human-in-the-loop (HITL) architecture is mandatory at this stage; domain experts must manually review, edit, and validate the synthetic dataset to ensure it accurately reflects real-world user intent and enterprise policy before it is committed to the repository.

2. Defining the evaluation criteria

Once the dataset is curated, engineers must design the evaluation criteria to compute a composite score for each model output. A robust architecture achieves this by assigning weighted points across a hybrid of Layer 1 (deterministic) and Layer 2 (model-based) asserts.

Consider an AI agent executing a “send email” tool. An evaluation framework might utilize a 10-point scoring system:

• Layer 1: Deterministic asserts (6 points): Did the agent invoke the correct tool? (2 pts). Did it produce a valid JSON object? (2 pts). Does the JSON strictly adhere to the expected schema? (2 pts).

• Layer 2: Model-based asserts (4 points): (Note: Semantic rubrics must be highly use-case specific). Does the subject line reflect user intent? (1 pt). Does the email body match expected outputs without hallucination? (1 pt). Were CC/BCC fields leveraged accurately? (1 pt). Was the appropriate priority flag inferred? (1 pt).

To understand why the LLM-Judge awarded these points, the engineer must prompt the judge to supply its reasoning for each score. This is crucial for debugging failures.

The passing threshold and short-circuit logic 

In this example, an 8/10 passing threshold requires 8 points for success. Crucially, the evaluation pipeline must enforce strict short-circuit evaluation (fail-fast logic). If the model fails any deterministic assertion — such as generating a malformed JSON schema — the system must instantly fail the entire test case (0/10). There is zero architectural value in invoking an expensive LLM-Judge to assess the semantic “politeness” of an email if the underlying API call is structurally broken.

3. Executing the pipeline and aggregating signals

Using an evaluation infrastructure of choice, the system executes the offline pipeline — typically integrated as a blocking CI/CD step during a pull request. The infrastructure iterates through the golden dataset, injecting each test payload into the production model, capturing the output, and executing defined assertions against it.

Each output is scored against the passing threshold. Once batch execution is complete, results are aggregated into an overall pass rate. For enterprise-grade applications, the baseline pass rate must typically exceed 95%, scaling to 99%-plus for strict compliance or high-risk domains.

4. Assessment, iteration, and alignment

Based on aggregated failure data, engineering teams conduct a root-cause analysis of failing test cases. This assessment drives iterative updates to core components: refining system prompts, modifying tool descriptions, augmenting knowledge sources, or adjusting hyperparameters (like temperature or top-p). Continuous optimization remains best practice even after achieving a 95% pass rate.

Crucially, any system modification necessitates a full regression test. Because LLMs are inherently non-deterministic, an update intended to fix one specific edge case can easily cause unforeseen degradations in other areas. The entire offline pipeline must be rerun to validate that the update improved quality without introducing regressions.

The online evaluation pipeline

While the offline pipeline acts as a strict pre-deployment gatekeeper, the online pipeline is the post-deployment telemetry system. Its objective is to monitor real-world behavior, capturing emergent edge cases, and quantifying model drift. Architects must instrument applications to capture five distinct categories of telemetry:

1. Explicit user signals

Direct, deterministic feedback indicating model performance:

• Thumbs up/down: Disproportionate negative feedback is the most immediate leading indicator of system degradation, directing immediate engineering investigation.

• Verbatim in-app feedback: Systematically parsing written comments identifies novel failure modes to integrate back into the offline “golden dataset.”

2. Implicit behavioral signals

Behavioral telemetry reveals silent failures where users give up without explicit feedback:

• Regeneration and retry rates: High frequencies of retries indicate the initial output failed to resolve user intent.

• Apology rate: Programmatically scanning for heuristic triggers (“I’m sorry”) detects degraded capabilities or broken tool routing.

• Refusal rate: Artificially high refusal rates (“I can’t do that”) indicate over-calibrated safety filters rejecting benign user queries.

3. Production deterministic asserts (synchronous)

Because deterministic code checks execute in milliseconds, teams can seamlessly reuse Layer 1 offline asserts (schema conformity, tool validity) to synchronously evaluate 100% of production traffic. Logging these pass/fail rates instantly detects anomalous spikes in malformed outputs — the earliest warning sign of silent model drift or provider-side API changes.

4. Production LLM-as-a-Judge (asynchronous)

If strict data privacy agreements (DPAs) permit logging user inputs, teams can deploy model-based asserts. Architecturally, production LLM-Judges must never execute synchronously on the critical path, which doubles latency and compute costs. Instead, a background LLM-Judge asynchronously samples a fraction (5%) of daily sessions, grading outputs against the offline rubric to generate a continuous quality dashboard.

Engineering the feedback loop (the “flywheel”)

Evaluation pipelines are not “set-it-and-forget-it” infrastructure. Without continuous updates, static datasets suffer from “rot” (concept drift) as user behavior evolves and customers discover novel use cases.

For example, an HR chatbot might boast a pristine 99% offline pass rate for standard payroll questions. However, if the company suddenly announces a new equity plan, users will immediately begin prompting the AI about vesting schedules — a domain entirely missing from the offline evaluations.

To make the system smarter over time, engineers must architect a closed feedback loop that mines production telemetry for continuous improvement.

The continuous improvement workflow:

1. Capture: A user triggers an explicit negative signal (a “thumbs down”) or an implicit behavioral flag in production.

2. Triage: The specific session log is automatically flagged and routed for human review.

3. Root-cause analysis: A domain expert investigates the failure, identifies the gap, and updates the AI system to successfully handle similar requests.

4. Dataset augmentation: The novel user input, paired with the newly corrected expected output, is appended to the offline Golden Dataset alongside several synthetic variations.

5. Regression testing: The model is continuously re-evaluated against this newly discovered edge case in all future runs.

Building an evaluation pipeline without monitoring production logs and updating datasets is fundamentally insufficient. Users are unpredictable. Evaluating on stale data creates a dangerous illusion: High offline pass rates masking a rapidly degrading real-world experience.

Conclusion: The new “definition of done”

In the era of generative AI, a feature or product is no longer “done” simply because the code compiles and the prompt returns a coherent response. It is only done when a rigorous, automated evaluation pipeline is deployed and stable — and when the model consistently passes against both a curated golden dataset and newly discovered production edge cases.

This guide has equipped you with a comprehensive blueprint for building that reality. From architecting offline regression pipelines and online telemetry to the continuous feedback flywheel and navigating enterprise anti-patterns, you now have the structural foundation required to deploy AI systems with greater confidence.

Now, it is your turn. Share this framework with your engineering, product, and legal teams to establish a unified, cross-functional standard for AI quality in your organization. Stop guessing whether your models are degrading in production, and start measuring.

Derah Onuorah is a Microsoft senior product manager.