Agent Failure Drill (QA)

Agent Failure Drill (QA)

1. 01

   Freeze the workflow version and capture the exact input that broke the chain

   Record the run ID, the model version behind each agent, and the payload that entered the first node. Without a frozen snapshot, a re-run tells you nothing about what actually failed.

2. 02

   Kill each agent in turn and log what the downstream nodes do

   A chain that silently returns partial output is more dangerous than one that errors loudly. Note whether the orchestrator retries, passes nulls forward, or halts.

3. 03

   Classify the failure as model drift, schema mismatch, tool timeout, or bad retrieval

   Model swaps land weekly now, and a prompt tuned for one version can degrade on the next. Tagging the cause prevents the team from blaming the wrong layer.

4. 04

   Write the fallback branch before the client sees the failure

   Every node needs a defined degraded path: a cheaper model, a cached response, or a human queue. Document which one fires and who gets paged.

5. 05

   Re-run the full chain against a stored test set of 20 to 50 real client inputs

   Platforms such as AgentX ship evaluation pipelines for exactly this purpose, running agents against test sets before deployment. Use the same discipline on every change.

6. 06

   Log the incident in a shared register with time-to-detect and time-to-recover

   Two numbers per incident, tracked over a quarter, tell you whether the retainer is priced for the real support load. Agencies that skip this step discover the cost during renewal.

7. 07

   Update the client-facing reliability note with the new known-limits line

   State plainly what the workflow does not handle. A documented boundary is a stronger sales asset than a vague uptime claim.