Production Failure Triage and Fix Loop (QA)

Production Failure Triage and Fix Loop (QA)

1. 01

   Freeze the failing workflow and capture the exact trace window before anyone edits a prompt

   Pull the session ID, timestamp range, and model version from the observability layer so the reproduction is anchored to real production state rather than a paraphrase of the complaint.

2. 02

   Classify the failure into one of four buckets: retrieval miss, tool-call error, instruction drift, or latency and cost blowout

   Bucketing first prevents the common agency mistake of rewriting system prompts when the actual defect is a stale vector index or a tool returning malformed JSON.

3. 03

   Reproduce the failure against a frozen eval set of at least 30 stored cases from the same client account

   Arize and Langfuse both support replaying stored spans, so the reproduction runs on the same inputs the client already saw instead of freshly invented test prompts.

4. 04

   Score the reproduction with the same judge configuration used in the original baseline

   Changing the judge mid-investigation invalidates the comparison; Confident AI and Braintrust both let teams pin a scoring rubric version so before-and-after numbers stay comparable.

5. 05

   Write the fix as a single change, then re-run the full eval set and record the delta

   One variable per iteration. If the fix touches both the retrieval threshold and the prompt, a later regression cannot be attributed to either.

6. 06

   Add the failing case to the permanent regression suite with a named owner and a review date

   Cekura and Agnost AI both surface recurring production failure patterns, and those patterns are the highest-value candidates for permanent test cases.

7. 07

   Log the incident in the client-facing reliability note with time-to-detect and time-to-fix

   Two numbers per incident, no narrative. This is the artifact that justifies a production-readiness premium at the next retainer conversation.