Retrieval Quality Baseline (Onboarding)

Retrieval Quality Baseline (Onboarding)

1. 01

   Freeze a 50-question golden set from the client's own documents before any indexing work starts

   Pull real questions from support tickets, sales call notes, and the last two quarters of client email threads. Store the set in version control so every later retrieval change is measured against the same yardstick.

2. 02

   Record the client's acceptable answer standard in writing, including what counts as a citation failure

   Agencies that skip this step end up arguing about accuracy after go-live. Get the client to sign off on two or three worked examples of a passing and a failing answer.

3. 03

   Inventory every source system and note its update cadence and access method

   SharePoint, Drive, Zendesk, and a shared inbox each carry different refresh rates. A source that changes hourly needs a different ingestion schedule than a policy PDF revised twice a year.

4. 04

   Run the golden set against two retrieval configurations and log hit rate, latency, and cost per query

   Managed context engine APIs such as Ragie handle parsing, entity extraction, and hybrid vector plus keyword indexing out of the box, which makes a two-configuration bake-off a same-day exercise rather than a two-week build.

5. 05

   Confirm whether the client's domain requires deterministic, auditable decisions rather than probabilistic retrieval

   Regulated work such as loan underwriting or fraud review may need a rule engine deciding the outcome while retrieval only grounds the explanation, the pattern ai·rete·rag is built around. Flag this before the architecture is chosen, not after.

6. 06

   Write the baseline numbers into the statement of work as the accuracy floor for the engagement

   State the measured hit rate, the p95 latency, and the per-query cost. Anything below the floor becomes a rework item rather than a scope debate.

7. 07

   Name the internal owner who will re-run the golden set on a fixed schedule

   One named person, one recurring calendar slot. Retrieval quality drifts as source documents change, and an unowned benchmark quietly becomes stale.