Generic chatbot → 45% support deflection

The problem

A DTC brand was paying for tier-1 support tickets that were mostly answered by their own help center, just buried under bad search.

The solution

A context-aware RAG chatbot trained on product docs, returns/shipping policy, and order data — with grounded answers and citations, plus a graceful handoff to humans.

Approach

Retrieval was the real problem, not generation. I rebuilt the index around product taxonomy and added a hybrid retriever that handles SKUs and natural language equally well.

Detail

Every answer ships with citations and a confidence score. Below a threshold, we hand off to a human and pre-fill the ticket with the conversation summary.

Sample evaluation harness

// evals/extraction.eval.ts
import { runEval } from "./harness";
import { invoiceCases } from "./cases";

await runEval({
  name: "invoice-extraction",
  cases: invoiceCases,
  judge: "strict-json-schema",
  passThreshold: 0.95,
});