The golden dataset that gates your CI: building it, scoring it, setting the bar

Let me make this concrete with one product and carry it through the whole post. Picture the support assistant for an online travel agency — a chatbot that answers traveller questions like 'can I cancel my Bali package and get a refund?' or 'what's the baggage allowance on my flight?' by retrieving from cancellation policies, fare rules, and package terms, then writing a reply. It's a textbook RAG system, and it's exactly the kind that fails quietly.

Here's the failure that keeps a founder up at night. An engineer tweaks the assistant's prompt on a Friday to make answers 'friendlier'. It reads fine on the three questions they test. What they don't notice: the new phrasing makes the model soften the cancellation rules, so a traveller cancelling 10 days before departure — squarely in the non-refundable window — is now told they'll get 'a partial refund'. Travellers act on that. They cancel expecting money back, then dispute the charge when it doesn't come. Two weeks later, support is drowning in 'but your bot promised me a refund', and finance is eating chargebacks the fare rules never allowed. Nobody wrote a bug. The demo looked great. Quality drifted, silently, and it cost real money.

A golden dataset is what catches that on the pull request, before it ships. The eval-discipline post argues for the principle; this is the practical how, worked through that agency. It's harder than it sounds — you have to decide what a 'row' is, build thousands of them, score them with the right metrics, and turn a fuzzy sense of quality into a number that either passes or blocks the merge.

What one row looks like

A golden row is not just a question-and-answer pair. It's a question plus everything you need to grade the answer automatically. For our cancellation question, one row looks like this:

{
  "id": "cancel-071",
  "input": "Can I cancel my Bali package 10 days before departure
            and get a refund?",
  "expected": "No. Cancellations within 15 days of departure are
               non-refundable; 15-30 days before, 50% is refunded.",
  "must_retrieve": ["cancellation-policy", "bali-package-terms"],
  "surface": "support-chat",
  "difficulty": "medium",
  "tags": ["cancellation", "refund", "packages"]
}

The input is the traveller's question. The expected answer is the acceptance criteria — sometimes an exact answer, sometimes a rubric ('must state the cut-off window and the refund tier'). The metadata (surface, difficulty, tags) lets you slice results later — 'how are we doing on cancellation questions specifically?'. And the field that matters most for RAG is must_retrieve: the IDs of the policy documents that should have been pulled. Without knowing the right documents, you can't tell whether a wrong answer came from fetching the wrong policy or from misreading the right one.

Building thousands of rows: three sources

You are not going to hand-write five thousand of these. You assemble the set from three sources, each covering the others' blind spots. For the travel agency:

• Synthetic generation — point a strong model at your policy documents and fare rules and have it generate questions each one should answer ('given the cancellation policy, write 10 questions a traveller might ask'). This gets you broad coverage in an afternoon — every policy, every destination, every fare class. The catch: synthetic questions are politely phrased and complete, and real travellers are neither.
• Production log mining — pull real questions from actual chat logs. This is the gold, because it's how people really type: 'cancel bali trip get money back', misspellings, no punctuation, a visa question and a baggage question crammed into one message. Cluster them by topic (cancellations, refunds, baggage, date changes, visas) and sample across clusters so cancellations doesn't swamp the rare visa edge cases — and scrub names, emails, and booking references before anything is saved.
• Human-in-the-loop curation — a support lead reviews a slice, fixes any expected answers the model got wrong, and — most valuable — turns every real complaint into a permanent row. The 'partial refund' incident above becomes row cancel-118, so that exact failure can never ship again. The set gets harder and more realistic every week; it's a living asset, not a one-time export.

Balance and freshness are what make the final number trustworthy. Keep the hard cases over-represented — an easy-heavy set stays a cheerful green while real quality slips, because the regressions hide in the edge cases. And refresh it when the business changes: the day an airline updates its fare rules or the agency runs a peak-season cancellation waiver, every affected row's expected answer is now wrong, and a stale golden set will happily bless a bot that quotes the old policy.

Scoring: check retrieval and answer separately

When the bot gives a bad answer, there are two very different culprits, and you must tell them apart (the evaluating-retrieval-vs-generation post is entirely about this). Did it fetch the wrong policy, or did it fetch the right one and then write a bad reply?

• Retrieval metrics — measured against the row's must_retrieve list. Did the cancellation-policy and package-terms documents actually come back in the top results? Context recall asks 'did we get the right docs at all?'; context precision asks 'or did we also drag in ten irrelevant ones?'. If the wrong policy was fetched, no amount of prompt-tuning saves the answer.
• Generation metrics — given the right policy, was the reply any good? Faithfulness: is every claim backed by the retrieved rules, or did the model invent a refund the policy never offered? Answer relevancy: did it actually address the question? Correctness: does it match the expected answer? These are mostly scored by an LLM-as-judge, so you inherit its biases (the eval-metrics and eval-discipline posts cover them) — grade claim-by-claim rather than rating the whole paragraph to keep the noise down.

Split this way, the 'partial refund' incident is unmistakable: retrieval scores stay perfect (it fetched the right cancellation policy), but faithfulness craters (the answer contradicts it). You instantly know it's a generation problem — the prompt, not the retriever — instead of guessing.

Setting the bar: from score to pass/fail

A threshold nobody can defend is theatre, so don't invent one. Start by running your current, live production assistant across the whole set — whatever it scores is your floor, and nothing ships that drops below it. Then set a bar per metric, tied to what the business can actually tolerate. For a customer-facing bot quoting refund and cancellation rules, faithfulness is close to sacred — a wrong answer becomes a chargeback — so you might require faithfulness ≥ 0.95 and context-recall ≥ 0.90; an internal tool that a human double-checks can run much looser. Average several judge runs per row so normal LLM randomness doesn't flip a build. And gate on the pass-rate across the whole set, not any single row — one flaky example shouldn't block a release, but the mean sliding two points should.

Wire it into CI — including the changes people forget

The eval suite runs automatically on every pull request that touches the three things which silently move quality: the prompt, the chunking settings, and the embedding model. The Friday prompt tweak now trips the faithfulness gate and the merge is blocked — the 'partial refund' answer never reaches a traveller. The gate itself is boringly simple:

score = run_evals(candidate, golden_set)   # ~5,000 rows

# every gating metric must hold vs the production baseline
if score.faithfulness < 0.95 or score.context_recall < 0.90:
    fail("Regression: cancellation/refund answers dropped below the bar")

# one flaky row is fine; the average sliding is not
if score.pass_rate < baseline.pass_rate - 0.02:
    fail("Pass-rate fell too far vs production")

# otherwise: green — safe to merge

Prompts are the obvious thing to gate. The two everyone forgets are chunking and the embedding model — and they're the more dangerous ones precisely because they don't touch a line of visible logic. Say an engineer swaps the embedding model for a cheaper one to cut the monthly bill. No prompt changed, the code diff looks trivial, it'll sail through review. But re-embedding rebuilds retrieval for every query in the system — and if that cheaper model is worse at matching 'cancel my trip' to the 'cancellation & refund policy' document, recall quietly drops and the bot starts missing policies it used to find. The golden gate catches it: context-recall falls below 0.90, CI goes red, and the 'harmless' cost optimization is stopped before it degrades every traveller's answers. A regression blocks the merge exactly like a failing unit test (the testing-the-untestable post draws the line between what to unit-test and what to eval). Then you close the loop — every new production surprise becomes a new row, so the bar only ever rises.

One honest caveat: the gate is only as good as the set behind it. If your golden dataset is all easy, politely-phrased questions and none of the messy real ones, you'll get a green CI and unhappy travellers — and false confidence is worse than none. Garbage golden set in, green-but-wrong out. That's why the production-log and complaint-mining sources aren't optional niceties; they're what keeps the number honest.

A golden dataset turns 'the new prompt feels friendlier' into 'it holds faithfulness at 0.96 and never promises a refund the fare rules forbid' — and turns shipping from a Friday-afternoon gamble into a gate.