Testing the untestable: unit tests, mocked LLMs, and eval suites

The mistake teams make testing AI apps is treating the whole system as one thing — either giving up on tests because 'it's non-deterministic', or trying to assert exact model outputs and drowning in flaky failures. The fix is to split the system: deterministic machinery gets unit tests, probabilistic intelligence gets evals.

Deterministic machinery → unit tests

Most of an LLM app is ordinary, deterministic code: parsing, routing, schema validation, tool execution, state transitions, retries and fallbacks. All of it is unit-testable with pytest — and you mock the LLM so the tests are fast, stable, and free. This is where most of your bugs actually live, so test it hard.

def test_router_picks_refund(monkeypatch):
    # mock the LLM so the test is deterministic
    monkeypatch.setattr(llm, 'classify', lambda t: 'refund')

    state = {'intent': llm.classify('I want my money back')}
    assert route(state) == 'refund_node'

# note: the model's answer *quality* is NOT asserted here — that's an eval

Probabilistic intelligence → evals

The model's actual outputs — is the answer correct, faithful, well-judged — don't belong in an assertEqual. They belong in an eval suite scored against a golden dataset, where you measure a pass rate over many cases, not a single truth (see the eval-discipline post). Asserting a creative output exactly is how you get a test that fails every time the model rephrases.

The dividing line

The rule is simple: if the output is deterministic, unit-test it; if it depends on the model's judgement, eval it. Draw that line cleanly and both halves become tractable — fast, reliable unit tests around a well-measured probabilistic core.

Don't unit-test the model's creativity, and don't eval your JSON parser. Mock the brain to test the plumbing; measure the brain where the plumbing ends.