Measuring a GenAI solution: technical metrics and business metrics

Ask how a GenAI feature is doing and you'll get one of two unsatisfying answers: a pile of technical scores that don't obviously mean anything to the business, or a business number nobody can trace back to a model change. The fix is to measure both layers deliberately, and wire them together.

Model-centric (technical) metrics

These measure the quality of the model's output directly:

• Correctness / accuracy — is the answer right, against a labelled set?
• Faithfulness / groundedness — for RAG, does the answer stick to the retrieved sources rather than inventing?
• Relevance — does it actually address what was asked?
• Hallucination rate — how often it states something false or unsupported.
• Latency and cost — response time and tokens per request, which are product constraints, not footnotes.
• LLM-as-judge scores — a strong model grading outputs against a rubric, increasingly the practical way to score open-ended quality (older overlap metrics like BLEU/ROUGE have their uses but miss meaning).

Business-centric metrics

These measure whether the feature moves the outcome you actually care about:

• Task success / resolution rate — did the user get their job done?
• Deflection rate — for support, how many issues resolved without a human.
• Conversion and revenue impact — the money line, where it applies.
• Time saved — hours removed from a workflow.
• CSAT, adoption, and retention — do people trust it enough to keep using it?

Connect the two

Technical metrics are only proxies — they matter exactly insofar as they predict a business outcome. So pick a north-star business metric first, then build the technical evals that correlate with it, and gate changes on those (this is the observability-and-evals discipline in practice). A green dashboard that doesn't move the business metric is decoration.

Technical metrics tell you the model is good. Business metrics tell you it matters. Ship on the second, debug with the first.