System design: an autonomous document-evaluation agent

Here's a system-design problem that exercises everything worth knowing about applied AI: build an agent that reads supplier submissions to a request-for-quotation (RFQ), evaluates each against the RFQ's requirements, and recommends the top three vendors. It's a great test because it combines messy unstructured input, structured decision-making, and — crucially — the discipline to keep those two apart. Here's how I'd design it.

The domain, briefly: an RFQ is a procurement questionnaire. An enterprise asks vendors to bid; each vendor submits quotes and answers; the buyer evaluates the submissions and awards a contract. The agent automates that evaluation, at scale and consistently.

1. Multi-format ingestion and normalization

Submissions arrive as PDFs, Excel sheets, and emails — unstructured and inconsistent between vendors. Step one is extraction (a service like AWS Textract for scanned documents and tables — the Document-AI post covers this layer) and normalization: map each supplier's answers onto canonical question IDs, so 'their answer to Q7' is comparable across every vendor regardless of the format it arrived in. This layer is garbage-in-garbage-out; it quietly sets the ceiling on everything above it.

2. A LangGraph state machine

Model the flow as a graph, not a linear script: an extractor node pulls structured data from the raw documents, a validator node schema-checks it and flags gaps, an evaluator node scores each answer against the rubric, and a reasoning node justifies and summarises — with conditional routing between them and a shared state object the orchestrator owns (the production-LangGraph and LangGraph-internals posts cover the mechanics). Stateful, resumable, and legible.

3. Rubric scoring: judgment at the leaves

Each supplier answer is classified against a rubric criterion — Yes, No, or Partially — with a justification drawn from the supplier's own comments. This is exactly where the LLM earns its place: turning free-text answers into a structured, justified verdict per criterion. Low-confidence or 'Partially' verdicts get a confidence signal attached (the confidence-scoring post) so the next stage can treat them with appropriate caution.

4. The decision engine: deterministic ranking

The final ranking is not the LLM's job. The per-criterion labels feed a deterministic decision engine that applies multi-criteria decision analysis: section-level weightings, mandatory-constraint filtering (drop any vendor failing a must-have), and weighted aggregation to rank the top three (the decision-engine post argues this at length). The result is reproducible and auditable — you can show precisely why vendor A beat vendor B, which is the whole point of a procurement decision.

5. Anomaly flagging and human-in-the-loop

Don't silently score bad input. Flag anomalies — a missed mandatory requirement, an empty or garbage submission, a price wildly out of range — and route them for human review instead of scoring them as if valid. And gate the low-confidence and 'Partially' verdicts behind a human approve-or-override step before the ranking is finalised (the human-in-the-loop post). The agent drafts the evaluation; a person signs off on the uncertain cases.

The platform around it

The AI is one layer in an otherwise normal, well-instrumented system: a FastAPI service (client → auth gateway → RFQ CRUD → the LangGraph orchestration → the data layer), multi-tenant MongoDB with S3 for the raw documents, an event-driven ingestion flow (upload → S3 → event → queue → extraction → persist), per-tenant isolation by namespace and tenant_id, and LangSmith tracing with tenant-tagged audit logs plus a golden-dataset eval of the scoring logic (the multi-tenant, data-pipeline, polyglot, and observability posts each cover a piece). The interesting AI sits inside a boringly solid platform — which is exactly how it should be.

The design isn't 'an LLM that picks the winner'. It's a pipeline that extracts messy documents into structured facts, judges each fact where judgment is needed, and ranks deterministically where it isn't — with a human on the uncertain calls.