The 30-point production-readiness checklist.

Chunking was chosen by measurement, not by default

You tested at least two chunk sizes and overlaps against a labelled query set, and you can say how much the winner won by.

Retrieval is measured separately from generation

Recall@k and precision exist for the retriever alone, so a bad answer can be traced to the wrong document rather than blamed on the model.

The index rebuild is reproducible and versioned

Rebuilding from source yields the same index, the embedding model is pinned, and you can roll back without re-ingesting everything.

Metadata filtering happens before the vector search, not after

Tenant, permission and date filters are applied in the query itself, so a user can never retrieve a chunk they are not entitled to see.

There is a defined answer for "it is not in the corpus"

Low-relevance retrievals are detected and the system says it does not know, rather than passing weak context along and letting the model improvise.

A labelled evaluation set exists, and it predates the fix you are shipping

Held out, written from real user queries, and not generated by the same model you are testing.

Every prompt change runs the eval suite before merge

Automated in CI with a pass threshold, so a one-word prompt edit cannot silently regress the whole system.

At least one metric is something a user would recognise

Task success or answer correctness — not only cosine similarity, BLEU, or another number that moves without the product improving.

LLM-as-judge is calibrated against human labels

You have measured how often the judge agrees with human raters on a sample, so you know the error bars on your own scoreboard.

Regression failures are diffable

When the suite fails you can see the exact input, the previous output and the new output side by side, without re-running anything by hand.

You know the cost per request, not just the monthly bill

Broken down across model calls, retrieval and re-ranking, so you know which component to attack first.

Token growth has a ceiling

Context assembly runs to a hard budget, so a long conversation or an oversized document cannot quietly triple the unit cost.

Caching exists wherever the traffic is repetitive

Prompt or semantic caching is in place and you track the hit rate, because an uninstrumented cache is a guess.

p95 latency is a tracked objective, not an anecdote

Measured end to end, including retrieval and time-to-first-token, on the same percentile you would defend in a customer meeting.

There is a cheaper fallback path and you know its quality cost

The smaller model has been run against the eval suite, so degrading under load is a decision rather than a surprise.

Every generation is traced end to end

One request id links retrieval, prompt, model, token counts, latency and output into a single record you can pull up.

Prompts and outputs are logged under a retention and redaction policy

You can debug a complaint from last Tuesday without holding personal data you were never supposed to keep.

You can answer "what changed?" in under five minutes

Model version, prompt version and index version are stamped on every trace, so a quality cliff has a suspect list.

User feedback joins back to the trace

A thumbs-down resolves to the exact request that caused it, rather than to a date and a vague description.

Alerts fire on quality, not only uptime

Refusal rate, empty-retrieval rate and schema-validation failures are alarmed — a system can be fully available and completely wrong.

Prompt injection is tested, not assumed handled

You keep an adversarial input set, including injections that arrive inside retrieved documents rather than from the user.

Untrusted content cannot reach a privileged tool

Any tool call with a side effect is gated by validation or explicit confirmation, so a retrieved document cannot trigger an action.

Output is validated against a schema before it reaches another system

Structured output is parsed and rejected on failure, never passed downstream on the assumption that it parsed correctly.

Rate limits and spend caps exist per tenant

One enthusiastic customer, or one retry loop, cannot exhaust the budget for everyone else.

There is a documented degradation path

When the provider is slow or unavailable the product does something defined and tested, rather than hanging until the request times out.

Someone other than the author can ship a prompt change

Prompts are versioned in the repository, reviewed, and released like code rather than edited live in a console.

The runbook covers the three most likely failures

Provider outage, stale index and quality regression, each with a named owner and a first step.

Provider keys rotate without a redeploy

Secrets live in a secret store, scoped per environment, so rotating one is a routine operation rather than a release.

There is a staging environment with realistic data

Evaluation and manual QA run somewhere that is not production, against data shaped like the real thing.

The team can explain the system without you

Architecture, the decisions behind it and the known trade-offs are written down, including what you deliberately chose not to build.