GuidesAI reliability
Validate a model’s output before using it
Three levels of checks
- 1FormatParsable JSON, fields present.
- 2Types and valuesNumber, date, value from a list.
- 3Business rulesThe product exists, the discount is capped.
With Zod
import { z } from 'zod';
const Order = z.object({
productId: z.string(),
quantity: z.number().int().min(1).max(50),
shipping: z.enum(['standard', 'express']),
});
async function extract(text: string, retries = 2) {
let prompt = text;
for (let i = 0; i <= retries; i++) {
const raw = await callModel(prompt);
const r = Order.safeParse(raw);
if (r.success && (await productExists(r.data.productId))) return r.data;
prompt = `${text}\n\nThe previous answer was invalid: ${r.success ? 'unknown product' : r.error.message}`;
}
return null; // fallback: human review
}
What is often checked
| Output | Check |
|---|---|
| Amount | Positive, capped, right currency |
| URL | Domain on an allowlist |
| Recipient | Existing contact, not an invented address |
| Category | Value from a closed list |
| Published text | Length, banned words, review if sensitive |
The technique: reliable JSON from an LLM. The risks: AI hallucinations and prompt injection.
Frequently asked questions
- Doesn’t a forced tool already guarantee the format?
- It makes the right format very likely, and some modes guarantee the schema is respected. Business rules (a positive price, a future date, an ID that exists) still have to be checked in your code.
- How many retries?
- One or two are usually enough. Beyond that, the problem lies in the instructions or the input data: fall back and log the case.
- What fallback should I plan?
- A safe default value, a human review queue, or a clear message to the user. Never invalid data saved silently.