GuidesAI development
Observing an AI application in production: traces, costs and errors
What a trace contains
| Data | Example |
|---|---|
| Request ID | To link every step |
| Model and parameters | Model name, temperature, max_tokens |
| Prompt and answer | Full text, personal data masked |
| Tokens | Input, output, cache |
| Tools called | Name, arguments, duration, result |
| User feedback | Thumbs up or down, comment |
The token count is in the response
const res = await client.messages.create({ model: process.env.CLAUDE_MODEL!, max_tokens: 1024, messages });
log.record({
request: requestId,
model: res.model,
input: res.usage.input_tokens,
output: res.usage.output_tokens,
stop: res.stop_reason,
duration_ms: Date.now() - start,
});
Useful dashboards
- CostPer day, per feature, per customer.
- LatencyMedian and slowest cases.
- ErrorsRate limits, timeouts, invalid output.
- QualityUser feedback, test set scores.
From trace to fix
- 1SpotAn answer flagged by a user.
- 2ReplayThe trace shows the exact context.
- 3Add to the test setThe case will not come back silently.
Testing: test an AI chatbot. Cost: cap your AI API bill.
Frequently asked questions
- Can we store prompts and answers?
- Yes, if your privacy policy covers it, with a retention period and limited access. Mask personal data when possible.
- Which tools should I use?
- OpenTelemetry for an open standard, specialized tools such as Langfuse (open source), or simply structured logs in your database at first.
- Which alerts should I set?
- Daily cost above a threshold, rising error rate, abnormal latency, and a drop in positive user feedback.