GuidesAI development

Observing an AI application in production: traces, costs and errors

By the Banzai Labs teamUpdated on 2 min read

Trace of one requestRequestRetrievalClaudeToolClaudeTokensLatencyCostErrorsLog everything: prompt, answer, tools, duration, cost, errors
The trace of a request shows each step, its duration, its tokens and its errors.

What a trace contains

Data Example
Request ID To link every step
Model and parameters Model name, temperature, max_tokens
Prompt and answer Full text, personal data masked
Tokens Input, output, cache
Tools called Name, arguments, duration, result
User feedback Thumbs up or down, comment

The token count is in the response

const res = await client.messages.create({ model: process.env.CLAUDE_MODEL!, max_tokens: 1024, messages });
log.record({
  request: requestId,
  model: res.model,
  input: res.usage.input_tokens,
  output: res.usage.output_tokens,
  stop: res.stop_reason,
  duration_ms: Date.now() - start,
});

Useful dashboards

  • CostPer day, per feature, per customer.
  • LatencyMedian and slowest cases.
  • ErrorsRate limits, timeouts, invalid output.
  • QualityUser feedback, test set scores.

From trace to fix

  1. 1SpotAn answer flagged by a user.
  2. 2ReplayThe trace shows the exact context.
  3. 3Add to the test setThe case will not come back silently.

Testing: test an AI chatbot. Cost: cap your AI API bill.

Frequently asked questions

Can we store prompts and answers?
Yes, if your privacy policy covers it, with a retention period and limited access. Mask personal data when possible.
Which tools should I use?
OpenTelemetry for an open standard, specialized tools such as Langfuse (open source), or simply structured logs in your database at first.
Which alerts should I set?
Daily cost above a threshold, rising error rate, abnormal latency, and a drop in positive user feedback.

Read next

All guides