GuidesUnderstanding AI

Batch processing: thousands of AI calls, for less

By the Banzai Labs teamUpdated on 3 min read

10,000 requestsBatchprocessed laterResultsNo need for an instant answer: send in bulk, pay less
Thousands of requests sent as one batch are processed later, at lower cost.

Real time or batch

Two ways to call the model
Real time
  • One request, one immediate answer
  • For a user who is waiting
  • Standard price
Batch
  • Thousands of requests at once
  • Results later
  • Reduced price

A batch with the Claude API

const batch = await client.messages.batches.create({
  requests: products.map((p) => ({
    custom_id: p.id,
    params: { model: process.env.CLAUDE_MODEL!, max_tokens: 512, messages: [{ role: 'user', content: instruction(p) }] },
  })),
});

// later
const status = await client.messages.batches.retrieve(batch.id);
if (status.processing_status === 'ended') {
  for await (const r of await client.messages.batches.results(batch.id)) {
    if (r.result.type === 'succeeded') save(r.custom_id, r.result.message);
  }
}

Good use cases

Case Why batch
Rewrite 5,000 product pages Nobody is waiting for the answer
Classify ticket archives Large volume, one time
Extract data from the month’s invoices Overnight processing
Replay a test set Hundreds of questions

The custom_id links each result to its request: results do not necessarily come back in order.

Cost: cap your AI API bill and prompt caching. The basics: call the Claude API from Node.js.

Frequently asked questions

How long do I have to wait?
At Anthropic, most batches finish in under an hour, with a maximum of 24 hours. Plan your processing accordingly.
How much cheaper is it?
Providers apply a significant discount to batches compared with standard calls. Check the current rate on their pricing page.
Can batch be combined with prompt caching?
Yes. When every request shares the same long beginning (instructions, examples), the cache lowers the cost further.

Read next

All guides