GuidesAI development

Version your prompts like code

By the Banzai Labs teamUpdated on 2 min read

v172 %v281 %v378 %v488 %regressionEach prompt version has its score on the same test set
Each prompt version is scored on the same test set before going live.

Storing prompts

prompts/
  support/
    system.md         # the prompt
    meta.json         # model, parameters, version
  invoice-extraction/
    system.md
    meta.json
tests/
  support.json        # questions and expected answers

The life of a change

  1. 1EditOn a branch, with the reason.
  2. 2TestThe full set, automatically.
  3. 3CompareScore and cases that change.
  4. 4ReviewLike a code change.
  5. 5DeployRollback in one command.

A useful history

Version Change Score
v1 First version 72%
v2 Examples added 81%
v3 Shorter instructions 78%, regression
v4 v2 + rule on refunds 88%

Every saved answer keeps the prompt version that produced it: when something goes wrong, you know where it came from.

Writing: write a system prompt. Testing: LLM as a judge. The model: switching AI models.

Frequently asked questions

Do I need a dedicated tool?
Not at first. Text files in the repository, a test set and a comparison script are enough. A dedicated tool helps when non-technical people edit the prompts.
What should be recorded with each version?
The model and its parameters, the date, the reason for the change and the score on the test set.
How do I compare two versions?
Run them on the same set of questions, with the same rubric or judge, and look at the cases that change, not just the average.

Read next

All guides