GuidesAI development
Version your prompts like code
Storing prompts
prompts/
support/
system.md # the prompt
meta.json # model, parameters, version
invoice-extraction/
system.md
meta.json
tests/
support.json # questions and expected answers
The life of a change
- 1EditOn a branch, with the reason.
- 2TestThe full set, automatically.
- 3CompareScore and cases that change.
- 4ReviewLike a code change.
- 5DeployRollback in one command.
A useful history
| Version | Change | Score |
|---|---|---|
| v1 | First version | 72% |
| v2 | Examples added | 81% |
| v3 | Shorter instructions | 78%, regression |
| v4 | v2 + rule on refunds | 88% |
Every saved answer keeps the prompt version that produced it: when something goes wrong, you know where it came from.
Writing: write a system prompt. Testing: LLM as a judge. The model: switching AI models.
Frequently asked questions
- Do I need a dedicated tool?
- Not at first. Text files in the repository, a test set and a comparison script are enough. A dedicated tool helps when non-technical people edit the prompts.
- What should be recorded with each version?
- The model and its parameters, the date, the reason for the change and the score on the test set.
- How do I compare two versions?
- Run them on the same set of questions, with the same rubric or judge, and look at the cases that change, not just the average.