Building a custom AI application: from idea to production

An application before an AI

Projects that fail often started with the model. Those that succeed start with the product: who uses it, to do what, with which data.

Votons is a good example. It is a consumer application: a quiz, candidate profiles, a comparison tool, a website, an API and mobile apps. AI has two precise roles in it: an assistant that answers on the manifestos from the exact text, and agents that prepare the quiz cards, which are then reviewed.

The typical architecture

Layer	Role
Interface	Web or mobile, what the user sees
API	Accounts, permissions, business rules
Data	Database, documents, search index
Orchestration	Chains search, model calls and tools
Model	Called through an API, replaceable
Observation	Log, costs, answer quality

The model is one layer out of six. The other five are ordinary development, and that is where most of the work goes. See Building a web app.

Choosing the model

• Test on your casesNot on general rankings.
• A light model for simple workA powerful one for reasoning.
• Data locationWhen it matters.
• The ability to switchModels are replaced every few months. The application must not depend on a single one.

Our projects do not all use the same one: Votons relies on Mistral, Alfred and Comptoir on Claude.

The data

The model knows nothing about your business. You bring it what it needs at the right moment:

• Document searchTo answer from texts. See RAG.
• ToolsTo read and write in your systems. See MCP server.
• IngestionTo keep the data up to date. Votons collects about twenty public sources automatically.

Testing an AI application

A model does not answer exactly the same way twice. So you do not test it like ordinary software.

1. 1A set of real casesWith the expected answer.
2. 2CriteriaAccuracy, faithfulness to the source, tone, refusing when it should.
3. 3Automatic evaluationReplayed at every change of instruction or model.
4. 4Human reviewOn a sample.

Without this test set, every improvement risks breaking something else unnoticed.

Rules in the code

What must always be true is not left to an instruction. Votons’ neutrality is enforced by code: anonymised positions, left-right parity. Access rights, caps and approvals follow the same rule.

Going to production

• A spending cap and alerts.
• Timeouts and a planned behaviour when the model does not answer.
• A log of requests and answers, respecting personal data.
• Simple user feedback: was this answer useful?
• A gradual rollout: internal team, small group, everyone.

After go-live

An AI application degrades if nobody looks at it: content changes, models are replaced, usage evolves. Plan a monthly review of poorly rated answers, updates to the test set and cost tracking.

For the budget, see How much an AI project costs. To start smaller, see Building an MVP with AI.