GuidesAI technical choices
A local AI model for sensitive data
Local or API
Local model
API of a large model
The tools
| Tool | For |
|---|---|
| Ollama | Trying a model with one command |
| llama.cpp | Running a quantized model on few resources |
| vLLM | Serving a model to many users, on GPU |
ollama run mistral
# then a local API on http://localhost:11434
Setting it up
- 1Define the taskClassify, summarize, extract.
- 2TestTwo or three models on your cases.
- 3SizeMemory, number of users.
- 4SecureClosed network access, logs, updates.
The hybrid approach
The local model handles the sensitive data (extraction, pseudonymization), then a large model receives only what is allowed to leave.
The choice: open source model or API. The technique: quantization. The alternative: pseudonymize before sending.
Frequently asked questions
- What hardware is needed?
- It all depends on the size of the model. A small quantized model runs on a workstation with a good graphics card; a larger model needs one or more server GPUs. GPU memory is the first constraint.
- Is the quality comparable?
- Not on complex tasks. On targeted tasks (classification, extraction, short summaries), a good, well-guided open source model often gives sufficient results. Test on your cases.
- Which models should I choose?
- Open-weight families such as Llama, Mistral, Qwen or Gemma come in several sizes. Check the license before commercial use.