GuidesAI technical choices

A local AI model for sensitive data

By the Banzai Labs teamUpdated on 2 min read

Your premisesExternal APISensitive data: the model comes to it, it never leaves
The model runs on a company server, next to the data, with no call to an external API.

Local or API

Two ways to use a model
Local model
  • No data leaves
  • Hardware and maintenance to plan
  • Less capable on hard tasks
API of a large model
  • Better quality, nothing to install
  • Data sent to the provider
  • Pay per use

The tools

Tool For
Ollama Trying a model with one command
llama.cpp Running a quantized model on few resources
vLLM Serving a model to many users, on GPU
ollama run mistral
# then a local API on http://localhost:11434

Setting it up

  1. 1Define the taskClassify, summarize, extract.
  2. 2TestTwo or three models on your cases.
  3. 3SizeMemory, number of users.
  4. 4SecureClosed network access, logs, updates.

The hybrid approach

The local model handles the sensitive data (extraction, pseudonymization), then a large model receives only what is allowed to leave.

The choice: open source model or API. The technique: quantization. The alternative: pseudonymize before sending.

Frequently asked questions

What hardware is needed?
It all depends on the size of the model. A small quantized model runs on a workstation with a good graphics card; a larger model needs one or more server GPUs. GPU memory is the first constraint.
Is the quality comparable?
Not on complex tasks. On targeted tasks (classification, extraction, short summaries), a good, well-guided open source model often gives sufficient results. Test on your cases.
Which models should I choose?
Open-weight families such as Llama, Mistral, Qwen or Gemma come in several sizes. Check the license before commercial use.

Read next

All guides