On-Premise AI Agents: How to Run Without Sending Data to OpenAI

There are three reasons to run AI agents on-prem in 2026:

1. Data residency or confidentiality requirements that forbid sending data to hosted providers (healthcare PHI, financial MNPI, legal privilege, EU data localization).
2. Cost at sustained scale — once you’re past a few million tokens per day, a self-hosted GPU is cheaper than a hosted API.
3. Latency for tight loops where round-trip to a hosted vendor is the bottleneck.

This is a practical guide to all of it. The stack is the one ONTO defaults to, but the architecture applies to any agent SDK that supports on-prem deployment.

The stack

A complete on-prem agent stack has five parts:

1. Model serving. Ollama or vLLM running an open-weight model.
2. Embedding model. fastembed or sentence-transformers, locally.
3. Vector store. pgvector inside Postgres, or qdrant locally.
4. Typed memory store. Postgres for production, Sled for single-machine dev.
5. Agent runtime. ONTO (or your SDK of choice) running on the same network.

Everything inside your VPC. No external API call required for any agent operation.

Model serving

For a single-machine setup, Ollama is the easiest:

# Install Ollama
curl -fsSL https://ollama.com/install.sh | sh

# Pull a model
ollama pull gpt-oss:20b

# Verify it serves
curl http://localhost:11434/api/generate -d '{
  "model": "gpt-oss:20b",
  "prompt": "Hello"
}'

For multi-machine or higher-throughput deployments, vLLM gives better serving performance. The OpenAI-compatible API means it’s a drop-in for any SDK that talks to OpenAI:

# vLLM with OpenAI-compatible server
python -m vllm.entrypoints.openai.api_server \
  --model gpt-oss:20b \
  --tensor-parallel-size 2 \
  --port 8000

Configure ONTO to point at either:

# Ollama
export OLLAMA_BASE_URL=http://localhost:11434

# vLLM (OpenAI-compatible)
export OPENAI_BASE_URL=http://localhost:8000/v1
export OPENAI_API_KEY=anything  # vLLM doesn't check; required by SDK

The same agent code runs against either.

Choosing a model

For 2026, three open-weight model families are practical for production agents:

• gpt-oss family (OpenAI’s open-weight models). 20B and 120B variants. Strong on tool calling and structured output. Default for ONTO.
• Llama 3 family (Meta). 8B, 70B, 405B. Stable; broad community support.
• Qwen 2.5 family (Alibaba). 7B, 32B, 72B. Strong on multilingual.

For the bulk of agent workload (extraction, tool calling, plan generation, summarization), the 20B-class models are sufficient. For complex reasoning that the 20B class struggles with, route those specific calls to a 70B or larger.

Embeddings

For vector search (LlamaIndex-style RAG, semantic search over notes), use a local embedding model:

from fastembed import TextEmbedding

emb = TextEmbedding("BAAI/bge-small-en-v1.5")
vectors = list(emb.embed(["text 1", "text 2"]))

fastembed runs in CPU at acceptable speeds for moderate volumes. For high volume, sentence-transformers on a GPU.

ONTO’s vector layer accepts any embedding function; configure yours:

from onto import Onto

onto = Onto(
    embedding_provider=fastembed_provider("BAAI/bge-small-en-v1.5"),
    ...
)

Typed memory store

For production, Postgres + pgvector:

# Set up Postgres
docker run -d -p 5432:5432 \
  -e POSTGRES_PASSWORD=secret \
  pgvector/pgvector:pg16

# Configure ONTO
export ONTO_STORAGE=postgres
export DATABASE_URL=postgres://postgres:secret@localhost/onto

ONTO’s schema migration creates the tables; the rest is operational. Index (tenant_id, subject, predicate) for fast lookups. Index the vector column for ANN.

For dev or single-machine deployments, Sled (embedded key-value):

export ONTO_STORAGE=sled
export ONTO_STORAGE_PATH=./onto-data

Sled needs no separate process. Backups are file copies. Good for dev and small-scale personal AI.

Kubernetes deployment

For production, a Helm chart deploys ONTO + Postgres + Ollama + your application:

# values.yaml (sketch)
onto:
  replicas: 3
  storage: postgres
  databaseUrl: postgres://onto:...@postgres:5432/onto

ollama:
  replicas: 2
  modelName: gpt-oss:20b
  gpuLimit: 1

postgres:
  storageSize: 200Gi
  resources:
    requests:
      memory: 4Gi
      cpu: 2

The framework repo includes a chart at deploy/helm/onto/. For most production deployments the chart is a starting point you customize.

The latency picture

Round-trip times for a typical agent turn, on-prem vs hosted:

• Hosted Claude/GPT: 800-2500ms for a typical agent turn
• On-prem Ollama 20B on A100: 400-1200ms for the same turn
• On-prem vLLM 20B on A100 with batching: 200-700ms

The latency win is real for moderate-complexity prompts. For very large prompts or complex reasoning, the larger hosted models can be faster despite the round-trip because their inference is faster.

The right pattern for latency-sensitive agents: keep the bulk of work on the 20B local model; route only the hardest reasoning to a larger model (local or hosted).

The cost picture

For an A100 (40 GB) on AWS at $2.10/hour (p4d.24xlarge breakdown), running 24/7, you pay ~$1,500/month. That’s amortized across whatever throughput you serve.

vLLM on an A100 serves roughly 20-40 requests/second for a 20B model with reasonable prompt sizes. At an average of 1000 tokens per turn, that’s ~24M tokens/hour at peak, ~600M tokens/day at peak.

A hosted API at $3/MTok (typical for a mid-size model) would bill ~$1,800/day for the same throughput. The break-even is roughly 10-20% utilization of the GPU.

If your traffic is bursty, hosted is more cost-effective. If your traffic is steady at moderate volume, on-prem wins. This is the calculation every team should run.

The operational surface

What you take on with on-prem:

• GPU monitoring (utilization, temperature, OOM events)
• Model serving health checks
• Postgres operations (backups, replication, vacuum)
• Network security (don’t expose the model server to the internet)
• Model upgrades (when a new gpt-oss version ships, you redeploy)
• Cost attribution per workload (a per-tenant token meter)

A small SRE team can run this. A single application engineer with no SRE support cannot.

The hybrid pattern

Most production deployments end up hybrid:

• Default: on-prem open-weight for the bulk of traffic.
• Burst: route to a hosted endpoint when GPU capacity is exhausted.
• Specialized: route hardest reasoning calls to Claude/GPT for higher quality.

ONTO supports this with provider-aware routing. The agent code is unchanged; the provider config decides where calls go.

Where on-prem doesn’t help

To be honest:

• One-shot research projects. The setup cost isn’t worth it for a 3-week project.
• Frontier reasoning workloads. The 20B class is below frontier; if your agent needs maximal reasoning quality, hosted frontier models are still ahead.
• Voice/realtime. OpenAI’s realtime API is fundamentally hosted; no equivalent on-prem story today.

For those, hosted is the right answer.

A summary

On-prem AI agents in 2026 are a real option, not a workaround. The stack is mature: Ollama or vLLM for serving, fastembed for embeddings, Postgres + pgvector for memory, Helm for deployment, ONTO (or any well-designed agent SDK) for the runtime.

The decision factors are compliance, data control, sustained-traffic cost, and your team’s operational capacity. If three of the four point on-prem, ship it.

---

ONTO defaults to local-friendly deployment — the docs page walks through the Ollama setup. The features page explains the provider model.

Frequently asked questions

What hardware do I need to run agents on-prem?

For a 20B-parameter model serving moderate traffic, a single A100 (40 GB) or 2x RTX 4090 / A6000 is enough. For larger models or higher throughput, an H100 or a multi-GPU setup. For non-production / dev, a M-series Mac with 32 GB unified memory runs gpt-oss:20b locally.

How does quality compare to GPT/Claude?

For tool-calling and structured extraction, modern open-weight models (gpt-oss:20b, Llama 3.1, Qwen 2.5) are competitive. For frontier reasoning, there's still a gap. Default to open-weight; route specific hard workloads to a hosted frontier model only where you measure a lift.

Isn't this more expensive than a hosted API?

For low volume, yes — you pay for idle GPUs. For sustained traffic, the GPU amortizes well. Most organizations hit break-even somewhere in the millions of tokens per day.