Building a Personal AI Assistant That Actually Remembers You

The hardest problem in personal AI isn’t quality. It’s continuity. After three months of using an assistant, you want it to know you. Not in the “model has been fine-tuned on me” sense — in the “it remembers what we worked out in March” sense.

Standard chat agents can’t deliver this. Conversation buffer forgets when the window rolls. Vector retrieval surfaces “similar past content” but not “the thing you decided.” The user re-explains the same context every month.

This post sketches the architecture for a personal assistant that doesn’t.

What the assistant has to know

A useful personal assistant knows things in four categories:

1. Identity facts. Name, timezone, employer, location, primary calendar.
2. Preferences. “Mornings, not evenings.” “Reply to emails the same day.” “Don’t book meetings on Fridays.”
3. Project context. Current work-in-progress, key people involved, deadlines.
4. Relationship facts. Who is who, how you know them, last interaction.

These aren’t chat history. They’re typed facts. And they have different lifecycles — identity facts decay slowly, project context turns over weekly, relationship facts age but don’t disappear.

ONTO’s UPO + SMO split is built for this:

# Identity — durable, profile-level.
profile = onto.memory.get_profile("you")
# profile.name, profile.timezone, profile.employer

# Preferences — durable, profile-level.
# profile.prefers, profile.meeting_window

# Project — domain-scoped to the current work.
projects = onto.memory.query_domain("you", domain="work")
# returns active projects with their context

# Relationships — domain-scoped to a "contacts" graph.
contacts = onto.memory.query_domain("you", domain="contacts")

Each query is O(1) lookup against typed storage. The model’s prompt loads what’s relevant, no more.

The async write pattern

For personal AI, latency is everything. If the assistant takes three seconds to reply because it’s writing memory synchronously, it feels sluggish.

ONTO’s extract_async=True flag spawns extraction writes into a background task:

result = onto.run_sync(
    user_message,
    user_id="you",
    consent_scopes=["memory:write", "calendar:read"],
    extract=True,
    extract_async=True,
)

The user gets the reply immediately. The memory writes happen behind the scenes. The next turn loads them when it starts — and because storage is fast, they’re ready in time.

Combined with background_memory=True, session summarization and conflict detection also move to a background task on a cheaper model:

onto = Onto(
    model=Models.GPT_OSS_20B,            # foreground
    background_memory=True,              # background uses a cheaper model by default
    background_model=Models.LLAMA_3_8B,  # explicit override
)

The foreground bill stays bounded; the background bill is small per task because the model is small.

The four-level extraction

When the user says something, where does it go? The four-level routing does the work:

User: "Move my Wednesday standup to 10am from now on."

→ Extracted assertions:
  session: (you, current_request, reschedule_standup, conf=0.95)
  profile: (you, prefers_standup_time, "10:00", conf=0.85)
  domain[work]: (standup_meeting, time, "10:00", conf=0.9, supersedes=prior)
  tool: (calendar, last_reschedule_target, "standup", conf=0.6)

The LLM labels each candidate assertion with a level. The runtime writes them to the appropriate stores. Next month, when you ask “what time is my standup,” the answer comes from the work-domain assertion, not a buffer search.

The conflict resolution

People change their minds. The user might say “I prefer mornings” in March and “I want to switch to afternoons” in July. Both are honest at the time. The agent has to handle the contradiction.

ONTO’s conflict detection: when two assertions point at the same (subject, predicate) with different objects, the system creates a Plan Mode task:

Task: "User previously preferred mornings (March 2026, confidence 0.85).
 Now states preference for afternoons (July 2026, confidence 0.92).
 Supersede?"

The user can resolve it (“yes, supersede”), the agent can resolve it (auto-supersede high-confidence over low-confidence), or it can sit until someone decides. The point is the contradiction surfaces — it doesn’t get silently averaged.

The on-device path

For users who want maximum privacy, the whole stack runs locally:

• A local Ollama serving an open-weight model (gpt-oss:20b runs on a M-series Mac).
• ONTO’s Sled storage for typed assertions (embedded; no separate database).
• Local fastembed for any vector search.

The setup:

# Run ollama locally
ollama serve

# Pull a model
ollama pull gpt-oss:20b

# Configure ONTO
export OLLAMA_BASE_URL=http://localhost:11434
export ONTO_STORAGE=sled
export ONTO_STORAGE_PATH=./my-assistant-memory

pip install onto-core

After this, the assistant runs on the user’s machine. No data leaves. The model is fully local.

The multi-device sync

If users want their assistant on multiple devices, swap Sled for a Postgres backend. The Postgres lives on a server the user controls (or a managed service with appropriate controls). Each device runs ONTO pointed at the same Postgres.

export ONTO_STORAGE=postgres
export DATABASE_URL=postgres://user:pass@user-server/onto

The user owns the database. Their facts live in their Postgres, not in a vendor’s. They can back it up, export it, query it, delete it.

What the subagent pattern enables

A daily-digest assistant has multiple jobs — calendar, email, notes. Each has its own tools. ONTO’s subagent pattern lets you decompose:

calendar = onto.subagent("calendar", tools=["mcp:gcal:list", "mcp:gcal:create"])
email    = onto.subagent("email",    tools=["mcp:gmail:search", "mcp:gmail:reply"])
notes    = onto.subagent("notes",    tools=["mcp:notion:query"])

digest = onto.run_sync(
    "Give me my morning digest.",
    user_id="you",
    consent_scopes=["memory:write", "calendar:read", "email:read", "notes:read"],
)

Each subagent has constrained tools and scopes. The parent agent dispatches and synthesizes. The cost rolls up to the parent. The depth budget prevents infinite delegation.

What still requires the user

Personal AI does well on accumulation; it does poorly on judgment. The assistant should not unilaterally:

• Reply to emails on the user’s behalf without review.
• Accept meeting invites that involve commitments.
• Make purchases.
• Share information with anyone.

Plan Mode handles this. The assistant proposes “draft a reply to Alex”; the user reviews; the user sends. Or the user configures an auto-approval policy (“auto-approve drafts to close colleagues, request review for everyone else”).

A summary

A personal assistant that actually remembers you is built on three primitives:

1. Typed memory (UPO + SMO) instead of conversation buffers or vector-only stores.
2. Async writes so memory persistence doesn’t slow turns.
3. Background summarization on a cheaper model so consolidation doesn’t dominate cost.

ONTO ships all three. The user’s facts stay on the user’s side. The assistant gets smarter about you over time, not because the model trained on you, but because the system stored what was learned.

That’s the bar for a useful personal AI. Everything else — frontier-model-of-the-month, voice support, novel UI — sits on top of it.

---

The personal AI industry page walks through the runnable daily-digest example. The thesis on agent memory explains why typed memory is the right primitive.

Frequently asked questions

Do I need to be on cloud to use ONTO for personal AI?

No. The local deployment uses Sled (embedded key-value store) and a local Ollama for the model. The whole stack runs on a single machine with no network. The user's facts never leave their device.

How does the assistant 'learn' you over time?

Every interaction extracts typed facts at the appropriate level (session, profile, domain, tool). Session facts evaporate; profile facts accumulate into your UPO. Background summarization on a cheaper model consolidates patterns into durable assertions.

What about privacy if I'm using a hosted model?

Use a hosted model that doesn't train on your data (any modern enterprise-tier offering). Or use a self-hosted Ollama for full privacy. ONTO's memory layer stays under your control regardless — the model is just an LLM the runtime calls.