Local Inference, Local Hardware: Building My Own Assistant
Most assistants I could just wire up to an API. I wanted to own the whole stack instead: the model, the memory, the hardware it runs on.
What it is
A local AI assistant, self-hosted end to end, with no dependency on an always-on external API to function day to day. The design goal was simple to state and hard to hit: something that adapts to me over time, remembers context across projects, and doesn’t rack up per-token costs just to answer a basic question.
The architecture
The core idea is a division of labor rather than one model doing everything:
- Local model (Ollama) handles everyday interaction and memory or context retrieval, the part that has to run cheap and fast, on hardware I control.
- Retrieval, not the model’s memory, is the source of truth. Facts get pulled from a vector store at query time rather than trusted to whatever the model remembers from training.
- LoRA fine-tuning, trained incrementally on a dedicated GPU box I built and run myself, handles tone and style, not fact storage. That split turned out to matter more than I expected.
The problem that actually taught me something
Early on, I let the model judge its own answers as worth remembering and write them straight into permanent memory. It seemed efficient. It also meant a hallucinated fact, invented outright with no basis in anything I’d said, got written to memory as if it were true. It happened through plain text input, not a transcription error, and it reproduced across two different base models, so it wasn’t a fluke of one model’s weights.
A generative judge checking its own output isn’t a reliable gate, no matter how carefully you narrow what it’s allowed to look at. The real fix was structural, not a smarter prompt: automatic self-directed memory writes are disabled entirely. Nothing becomes a permanent fact unless it goes through a separate, deliberate review pipeline I run and check myself.
How I talk to it
Beyond the local interface, it reaches me two other ways. A dedicated email address polls for messages from me specifically and replies through the same retrieval pipeline as everything else, so I can ping it from my phone without opening anything special. It also sends a proactive check-in most mornings if there hasn’t been any interaction in the last 20 hours or so.
A browser-based GUI handles the heavier sessions: voice input that works across browsers rather than just the two with built-in speech recognition, a toolbar for toggling web search and citations, and a password gate in front of it since it’s reachable outside my own network.
What runs while I sleep
A nightly job distills the day’s interactions, trains a candidate model against them, and evaluates that candidate against whatever is currently live. Nothing gets promoted automatically. I get a report in the morning and decide from there whether the new version is actually better or just different from the one already running.
The network it lives on
The hardware sits in a rack alongside the rest of the homelab, behind a FortiGate firewall and a managed switch. Storage is a dedicated Dell PowerVault NX3200 running TrueNAS SCALE, a RAIDZ1 pool across four SAS drives, set up specifically as the backend for datasets, the vector database, and model checkpoints rather than sharing space with anything else in the lab. The network itself is being segmented into zones, servers, IoT and embedded devices, and management, each behind its own VLAN with firewall rules controlling what can talk to what. Nothing on the IoT side reaches the server zone unless I’ve explicitly allowed it.
What’s next: eyes
Vision is still a hardware problem more than a software one. The plan is two cameras: a static one at the door and a pan-tilt one at the window, both running on the same Raspberry Pi that already handles DNS filtering and network monitoring for the rest of the lab. Once that’s wired in, arrival detection and basic scene description are the first real vision features on the list.
Where it stands now
Local inference and retrieval are working. Training started on short paid bursts of rented cloud GPU time, but that step is done now: everything runs on a dedicated on-prem GPU box I built and keep on around the clock, no more cloud rental. The pieces above, email, nightly training, the network, are all live in some form; vision is next. The current adapter is still undertrained, so more training data at real volume is the immediate priority before anything else.