Why bother, when ChatGPT is right there
Because every prompt you send to a hosted model is data leaving your control. For a lot of work that is fine. For plenty of the work I see (client records, contracts, medical and financial detail, anything under an NDA) it is a hard no, and "we'll just be careful what we paste in" is not a policy, it is a hope. A local model removes the question entirely: if the machine is yours and it never phones home, there is nothing to leak, nothing logged on someone else's server, and nothing quietly feeding a future training run.
The other half is control. A subscription is a tap someone else can turn off, reprice, or change the rules on. The model you rely on can be deprecated next quarter; the one you downloaded sits on your drive and runs the same way in three years. This is the same belief I run the whole business on (own your stack, don't rent your dependence) and AI is just the newest place it applies. You don't have to choose one side forever. You just stop defaulting to the rented option for work that never needed to leave the room.
What's actually possible today
This is where most articles either oversell or undersell, so let me be plain. Open-weight models (Llama, Qwen, Mistral, Gemma and the rest) have come far enough that a mid-sized one running on your own hardware is genuinely good at the bread-and-butter work: drafting and rewriting, summarising long documents, answering questions, pulling structure out of messy text, classifying and tidying data, simple coding help. For that bulk of everyday tasks, a local model is not a sad compromise. It is enough, and it does it privately.
Where local still loses, honestly, is the hard end: the trickiest reasoning, very long context, and the most demanding coding. The frontier hosted models are bigger than anything you'll run at home and they still win there. So the right mental model isn't "replace ChatGPT." It's two tools: the local server does the private, repetitive, high-volume work that you'd never want to send away anyway, and you reach for a hosted model on the rare job that genuinely needs the absolute top. That split, not a religious war, is how this actually works in practice.
The hardware, in plain terms
One number rules everything here: VRAM, the memory on your graphics card (or the unified memory on an Apple Silicon Mac). The model has to fit in fast memory to run fast. Miss that and it spills into ordinary RAM and crawls. Everything else is detail.
- The easy on-ramp: an Apple Silicon Mac. An M-series Mac with 32GB of unified memory will run an 8B-14B model quickly and a 30B-class model usably, because the GPU shares that whole memory pool. For a lot of people who already own one, this is the cheapest way in: nothing to buy.
- The sweet spot: one decent GPU. A single 16GB-24GB card (an RTX 4060 Ti 16GB, a 4070 Ti Super, or a second-hand 3090) is the genuine value pick. It'll run a quantised 8B-14B model fast and a 30B-class model at a sensible pace. A used 3090 with 24GB is, dollar for dollar, still the smartest local-AI buy going around.
- The quiet workhorse: a mini PC or old desktop with no GPU. It'll run a small 3B-8B model on the CPU. Slow, but real, and fine for light automation that runs in the background where you're not watching a cursor blink.
- What "quantised" means and why you care. Quantising shrinks a model so it fits in less memory for a small, usually unnoticeable quality hit. A 4-bit (Q4) quant is the standard sweet spot; it's how a 14B model squeezes onto a 16GB card. You almost never want the full-fat version at home.
You do not need a rack, a fan-screaming server, or a fortune. The same instinct that says run your own business tools on one quiet box you own applies here: one well-chosen machine quietly does the job, and you stop paying rent on capability you could host yourself.
The software is the easy part now
This used to be a weekend of dependency hell. It isn't anymore. Ollama is the one I'd start almost anyone on: install it, run one command to pull a model, and you've got a local AI answering on your own machine in minutes, with a clean API other apps can talk to. Prefer a window with buttons? LM Studio gives you the same thing with a proper interface and a model browser. Want the ChatGPT-style chat page for the whole household or office? Put Open WebUI in front of Ollama and everyone on your network gets a familiar chat box, pointed at a model that never leaves the building.
The genuinely useful trick is pointing the model at your own files: retrieval, or "RAG". Instead of a generic assistant, you get one that can answer questions over your own documents, notes and records, locally, with nothing uploaded anywhere. That, far more than chatting for its own sake, is where a private model earns its keep for a real business.
The honest catches
I'd be doing you a disservice if I only sold the upside. Three things are real:
- It is slower than the cloud, and you'll feel it. A big hosted model answers from a data centre full of top-end accelerators. Your one card is your one card. For most work it's perfectly quick; on a long, heavy task you'll notice the difference. That's the tax for keeping it in-house.
- Quality won't match the very best on the hard stuff. Covered above, but worth repeating because it's the thing people get burned on. Match the model to the job and you're happy. Expect a 14B local model to out-reason the frontier and you won't be.
- It's a machine you now have to look after. Self-hosting moves the patching, the updates and the security onto you. A local AI box is still a computer on your network; keep it updated and behind a properly set-up firewall. Convenience you outsource; responsibility you don't. That's the deal with owning anything.
Where I land on it
If your AI work is the private, repetitive, sensitive bulk (and for most businesses it is) a local LLM server is one of the better-value moves you can make right now. You stop renting a dependency, your data stops leaving the room, and nobody can reprice or kill the tool out from under you. Buy or repurpose one sensible machine, put Ollama on it, pull a model that fits the card, and live with it for a fortnight before you decide. Keep a hosted model in your back pocket for the rare job that truly needs the frontier. That's not anti-cloud; it's just refusing to send away work that never needed to leave. It's the same reason I run my own data centre instead of renting a rack with my logo on it. When you own the thing, you control the thing.
Where this comes from
The primary sources behind the above, so you can check any of it yourself rather than take my word for it:
- Large language models, background on open-weight models and what they are.
- Retrieval-augmented generation, the technique behind answering questions over your own documents.
- Australian Signals Directorate: the Essential Eight, the baseline for keeping a self-hosted box patched and protected.
- Office of the Australian Information Commissioner, the national privacy regulator, on handling personal information.
Weighing up a private AI setup for your business and want a straight answer on what hardware to buy and what's actually worth running locally? That's exactly the kind of thing I work through with people: no lock-in, no pushing gear you don't need. Tell me what you're trying to do and I'll point you straight, even if the honest answer is that a subscription suits you better.