Start with the people, not the hardware
Write down who will use it and for what. Five people drafting emails and summarising documents through the day is a light load. Twenty people all asking questions at 9am is a different load. The model answers one request at a time per "slot", and every slot that runs together needs its own working memory on top of the model.
So count three numbers before you look at any price list: how many people have access, how many are likely to ask at the same moment, and the longest document anyone will paste in. The second number sets the memory you need. The third sets how long an answer takes.
Size the memory for the model plus the conversations
Memory on the graphics card (or unified memory on an Apple Silicon Mac) is the number that matters. The model sits in it, and each live conversation takes more. A rule that holds up: take the memory the model needs, then leave a good margin for the people chatting at once.
- Small team, light use. One 16GB to 24GB graphics card, or an M-series Mac with 32GB of unified memory, runs an 8B to 14B model for several people across a working day.
- Busier office or a bigger model. More memory comes first: a 24GB card, then two cards, or a Mac with more unified memory. A second box comes after that, not before.
- Background jobs only. Sorting email, tagging documents and tidying data run overnight on a small machine with no graphics card. Nobody waits on them.
The longer explanation of memory, quantised models and the specific cards is on the page about running a local LLM server on hardware you own.
One box on the network, not one box per person
An AI box is a dedicated machine on your office network. Install a runner such as Ollama, put Open WebUI in front of it, and every staff member gets a familiar chat page in the browser. You manage one model, one set of updates and one set of access rules instead of a copy on every laptop.
Give it a fixed address on the office network, create a login for each person, and keep it off the public internet. If staff work from home, they connect to the office through a VPN, not through a port opened on the router.
Let it read your documents, with the permissions you already have
The most useful setup answers questions over your own files: quotes, procedures, contracts, past jobs. The method is retrieval, or RAG. The server indexes the files on the box, finds the passages that match a question, and hands them to the model to write the answer.
The assistant answers from anything it has indexed, so decide who may see which folders before you index them. Keep payroll and HR files out of a shared index. Index the folders each team already shares, and nothing wider.
For personal information, the Office of the Australian Information Commissioner publishes the rules for handling it. A local server keeps that information on your own network, which makes those rules easier to follow.
Decide who looks after it before you buy it
A local AI server is a computer on your network. It needs these things, every month:
- Operating system and software updates, following the patching advice in the Australian Signals Directorate Essential Eight.
- A backup of its settings, user list and document index, stored somewhere other than the box.
- A firewall rule that keeps it reachable from the office only.
- One named person to call when it stops, with a spare plan for a failed drive or card.
That person can be a staff member or your IT provider. What matters is that the name is written down on day one.
Questions to ask any provider quoting you one
- Which model will run, and how many people can use it at the same moment on this exact hardware?
- Where do prompts, answers and the document index live, and who can read them?
- What happens when a drive or graphics card fails, and how long until staff can use it again?
- Who patches it, how often, and is that in writing?
- Can I take the machine and the setup with me if I leave you?
Where this comes from
- Retrieval-augmented generation, the technique behind answering questions over your own documents.
- Australian Signals Directorate: the Essential Eight, the baseline for patching and protecting a self-hosted machine.
- Office of the Australian Information Commissioner, the national privacy regulator.
Sizing a private AI setup for your team? Tell me how many people and what for and I will give you a straight answer on hardware, setup and who looks after it, including when a subscription suits you better. The same own-what-you-rely-on thinking is in why I run my own data centre.