Angus Lewington

Notes from someone who builds the thing

A local AI server for a small business: what to size and what to ask

A local AI server is one machine in your office that runs a language model for your whole team. Staff open a chat page in a browser, the box answers, and no prompt or document leaves your network. Four things decide whether it works for you: how many people use it at once, how much memory the model needs, which documents it can read, and who looks after it. This page covers each one in order.

Start with the people, not the hardware

Write down who will use it and for what. Five people drafting emails and summarising documents through the day is a light load. Twenty people all asking questions at 9am is a different load. The model answers one request at a time per "slot", and every slot that runs together needs its own working memory on top of the model.

So count three numbers before you look at any price list: how many people have access, how many are likely to ask at the same moment, and the longest document anyone will paste in. The second number sets the memory you need. The third sets how long an answer takes.

Size the memory for the model plus the conversations

Memory on the graphics card (or unified memory on an Apple Silicon Mac) is the number that matters. The model sits in it, and each live conversation takes more. A rule that holds up: take the memory the model needs, then leave a good margin for the people chatting at once.

  • Small team, light use. One 16GB to 24GB graphics card, or an M-series Mac with 32GB of unified memory, runs an 8B to 14B model for several people across a working day.
  • Busier office or a bigger model. More memory comes first: a 24GB card, then two cards, or a Mac with more unified memory. A second box comes after that, not before.
  • Background jobs only. Sorting email, tagging documents and tidying data run overnight on a small machine with no graphics card. Nobody waits on them.

The longer explanation of memory, quantised models and the specific cards is on the page about running a local LLM server on hardware you own.

One box on the network, not one box per person

An AI box is a dedicated machine on your office network. Install a runner such as Ollama, put Open WebUI in front of it, and every staff member gets a familiar chat page in the browser. You manage one model, one set of updates and one set of access rules instead of a copy on every laptop.

Give it a fixed address on the office network, create a login for each person, and keep it off the public internet. If staff work from home, they connect to the office through a VPN, not through a port opened on the router.

Let it read your documents, with the permissions you already have

The most useful setup answers questions over your own files: quotes, procedures, contracts, past jobs. The method is retrieval, or RAG. The server indexes the files on the box, finds the passages that match a question, and hands them to the model to write the answer.

The assistant answers from anything it has indexed, so decide who may see which folders before you index them. Keep payroll and HR files out of a shared index. Index the folders each team already shares, and nothing wider.

For personal information, the Office of the Australian Information Commissioner publishes the rules for handling it. A local server keeps that information on your own network, which makes those rules easier to follow.

Decide who looks after it before you buy it

A local AI server is a computer on your network. It needs these things, every month:

  • Operating system and software updates, following the patching advice in the Australian Signals Directorate Essential Eight.
  • A backup of its settings, user list and document index, stored somewhere other than the box.
  • A firewall rule that keeps it reachable from the office only.
  • One named person to call when it stops, with a spare plan for a failed drive or card.

That person can be a staff member or your IT provider. What matters is that the name is written down on day one.

Questions to ask any provider quoting you one

  • Which model will run, and how many people can use it at the same moment on this exact hardware?
  • Where do prompts, answers and the document index live, and who can read them?
  • What happens when a drive or graphics card fails, and how long until staff can use it again?
  • Who patches it, how often, and is that in writing?
  • Can I take the machine and the setup with me if I leave you?

Where this comes from

Sizing a private AI setup for your team? Tell me how many people and what for and I will give you a straight answer on hardware, setup and who looks after it, including when a subscription suits you better. The same own-what-you-rely-on thinking is in why I run my own data centre.

Frequently asked questions

How many people can use one local AI server at the same time?

It depends on the model and the memory. Each person who chats at the same moment needs memory for their own conversation on top of the memory the model itself uses. A single 16GB to 24GB graphics card running an 8B to 14B model serves a small team comfortably when requests are spread across the day. If many people ask at once, or you want a bigger model, add memory first, then a second machine.

What is an AI box?

An AI box is one dedicated machine on your office network that runs a local language model. Staff open a chat page in their browser and the box answers. Prompts and documents stay on your network. The box can be a small desktop with one graphics card, or a Mac with plenty of unified memory.

Can a local AI server answer questions about our own documents?

Yes. The method is called retrieval, or RAG. The server indexes your documents on the box, finds the passages that match a question, and gives them to the model to write the answer. The documents are not uploaded to any outside service. Set who can see which folders before you index them, because the assistant can answer from anything it has indexed.

Who looks after a local AI server?

Someone must own it. The box needs operating system and software updates, a backup of its settings and document index, a firewall rule that keeps it off the public internet, and a person to call when it stops. That person can be a staff member or your IT provider. Decide who before you buy.

Weighing up a private AI setup?

Tell me what you're trying to do and I'll give you an honest read on what's worth running locally and what isn't.

Get in touch