SMB hardware for local AI with 16 GB

A 16 GiB graphics card lets a small or mid-sized company run quantized models from 12B to 35B, but speed drops sharply once the context is well filled. This piece gathers what the IT Lab Bench measured about that limit and the sizing criteria IT Lab draws from it.

Of 15.922 GiB nominal, 14.242 GiB turned out to be usable once a margin was reserved for the system.
Reading a long context is fast up to about 50,000 tokens and then collapses; with 80K to 110K tokens occupied, generation falls from 36-43 to 7.5-14 tok/s.
Everything was measured with one model loaded at a time; serving several simultaneous users is not measured.

The business question

A Chilean company that wants to keep documents and data inside its own network needs to know what hardware to buy and what work it can assign to it. A card's spec sheet states its nominal memory and a model's sheet states the context window it supports; neither says at what speed the pair works once the window fills. That figure decides whether a local AI project is viable on affordable hardware, and it is the one the bench measures.

What the bench measures and why it represents real work

The bench records, as logged by the inference server, context reading speed (the time the model takes to process a document before answering), generation speed and memory used. The long-context tests use complete real documents, prebuilt in a single piece with a token count, not an artificial fill in increments: 79,964, 109,874, 198,270, 227,733 and 247,581 tokens. They stand for office work a company wants to delegate, such as reading a long contract, a case file or a set of meeting minutes.

When a model does not fit under the card's memory limit, the protocol measures the maximum context it supports instead of excluding it, and documents it. The memory margin is calibrated against the real state of the machine, so that viable configurations are not discarded because of a wrongly set limit.

How speeds are recorded

Speeds are recorded on a new task and, when speculative decoding is used, the acceptance rate is declared, since repeated content can inflate the figure. In the long-context runs, the acceptance rate ranged from 33 to 61 %. Long-context and short-context speed figures are not mixed in a single comparison.

Results

What fits on the card

Of the card's 15.922 GiB nominal, 14.242 GiB are usable, because part is reserved as a margin for the system. Within that budget, in IT Lab's judgment, the following fit:

Quantized models from 12B to 35B, with a working context of 64K to 131K tokens.
A theoretical context ceiling of 262,144 tokens in four of the models evaluated.
Speeds of 40 to 172 tok/s with a short context, depending on the model.

Supporting a long context is not working well with it

"Supports 128K tokens" and "works at good speed with 128K tokens" are different statements. The bench measured the second:

Measurement	Short context	Full context
Context reading	about 665 tok/s at 51K tokens	54 tok/s at 52K tokens
Generation	36-43 tok/s	7.5-14 tok/s with 80K to 110K tokens occupied
Filling 110K tokens		about 34 minutes

Legend: one pass per measurement; IT Lab Bench, edition 1; the card's memory limit is the one described in the measurement sheet.

Reading keeps its pace until about 50,000 tokens and does not recover afterwards: it is an abrupt drop, not a gradual slope. The cause is probable, not confirmed: what fits in the card's memory would be exhausted and the rest of the attention memory would move to system RAM, which is much slower. The figure, by contrast, is measured and reported as is.

One model at a time

All measurements were taken with a single model loaded, in sequence. The bench did not measure how many queries per hour the machine sustains with several users. For that reason this piece offers no capacity figure for a whole company and no service-level commitment. Keeping two models loaded at once on 16 GiB was not evaluated either, and edition 1 does not support it.

What this means for a Chilean company

These sizing recommendations are IT Lab's judgment, based on its measurements.

Automate on this class of hardware. Bounded jobs on medium-length documents: extracting data, finding a specific fact, completing an existing code base. They fit comfortably within the memory budget and run at useful speed.
Review or size more carefully. Any workflow that depends on reading documents longer than about 50,000 tokens or on holding a long conversation with a full context. The answer arrives, but in times that can make interactive use impractical. One alternative is to split the document and work in parts.
Do not promise. Serving several users at once from a single 16 GiB machine. Before committing to it, the organization's real load should be measured.
What to buy. The bench card was chosen for its cost-benefit ratio for a small company. How a higher-capacity card would behave is not measured in this edition and is not claimed.

When evaluating a vendor or a product, IT Lab considers it advisable to ask for the speed with a full context, not only the maximum window declared.

Frequently asked questions

What hardware does a small business need for local AI? A 16 GiB graphics card can run quantized models from 12B to 35B.

What happens with long documents? Reading a long context is fast up to about 50,000 tokens and then collapses; with 80K to 110K tokens filled, generation drops from 36-43 to 7.5-14 tok/s.

Does it serve several users at once? Everything was measured with one model loaded at a time; serving several simultaneous users has not been measured.

How much does local AI cost? The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.

Limitations and scope

A single pass per measurement; small differences are noise.
The cause of the reading drop near 50,000 tokens is probable, not confirmed.
One model at a time; multi-user concurrency not measured.
Each figure corresponds to the memory limit of the configuration cited and to this machine; with another card or configuration the values will differ.
The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

Measurement sheet

Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
Quantization: 3 or 4 bits, depending on the model.
Context: prebuilt real documents of 79,964 to 247,581 tokens.
Card memory: 14.242 GiB usable of 15.922 GiB, with the margin reserved for the system.
Passes: one per measurement.
IT Lab Bench, edition 1.

Related method: How IT Lab evaluates a local AI model

Related technical report: Technical report 3: Running local AI on a 16 GB GPU

Contact: IT Lab sizes and evaluates local AI for Chilean organizations using their own documents and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.