Technical report 3: Running local AI on a 16 GB GPU

Code: ITL-IT-04 · IT Lab Bench, edition 1

Summary

This report describes what an organization can expect from a 16 GiB graphics card when running local AI models on real work, based on IT Lab Bench measurements. It covers five subjects: the memory that is actually usable, the effect of the inference-engine configuration, behavior with long real context, the cost of enabling reasoning, and the limits of what was measured.

In the bench, the same model went from 30.3 to 54.6 tok/s depending on the engine configuration, and the fastest configuration was also the one using the least memory (13.3 GB). With large real documents, reading speed fell 13-fold near 50,000 tokens, and generation dropped from 36-43 tok/s with short context to 7.5-14 tok/s with 80K-110K tokens occupied. Enabling reasoning turned two failing results into passing ones in a diagnostic, at the price of a considerable increase in tokens and time. In IT Lab's judgment, a 16 GiB card suits quantized models from 12B to 35B serving one person or one process at a time, and it is not enough to promise very long context at good speed or simultaneous use by several users, which was not measured.

1. Introduction and question

A small organization deciding to run local AI on its own documents usually starts from the hardware sheet: the amount of card memory and the context size the model advertises. Neither figure answers the operational question. What matters is which models fit, at what speed they work with real documents, and what happens when the context window fills up.

This report answers three sizing questions with bench measurements:

1. How much of the nominal memory of a 16 GiB card is really available to the model.
2. How much speed and memory change with the inference-engine configuration, for one and the same model.
3. How speed behaves when context is real and long, and what it costs to ask the model to reason.

The methodology report, Technical report 4: IT Lab Bench methodology, describes the general protocol. The articles Configuring a local model matters as much as choosing it and SMB hardware for local AI with 16 GB present a short version of the configuration and hardware findings that this report documents in detail.

2. Method

Equipment and measurement

Measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.

Speed and memory are reported as recorded by the inference server, not as estimates. Generation speed is expressed in tokens per second (tok/s); reading speed is the processing of the input text before generation begins. Memory is reported as the card's peak during the run.

Measurement families

The report uses three kinds of measurement, all with the same model or family of models and without changing the evaluation instrument between configurations:

• Engine-configuration benchmark. One real code case, a context of 65,536 tokens and a fixed output length, repeated over different engine configurations for the same model. It measures generation speed, reading speed and peak memory.
• Prebuilt real context. Complete real documents, assembled in a single piece with a token count (79,964, 109,874, 198,270, 227,733 and 247,581 tokens), instead of filling the context in increments. It measures reading and generation with the window occupied.
• Reasoning diagnostic. A repeat of an agentic-work exam with reasoning disabled and enabled, to measure how much it corrects and what it costs in tokens and time.

Judge, pass criterion and best opportunity

When a measurement includes a pass decision, a fixed, deterministic automatic judge makes it, with no model evaluating another and with the pass criterion set before the run. The protocol also requires giving each model its best configuration, so that differences in speed and pass results are not attributed to the model when they come from a poor setup. Configurations are documented per model and their fine values are not published.

Quality controls relevant to this report

• GPU memory control. The memory margin is calibrated against the real state of the machine: at least 100 MB of memory left unused above the normal state of the system is required, instead of a fixed limit of 15.5 GiB. A wrongly set fixed limit can discard viable configurations without warning.
• Prebuilt real context in a single piece. It prevents artificial incremental filling from hiding the behavior of the full window.
• Measuring the maximum context that fits in memory. If a model does not fit under the card's limit, the maximum context it supports is probed and documented; the model is not excluded. In an agentic-work battery, two models settled at 131,072 and 200,704 tokens under a 15.4 GiB cap.
• Speed with speculative decoding. Speculative decoding uses a draft of tokens that the model then confirms. Repeated content can inflate its speed, so the bench measures it on a new task and declares the draft acceptance rate.
• Separation of evidence. The reference pass is reported apart from probes and diagnostics, in labeled sections.

3. Results

Usable memory

Of the card's nominal 15.922 GiB, the bench's memory budget subtracts 1.28 GiB reserved for the system and 0.4 GiB of margin, leaving 14.242 GiB usable. The deduction is a reservation for the system and the safety margin, not the use of other applications during testing.

In IT Lab's judgment, that figure, and not the one on the spec sheet, is the one to use for sizing. With that memory, the bench loaded models from 12B to 35B in 3-bit or 4-bit quantization with a working context of 64K to 131K tokens. The context ceiling that memory allows varies by model, from 32,768 up to 262,144 tokens, and it is a loading ceiling, not a measure of performance or of comprehension of the full window.

With the full text context, vision exceeds the 15.5 GiB memory cap; to include images the context has to be reduced.

Engine configuration: reference pass (same task, same model)

The table compares five inference-engine configurations with the same model, Qwen3.8-27B-GSQ-RCO, quantized to 3 bits, on the same real code case with a context of 65,536 tokens. One run per condition. The sampling temperature is not declared in the source for these rows, so their figures are not compared with those of other tables in this report. IT Lab Bench, edition 1.

For the alternative without speculative decoding, the source also records a second run of 30.1 tok/s. For the configuration with speculative decoding without quantized cache, the second run records 40.1 tok/s. In the optimized configuration, reading speed was 841 tok/s and the draft acceptance rate was 0.67. An intermediate speculative-draft variant ran at 52 to 57 tok/s but did not close a result because of the memory limit, and it is reported without a figure.

Three readings by IT Lab. First, the reference configuration was not the fastest in the series. Second, the alternative without speculative decoding performed worse and used more memory than the reference. Third, the optimized configuration was the fastest and the one using the least memory, 3.6 GB less than the highest-consumption configuration in the series. The gain comes from combining a quantized attention cache with speculative decoding; with the cache at full precision and speculative decoding at 64K, the peak reached 16.9 GB.

In a battery of five programming cases, the optimized configuration passed all five at speeds of 50.7 to 57.8 tok/s and memory of 14.2 to 14.4 GB, in a single pass.

Speculative decoding: speed and acceptance rate

Speculative decoding was measured with the model's own draft actually loaded, on a new code case, using Swift Flash-Next quantized to 3 bits and with reasoning both enabled and disabled. One run per condition. Under those conditions, generation was 59.7 to 72.5 tok/s and the draft acceptance rate was 66.9% to 87.3%. Total times per case ranged from 19.5 s to 2 min 06.9 s, and tokens generated from 1,137 to 7,337.

Two conclusions hold. A longer draft does not mean more speed: within each mode, the best result was not at the extreme. And the acceptance rate changes with the reasoning mode and with the sampling. Because the outputs of the six runs had very different lengths, IT Lab does not declare a universally optimal draft configuration; the value of the figure lies in its range and in the fact that it was measured on a new task. For that reason, the bench does not mix speeds obtained with different sampling in the same table.

In the optimized configuration of the engine-configuration table, the acceptance rate was 0.67, and other production runs of the bench recorded rates between 0.57 and 0.67.

Triage of sizes and quantizations (probe)

This probe is reported apart from the reference pass. It is a two-case measurement with a context of 32,768 tokens and a quantized cache, and the sampling temperature was not recorded: the speed in each row is reliable as a measurement, but the comparison between rows is not controlled.

Short-context speeds across the bench's campaigns ranged from 40 to 172 tok/s depending on the model, and code profiles sustained between 51 and 58 tok/s, with 56 tok/s weighted at 64K.

Real context: the reading drop

With prebuilt real contexts, reading is fast up to about 50,000 tokens and then falls 13-fold, from about 665 tok/s to 54 tok/s, without recovering as the window fills. Filling a context of 110K tokens took about 34 minutes. A separate measurement, with five quantizations of the same base model and the same parameters confirmed in the server log, showed the same shape: reading held between 508 and 571 tok/s up to about token 54,510 and then dropped to about 52 tok/s.

The cause of this drop is probable, not confirmed: in IT Lab's judgment, the most plausible explanation is that what fits in card memory runs out and the rest of the attention cache is read from system memory. The bench did not isolate that mechanism; the shape of the drop is a measured fact, while its explanation is a hypothesis.

The drop is not a constant of the window size. In a different profile of the same bench, with Swift Flash-Next quantized to 3 bits, a session of 80,124 real tokens was read at 881.32 tok/s in 1 min 30.9 s, with generation at 50.4 tok/s and a peak of 15.33 GiB. The two measurements are not comparable with each other, but they show that long-context speed depends on how memory is configured, and that it must be measured with the profile that will be used in operation.

Generation with a full context

Generation also degrades with the window occupied. On the same model, speed went from 36-43 tok/s with short context to 7.5-14 tok/s with 80K-110K tokens occupied, with a memory peak of about 15.46 GB. The sizing implication is direct: a model that "supports 128K" and a model that works at good speed with 128K occupied are different statements, and only the second matters for reading long documents without taking them out of the organization.

Probing the maximum context that fits

When a model does not fit at its nominal context, the bench does not exclude it: it searches by bisection for the maximum context it supports under the memory cap and documents it. With the usable memory described above, the loading ceiling of Qwen3.8-27B-GSQ-RCO and of Qwopus3.6-35B is 262,144 tokens. This is a measure of loading capacity. Effective comprehension of that full window was not measured and must not be inferred from it.

Cost of reasoning (diagnostic)

This subsection is a diagnostic, not an official pass. It repeats an agentic-work exam with reasoning disabled and with reasoning enabled at medium level. One pass per condition. The output cap must be raised when reasoning is enabled; otherwise reasoning consumes the content and the response arrives empty.

In both Gemma models, enabling reasoning corrected one and the same arithmetic episode that was worth 30 out of 100 points with reasoning disabled, and with it they went from not meeting to meeting the criterion. In the other two models, the result already met the criterion without reasoning and reasoning only increased tokens and time. The smallest model in the table took its time from 17.6 s to 688.6 s for that change.

The diagnostic also left three observations about how these levels are compared:

• Requesting disabled reasoning does not guarantee that the server stops reasoning. Two models returned zero reasoning tokens, a third returned the same reasoning block with the switch off and on, and a paid model without a level control spent 4,910 reasoning tokens in the disabled official run. For that reason "reasoning disabled" is not homogeneous across families and is not compared between them.
• Levels do not exist evenly: one model supports several levels and rejects the highest without generating a single token; two others only distinguish on and off, and their outputs at higher levels are byte-identical.
• A higher level does not imply proportional spending: in one model, the top level used less reasoning than the medium one (300 versus 393 tokens).

In code, the same axis showed a similar pattern. With a Qwen3.8 variant and a 64K context, disabled reasoning passed 1 of 3 cases, low reasoning 2 of 3 in 10 min 34 s and medium reasoning 3 of 3 in 9 min 53 s. That comparison does not isolate a single variable, because the 24K and 64K passes used different memory and sampling configurations.

Concurrency: not measured

All measurements were taken in series, with one model loaded at a time. The bench did not measure how many queries per hour the machine sustains with several simultaneous users, nor latency under load, nor interference between requests. There is therefore no concurrency data in this report, and no figure should be read as service capacity for a work team.

4. Discussion

Signal. Three results exceed what could be attributed to single-pass noise. The first is the speed difference between engine configurations for one model, from 30.3 to 54.6 tok/s, accompanied by a memory difference of 3.6 GB. The second is the shape of the reading drop near 50,000 tokens, measured in two different series with the same shape. The third is the magnitude of the cost of reasoning in tokens and time, which in the case of the smallest model took the time from 17.6 s to 688.6 s.

Noise. Differences between draft configurations for one model, with outputs of very different lengths, do not allow declaring an optimal configuration. Neither do the comparisons in the triage probe, whose sampling was not recorded, nor those of the code exams with different context configurations. A 3 out of 3 from one campaign is not comparable with a 3 out of 3 from another, because the judges and contracts changed between campaigns.

Interpretation. In IT Lab's judgment, the engine configuration is part of the result and not an installation detail: a generic configuration would have attributed to the model a slowness that belonged to the setup, and in the engine-configuration series the same model file performed between 30.3 and 54.6 tok/s depending on the configuration. Likewise, the context a model advertises is not equivalent to performance at that context, and the second is the statement that matters to an organization.

What is not concluded. The report does not conclude that any draft configuration is best in general, that reasoning should always be enabled, or that the reading drop is due to a specific mechanism. Nor does it compare the performance of this card with that of others.

5. Implications for organizations with limited resources

The following recommendations are IT Lab's sizing judgment, based on its measurements. They are not a purchase recommendation.

1. Size on usable memory. On a 16 GiB card, plan on 14.242 GiB and reserve the system margin. A configuration whose peak sits above that budget may fail to load or destabilize the machine.
2. Choose the model in the 12B to 35B quantized range. That is the range the bench could load with a working context of 64K to 131K tokens, with useful speeds at short context.
3. Configure the engine per model. Keep a documented configuration per model and compare speeds only between runs with the same task, the same sampling and the same load. When using speculative decoding, measure on a new task and record the acceptance rate; a speed measured on repeated content overstates real performance.
4. Test with the real context of the work. Before committing a workflow to long documents, measure reading and generation with documents of the real size. Near 50,000 tokens, reading fell 13-fold in the bench, and with 80K-110K tokens occupied generation dropped from 36-43 to 7.5-14 tok/s. Workflows that demand very long context at good speed should not be promised on this class of card without a measurement of their own.
5. Split the work where possible. For tasks on long documents, in IT Lab's judgment it is preferable to design them to operate on shorter contexts rather than depend on the full window.
6. Enable reasoning only where it corrects. Reasoning corrected specific arithmetic in two models, but in two others it only raised the cost. Decide per task and per model, and raise the output cap when enabling it.
7. One model at a time. Size for serving one person or one process at a time. Simultaneous use by several users was not measured, and neither a service-level agreement nor capacity for the whole company should be promised.
8. Measure the meter. Require any speed figure to state the engine configuration, the sampling, the available memory and the context size at which it was obtained.

Frequently asked questions

How much memory of a 16 GiB card is usable? Of the nominal 15.922 GiB, 14.242 GiB proved usable once the margin for the system was reserved.

What happens with very long documents? With large real documents, reading fell 13-fold near 50,000 tokens, and generation dropped from 36-43 tok/s to 7.5-14 tok/s with 80K-110K tokens filled.

Does the inference-engine configuration matter? Yes: the same model went from 30.3 to 54.6 tok/s depending on configuration, and the fastest was also the one with the lowest memory (13.3 GB).

How much does local AI cost? The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.

6. Limitations and scope

• A single pass per condition and case unless another number is stated. Small differences are noise.
• The reference pass is kept separate from probes and diagnostics: the size triage is a probe and the cost of reasoning is a diagnostic.
• The sampling temperature of several engine-configuration runs was not declared in the source. Speeds from different sampling are not mixed.
• Memory cap of the configuration cited: 14.242 GiB usable of 15.922 GiB, with a monitoring cap of 15.5 GiB.
• "Reasoning disabled" is not homogeneous across model families; those levels are not compared between families.
• The cause of the reading drop is probable, not confirmed.
• The configuration with the longer speculative draft did not load, and an intermediate variant did not close a result.
• Concurrency was not measured.
• The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

7. Conclusions

1. In IT Lab's judgment, the usable memory of a 16 GiB card is 14.242 GiB, and it is the figure to size on.
2. In the bench, the engine configuration changed the speed of one model from 30.3 to 54.6 tok/s, and the fastest configuration was also the one with the least memory, 13.3 GB.
3. Reading with real context fell 13-fold near 50,000 tokens, and generation dropped from 36-43 to 7.5-14 tok/s with 80K-110K tokens occupied; the cause is probable, not confirmed.
4. In the diagnostic, enabling reasoning made two models go from not meeting to meeting the criterion, with an increase in tokens and time that in the smallest model took the run from 17.6 s to 688.6 s.
5. Concurrency was not measured. The sizing IT Lab proposes for this class of card is one model at a time, with the working context measured on the real task.

Measurement sheet

• Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
• Quantization: 3 or 4 bits, depending on the model.
• Context: 65,536 tokens in the configuration benchmark; real contexts of 79,964 to 247,581 tokens in the long-context tests.
• GPU memory cap: 14.242 GiB usable of 15.922 GiB nominal; monitoring cap of 15.5 GiB.
• Passes: one per condition unless another number is stated.
• IT Lab Bench, edition 1.

How to cite

IT Lab, Technical report ITL-IT-04, IT Lab Bench, edition 1.

Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.