Configuring a local model matters as much as choosing it

With the same model file, the IT Lab Bench measured differences in speed and in pass outcome attributable to configuration alone. In IT Lab's judgment, comparing models under a generic configuration measures the setup more than the model.

• The same model went from 30.3 to 54.6 tok/s and occupied 13.3 GB of card memory when only the inference-engine configuration changed.
• Two models that did not meet the criterion with reasoning disabled met it once allowed to reason, at the price of many more tokens and seconds.
• The configuration that works for one model and task does not carry over without measurement; it is therefore documented per model and per task.

The business question

A company adopting local AI decides on two facts: whether the model finishes the task, and how long it takes. Both depend on the model and on how it is set to work. If an evaluation does not state its configuration, the company cannot tell whether a poor result belongs to the model or to the setup, nor what to expect when reproducing it on its own machine.

What the comparison measures and why it represents real work

The bench protocol requires researching each model's documentation and giving it its best opportunity: its best parameters and its best way of being instructed. This is both a criterion of fair evaluation and a practical finding. The measurements in this piece isolate one variable at a time, with the same model file, and cover two things a company perceives directly: how fast the model responds, and whether it completes an agentic task (finding an answer with tools and delivering it) without human intervention.

How a model passes

Speed and memory are as recorded by the inference server. Passing is decided by the bench's fixed automatic judge: deterministic programs, with no model evaluating another, and a threshold set before the run. The diagnostics in this piece are labeled as such and are never mixed with the official pass.

Results

Speed: the same card, four configurations

Qwen3.8-27B-GSQ-RCO model, 3-bit quantization, on IT Lab's test machine. Generation speed and memory are reported; engine configurations are labeled generically.

The optimized configuration was the fastest and also the one that occupied the least memory. In IT Lab's judgment, memory is what makes the result matter: with an unquantized cache, speculative decoding left the model at the limit of the card, and only with the quantized cache did the combination fit with margin. Under the reference configuration, the model would have looked slow because of the setup and not because of its own capabilities.

More speculative drafting does not mean more speed

Speculative decoding proposes several tokens ahead and the model confirms which it accepts. In the bench, increasing how far ahead it proposes did not improve speed continuously: past an intermediate point, the acceptance rate falls and speed falls with it. The best point depended on the model's working mode, and the recorded speed varied with it. The bench therefore reports the acceptance rate alongside speed, and does not compare speed figures taken under different sampling conditions.

Pass outcome: allowing reasoning changes the result and has a cost

Diagnostic, not the official pass. Agentic test, reasoning disabled versus medium reasoning; scores are out of 100.

In both models the disabled score fell below the threshold; with reasoning, they cleared it. The pattern IT Lab observes is that reasoning repairs concrete calculation errors. The cost is substantial: the smaller model needed 688.6 s for what it previously attempted, and failed, in 17.6 s. If the task is interactive, that difference can rule the option out; if it runs in batches, it may be acceptable.

The reasoning switch is not homogeneous across families

"Reasoning disabled" does not mean the same thing in every model. In the diagnostic, some models produced no reasoning tokens; others produced the same reasoning with the switch on and off; and one with no control at all spent 4,910 reasoning tokens in a run labeled as non-reasoning. Intermediate levels do not exist in every family either: one model rejected the highest level with an error, and others only distinguish on from off. The bench therefore does not compare that level across families, and a higher level does not imply a proportionally higher reasoning spend. In addition, if the output limit is low, reasoning consumes the budget and the answer arrives empty; the protocol sets the output limit accordingly.

What this means for a Chilean company

• Before comparing two models, compare the conditions: engine, available memory, reasoning mode and output limit. A speed or pass difference between two reports lacking that information is not comparable.
• Document the configuration per model and per task. A configuration that passed one task does not guarantee another, or another model's, and an old result does not carry over to a new configuration without repeating the measurement.
• Decide with the cost of reasoning in view: a higher score can mean many times more time per answer. Which task can bear it is a business decision, not the model's.

Frequently asked questions

Why does the configuration of a local model matter? The same model went from 30.3 to 54.6 tok/s and used 13.3 GB of card memory when only the inference-engine configuration was changed.

Does enabling reasoning improve the result? Two models that did not meet the criterion with reasoning disabled met it once allowed to reason, at the price of many more tokens and seconds.

Can another model's configuration be copied? The configuration that works for one model and task does not carry over without measuring; that is why it is documented per model and per task.

How much does local AI cost? The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.

Limitations and scope

• A single pass per model and case unless another number is stated; small differences are noise.
• Speed was measured under one context and sampling configuration; figures are not mixed across different conditions.
• The reasoning diagnostics are probes kept separate from the official pass.
• "Reasoning disabled" is not homogeneous across model families: those levels should not be compared across families.
• The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

Measurement sheet

• Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
• Quantization: 3 or 4 bits, depending on the model.
• Passes: per measurement, stated in the text.
• IT Lab Bench, edition 1.

Related technical report: Technical report: running local AI on 16 GB

Full bench method: How IT Lab evaluates a local AI model

Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.