Technical report 4 (2/2): results and conclusions

Part 2 of 2. Read Part 1.

3. Results

This section brings together the evidence that the controls and principles of the method have a measurable effect. Per-model results for each family are in the reports on office capabilities and on code.

Effect of the controls on the instrument

Legend: IT Lab measurement; speed and memory values are those recorded by the inference server. IT Lab Bench, edition 1.

Effect of the inference-engine configuration

The same model, under an optimized inference-engine configuration combining a quantized attention cache and speculative decoding, went from 42.4 to 54.6 tok/s and occupied 13.3 GB of card memory. A generic configuration would have attributed to the model a slowness that belonged to the setup. A model's speed cannot be separated from the configuration with which it was measured. The sizing detail is developed in the technical report on running local AI on a 16 GB GPU, and the general principle in the article configuring a local model matters as much as choosing it.

The criterion's ability to distinguish models

A useful criterion neither passes everything nor fails everything. With the criterion set before the run, pass rates varied between families.

Legend: one pass per model and case; agentic work with threshold 80; in code, passing all stages of the judge. In code from scratch, two paid models pass all five stages. IT Lab Bench, edition 1.

Difficulty is not linear across families. For the method, what matters is that the judge discriminates: in reading, 12 of 14 models extract all 183 cells without error, and the difference is decided on other questions of the same exam.

Diagnostic: reasoning enabled

This subsection is a diagnostic and is not part of the official pass. On the agentic-work exam, the run of two models was repeated with reasoning enabled, to measure how much it corrects and what it costs.

Legend: diagnostic, not official pass; threshold 80; IT Lab Bench, edition 1. "Reasoning disabled" is not comparable across model families.

Enabling reasoning took both models over the threshold, at the price of a much larger number of tokens and much more time. For a model, the official pass shows the floor of the criterion; the diagnostic shows what it costs to clear it. Mixing the two would make a result obtained at a disproportionate cost look like a native capability of the model.

4. Discussion

Signal and noise. In IT Lab's judgment, four findings are signal. The first is the drop from 82.9% to 76.2% when a public exam is extended, which shows that the base score does not measure robustness. The second is that speed does not predict success: a model at 74.49 tok/s passed 2 of 8 tests. The third is the effect of configuration, from 42.4 to 54.6 tok/s on the same model, and from 76 and 74 to 100 and 97 in score once reasoning is allowed. The fourth is that the instrument's controls change conclusions: format repair rescues responses the judge would have counted as model failures.

Small differences obtained in a single pass are noise. The bench reports them but does not rank models with them.

Why a deterministic judge. A model that evaluates another introduces a second source of variation the bench cannot control. A fixed program that stops at the first failed step also delivers a diagnosis: it reports the point at which the work broke. The counterpart is that the judge is comparable only within the same brief and the same judge; a pass on a bench coding exercise does not equal "passed in that language" in general.

5. Implications for organizations with limited resources

In IT Lab's judgment, a Chilean company evaluating local AI with its own documents and data should require that any evaluation, its own or a vendor's, state at least six points: which task was measured, how it was passed, with what configuration, on what hardware, how many passes were made and what was done with format failures. A speed table without those answers cannot support a decision. A machine with a 16 GiB card is enough to evaluate local models seriously if one model runs at a time and the context that really fits is measured, and the choice between a local and a paid model should rest on the passing score and the time measured, not on an isolated speed figure. Evaluation with the company's own material, limited to the task it wants to automate, is more informative than a public score; IT Lab adapts the procedure to each client's documents and data.

Per-family results are in the related technical reports: office capabilities, code and running local AI on a 16 GB GPU. A plain-language version of this method is in how IT Lab evaluates a local AI model, and the discussion of public benchmarks is in why traditional AI benchmarks miss real work.

Frequently asked questions

What is the IT Lab Bench? It is the instrument with which IT Lab measures whether a local AI model completes office or development work on a machine within reach of a small company.

Why is a public benchmark not enough? In public code tests, the extended version of the same exam lowered one model's score from 82.9% to 76.2%, and a set of 148 tasks measured different work from what was meant to be automated.

How is it decided whether a model passes? With a deterministic, staged judge and a pass threshold set before the run.

How much does local AI cost? The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.

6. Limitations and scope

• A single pass per model and case unless the family report states another number; small differences are noise.
• The official pass is kept separate from probes, diagnostics and targeted retries.
• The GPU memory limit of the tests cited is 15.4 GiB; maximum-context results depend on that limit.
• "Reasoning disabled" is not homogeneous across model families; those levels are not compared between families.
• The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• Staged judges are comparable only within the same brief and the same judge.
• The figures are IT Lab measurements on a single machine and have not been recalculated by a third party.
• This report is not a purchase recommendation or a certification.
• Configuration values, engine components and test content are not described; IT Lab reserves them.

7. Conclusions

1. In IT Lab's judgment, public coding benchmarks are not enough to decide what to automate: the extended version of an exam lowered a score from 82.9% to 76.2%, and a set of 148 tasks measured work different from the work sought.
2. The IT Lab Bench measures recognizable office and development tasks through seven test families, with a deterministic judge, staged in code, and a threshold set before the run.
3. The protocol gives each model its best opportunity. In the bench, configuration changes the result as much as the choice of model: from 42.4 to 54.6 tok/s on the same model, with 13.3 GB of memory.
4. The official pass is reported apart from probes and diagnostics, and the instrument is subject to its own controls; one of them rescued 16 of 187 responses with no regressions.
5. Every relevant new model is run through the same bench and published with the edition number, which keeps results within one edition comparable.

Measurement sheet

• Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
• Quantization: 3 or 4 bits, depending on the model.
• Context: the maximum each model supports under the memory limit; prebuilt real contexts of 79,964 to 247,581 tokens in the long-context tests.
• GPU memory limit: 15.4 GiB in the tests cited.
• Passes: one official pass per model and case unless another number is stated.
• IT Lab Bench, edition 1.

How to cite

IT Lab, Technical Report ITL-IT-01, IT Lab Bench, edition 1.

Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.