Technical report 4: IT Lab Bench methodology (1/2)

Code: ITL-IT-01 · IT Lab Bench, edition 1

Summary

This report describes the methodology of the IT Lab Bench, the instrument IT Lab uses to measure whether a local AI model finishes office or development work on hardware within reach of a small company. It presents the test families, the deterministic and staged judge, the pass threshold set before the run, the declared single pass, the principle of the best opportunity for each model, the separation between the official pass, probes and diagnostics, the quality controls applied to the instrument itself, the metrics and the publication policy.

The motivation is measured. In public coding benchmarks, the extended version of the same exam lowered one model's score from 82.9% to 76.2%, and a set of 148 tasks measured work different from the work the lab wanted to automate. Among the controls, deterministic format repair rescued 16 of 187 responses with no regressions, and an optimized inference-engine configuration took one model from 42.4 to 54.6 tok/s with 13.3 GB of memory. In IT Lab's judgment, an evaluation result is useful for decisions only when it is published together with its method; this report sets out the method and reserves the content of the tests.

1. Introduction and question

An organization considering local AI needs to know which task it can delegate, which one a person must review, and which hardware makes it possible. The question of this report is whether the available evaluation instruments answer that and, if they do not, which instrument does. IT Lab's answer is the IT Lab Bench: a set of proprietary tests built around recognizable office and development tasks, with fixed automatic judges and a protocol that gives each model its best opportunity.

What the public benchmarks showed

IT Lab evaluated local models with public coding benchmarks before defining its own bench. The base score of a well-known exam turned out to be the easy version of the exam. On HumanEval (164 problems, one sample per problem), the extended version with additional test cases, HumanEval+, lowered the score of all four models measured.

Legend: pass@1, one pass per model and problem; pp: percentage points. IT Lab Bench, edition 1 (pre-bench phase).

In IT Lab's judgment, a model that drops from 82.9% to 76.2% once hidden cases are added has solutions that work but are fragile. The same comparison shows why a clean pass does not allow second chances: a variant with targeted retries of the first model scored 97.6% and 93.3%, and that figure is not compared with the direct pass because it is not the same test.

A second set, BigCodeBench Hard, has 148 tasks. Its content spans scientific plotting, machine learning, web data extraction, cryptography and language processing, among others. It asks for isolated functions, not complete applications with repository context, and many tasks depend on network access, libraries or external data. For these reasons a 22% on that set does not mean the model is useful for 22% of a web application, and failing an authentication or a web extraction does not justify concluding that a model cannot develop an application. A low score on a public benchmark does not prove that a model is of little use; it measures something else.

A real software engineering task, with compilation and a test suite, already separated the models better than those exams, and showed another point: speed is not enough.

Legend: ten local models were evaluated on one task, six are shown; one pass per model; tok/s as recorded by the inference server. IT Lab Bench, edition 1 (pre-bench phase).

A model generating at 74.49 tok/s passed 2 of 8 tests. Two design decisions follow from these observations. The first is to measure the work the organization wants to automate, not an abstract score. The second is to change the metric: instead of the percentage of problems solved, the design starts from four criteria, namely the percentage of real tasks completed without manual intervention, the total time to the correct solution, later corrections and regressions introduced, and the ability to continue the work in a second iteration. The metrics section details what the bench applies today.

2. Method

Test families

Each family corresponds to a capability that a small or mid-sized company uses or would like to use. IT Lab publishes what capability it measures, why it represents real work and the general size of the test; the content of the cases is not published.

Legend: size and criterion as declared in each family report. IT Lab Bench, edition 1.

Edition 1 evaluated 14 models in the office families: ten local and four paid, under the same protocol for both groups. A model lacking the required capability, for example vision, is omitted from that family and is not penalized.

Deterministic judge and staged judge

Passing is decided by a fixed automatic judge: deterministic programs, with no model evaluating another. Valid answers are admitted through acceptance lists defined before the run, and the ground truth of each case is fixed by measuring the material, not by transcribing it from memory.

In the code families the judge advances through chained stages and stops at the first one that fails. The stages are generic and are listed in execution order:

1. Compiles. The delivered code builds without errors.
2. Behavior. The core component behaves as the brief requires.
3. Internal tests. The component's own tests pass.
4. Web layer. The application layer and its tests pass.
5. End to end. A real call crosses both languages.

The stages vary by exercise; the existing-base exercise has six and the from-scratch exercise has five. This design turns the result into a diagnosis: delivering no files, failing to compile, compiling without meeting the contract and failing only at integration are different situations, and each model's report shows where it stopped. The tests are blind to the model. Editing them, or delivering files the exercise does not ask for, invalidates the run. An output truncated by a generation limit, or reasoning that consumes the entire output budget, is not partially penalized: it is recorded as no delivery.

The instrument is not modified to favor a model. A correction to the judge or to the format is admitted only if it holds for every model and is applied equally to all runs; otherwise it is a patch and it is not made. Reports are written from the run data, and conclusions are not drafted in advance.

Threshold set in advance and a declared single pass

Each family sets its criterion before the run. In agentic work the pass threshold is 80 out of 100. A model meets the criterion or does not; the bench does not use expressions such as winning or failing, and does not rank on a difference of a few points.

The official pass is a single one per model and case, unless the family report declares another number. The bench states this in every table: a small difference obtained in a single pass is noise, and it does not establish that one model is superior to another. Where a family includes repetitions, they are declared in its report and are not averaged with the official pass of another family.

Best opportunity for each model

The protocol requires researching each model's documentation and using its best parameters and its best way of being instructed, including its reasoning scale, so that each model solves each case as well as it can. There is no universal configuration. The reasoning scale differs between model families: some distinguish several levels, others only on and off, and in at least one family requesting reasoning disabled does not guarantee that the server stops reasoning. Configuration is therefore documented per model, and the specific values are reserved.

The principle has two functions. It is a criterion of fair evaluation, because a generic configuration attributes to the model defects that belong to the setup. And it is a practical finding: in the bench, configuration changes the result as much as the choice of model. The same work served to refine the tests and remove ambiguities in the briefs. The consequence for comparability is that results from campaigns with different judges or contracts are not equated with one another, hence the use of numbered editions.

Official pass, probes and diagnostics

The bench distinguishes three kinds of run and reports them separately, in labeled paragraphs or tables.

• Official pass. Strict format, one pass per model, sequential single-session execution, with the maximum context the model supports and the same protocol for local and paid models. It is the only run that determines whether a model meets the criterion.
• Probe. A short run that answers a design question about the instrument, for example whether a failure belongs to the delivery format or to the content, or how to order images relative to their questions. A probe does not produce per-model results.
• Diagnostic. A repeat with a single variable changed, such as enabling reasoning, to measure how much it corrects and what it costs in tokens and time. Its result is not mixed with that of the official pass.

Targeted retries are not mixed with the official pass either: the retry variant described in the introduction was reported apart for that reason. In addition, reasoning disabled is not homogeneous across model families, and equivalent levels are not compared between them.

Quality controls on the instrument

A poorly calibrated instrument can discard viable configurations or approve ones that are not. The current protocol applies the following controls, described as functions and not as components.

Deterministic format repair, separate from the model. A fixed chain corrects formatting defects in responses, such as stray quotation marks, trailing commas or control characters, without interpreting their content and without ever asking the model to correct itself. An output cut off by a limit cannot be repaired: the output budget is raised. The control exists because part of the delivery failures belong to the wrapper and not to the model: in one documented case, a model failed to process its format in both runs and the judge passed it completely when it evaluated the same content without the wrapper.

Each image next to its questions. In vision tests the official format interleaves each image with the questions that belong to it. Placing all attachments at the start, with labels, mixed up the association between image and question in two models; the interleaved layout is the official one.

Prebuilt real context in a single piece. Long-context tests use complete real documents, assembled before the run and recorded with their token count by tokenizer, not an artificial fill in increments of 2,000 tokens. The bench has five lengths, from 79,964 to 247,581 tokens.

Measuring the context that fits in memory. If a model does not fit with the maximum context under the card's memory limit, the maximum context it supports is measured with a load probe and a bisection search over memory consumption, instead of excluding it. Two models were thus assigned 131,072 and 200,704 tokens under a limit of 15.4 GiB. The decision is documented in the test and is not taken silently.

GPU memory control. The margin is calibrated against the real state of the machine: a minimum of 100 MB of headroom over that reference state is required, instead of a fixed absolute limit of 15.5 GiB. A wrongly set limit silently discards configurations that do fit.

Speed with speculative decoding. Speed is measured on a new task and the acceptance rate of the draft is declared, because repeated content inflates the figure. In the bench runs the rate was 0.57 to 0.67. This technique also performs differently depending on sampling, so figures obtained with different sampling are not mixed in one chart.

Metrics

The central metric is the task completed without human intervention: a run passes or not according to the family's criterion, not according to how convincing the answer looks. The bench also records:

• Time. Wall-clock seconds of the run; for the office families it is reported in each table.
• Speed. Tokens per second for generation and for reading the context, as recorded by the inference server.
• Memory. Peak card memory during the run, as recorded by the server, set against the limit of the configuration cited.

Equipment and execution

The equipment, specified and built by IT Lab specifically for testing, is described in the measurement sheet.

Local models are quantized to 3 or 4 bits, depending on the model, and run one at a time: two models at once on the same card corrupt the measurement and breach the memory control. Paid models are evaluated with the same judge and the same delivery format.

Publication policy and editions

IT Lab publishes what is measured, what the result is for, results per model, times, memory and failure patterns in general terms. It reserves the task statements, the scoring rubrics, the internal weighting of scores, the correct answers, the content of test documents and images, and the fine configuration of each model, because they are accumulated knowledge it places at the service of its clients.

Every relevant new model is run through the same bench, with the same judge and the same criterion, and is published with the edition number, so results within one edition are comparable with one another. If the instrument changes, results from different editions are not mixed without saying so. The bench results table is updated with each model added.

Continues in Part 2.

Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.