How IT Lab evaluates a local AI model

The IT Lab Bench is IT Lab's own local AI benchmark for businesses: it measures whether a local AI model finishes office or development work, using automatic judges that do not change between runs and the best possible configuration for each model. This piece describes the method and is the entry point to the rest of the publication.

Each test represents a recognizable task and is passed or failed by a deterministic program, with no model grading another.
The protocol gives every model its best opportunity; in the bench, configuration changes the result as much as the choice of model.
The official pass is reported apart from probes and diagnostics, and the instrument itself is subject to quality controls.

The business question

A company evaluating local AI needs to know which task it can automate, which one a person must review, and which hardware makes it possible. Traditional public benchmarks measure short, isolated problems, and their high scores do not rank models by usefulness in everyday work. The IT Lab Bench turns the evidence into decisions: what can be delegated, and with what margin of error.

What the bench measures and why it represents real work

Each test family corresponds to a capability that a small or mid-sized company uses or would like to use: reading a document, interpreting an image, completing a task with tools, reasoning about a case, modifying or writing code. Results are measured by task completed without human intervention, not by how convincing the answer looks.

IT Lab publishes what capability each test measures and what the result is for; the content of the tests is not published.

How a model passes

Passing is decided by a fixed automatic judge: deterministic programs, with no model evaluating another. The threshold is set before the run, and the instrument is never modified to favor a model. In code tests the judge advances in stages and stops at the first one that fails, so the result shows where the work broke, not only that it did.

When a test has a single pass per model, the bench says so. A small difference in a single pass is treated as noise and does not establish that one model is superior.

The best opportunity for every model

The protocol requires researching each model's documentation and using its best parameters and its best way of being instructed, so that each model solves each case as well as it can. This is both a criterion of fair evaluation and a practical finding: in the bench, configuration changes the result as much as the choice of model.

With the same model and the same machine, an optimized inference-engine configuration (quantized attention cache and speculative decoding) can be faster and use less memory than a generic one. A generic configuration would have attributed to the model a slowness that belonged to the setup. That work also served to refine the tests and remove ambiguities.

Official pass, probes and diagnostics

The official pass is the one that decides whether a model meets the criterion. Probes and diagnostics, for example a repeat run with reasoning enabled to measure how much it corrects and what it costs in tokens and time, are reported in labeled paragraphs or tables and never mixed with the official pass. "Reasoning disabled" is not homogeneous across model families and is not compared between them.

Quality controls on the instrument

A poorly calibrated instrument can discard viable configurations or approve ones that are not; the protocol therefore subjects it to controls.

Deterministic format repair, separate from the model. It fixes formatting defects in responses without interpreting their content. An output cut off by a limit is not repaired: the output budget is raised instead.
Each image next to its questions. In vision tests the official format interleaves each image with its own questions, to avoid association mix-ups.
Prebuilt real context in a single piece. Long-context tests use complete real documents with a token count, not an artificial fill in increments.
Measuring the context that fits in memory. If a model does not fit under the card's limit, the maximum context it supports is measured instead of excluding it, and documented.
GPU memory control. The memory margin is calibrated against the real state of the machine, so that configurations are not discarded because of a wrongly set limit.
Speed with speculative decoding. It is measured on a new task and the acceptance rate is declared, since repeated content can inflate the figure.

What the bench produces

Each family has its own article, with official-pass tables, failure patterns and limitations:

Document reading. Extracting data from documents and answering about them; separates what can be automated from what needs review. Documents: extract yes, count no
Vision. Interpreting images and scanned documents through closed questions. Vision: images and scanned documents
Agentic work. Finding an answer using tools and finishing the job. Agentic work
Code on an existing base and from scratch. Completing a base with marked gaps versus creating an application from scratch. Complete vs create
Applied reasoning. Chilean small-business cases scored with automatic controls. Reasoning on Chilean business cases
Realistic programming. Repositories with interacting faults, solved unaided. Hard programming in real repositories

Two articles cover what surrounds the model: configuring a local model matters as much as choosing it and which small-business hardware suffices for local AI with 16 GB.

Publication policy

IT Lab publishes what is measured, what the result is for, and the results per model. It reserves the task statements, the scoring rubrics and the fine configuration, because they are accumulated knowledge it places at the service of its clients. Every relevant new model is run through the same bench and published with the edition number, so results remain comparable with one another.

What this means for a Chilean company

In IT Lab's judgment, a result without its method is not enough to decide on. Before adopting local AI, a company should require the evaluation to state which task was measured, how it was passed, with what configuration and on what hardware.

Frequently asked questions

What is local AI? An artificial intelligence model that runs on the organization's own machines, so documents are not sent to an outside provider.

How is a pass decided? A fixed automatic judge made of deterministic programs applies a criterion defined before the run; no model grades another.

Why does each model get its best configuration? Because configuration changes the result as much as the choice of model; evaluating with generic settings would blame the model for defects of the setup.

Are the test questions published? No. IT Lab publishes what is measured and what the result is for; it reserves the prompts and rubrics.

Are the results a buying recommendation? No. They are laboratory measurements on a single machine.

Limitations and scope

A single pass per model and case unless another number is stated; small differences are noise.
The official pass is kept separate from probes and diagnostics.
The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

Measurement sheet

Equipment: a machine IT Lab specified and built specifically for testing, with a 16 GiB graphics card chosen as the most affordable good cost-benefit option for a small company. Details in SMB hardware for local AI with 16 GB.
Quantization: 3 or 4 bits, depending on the model.
Passes: per family, stated in each article.
IT Lab Bench, edition 1.

Related technical report: Technical report 4: IT Lab Bench methodology

Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.