IT Lab Bench results, edition 1
Per-model results by test family from the IT Lab Bench, edition 1: pass criterion and time, with a summary matrix.
This page gathers in one place the per-model results of each test family in the IT Lab Bench: the result and how long it takes.
- There is one table per family, with its legend of passes and criterion, and a link to the article that explains it.
- A summary matrix crosses models and families; it marks only what the bench records explicitly.
- The table is updated with every relevant new model, which is run through the same bench and published with its edition number.
What this table is
It is the public record of the results of the IT Lab Bench, edition 1. Each row is a model in a test family; each result corresponds to the official pass. Probes and diagnostics do not appear here: they are reported, labeled, in each family's article.
When a new model is added, it is entered in the tables of the families where it was evaluated and in the matrix, together with the bench edition in which it was measured. Results from different editions are not mixed without labels.
Conventions: "Local" runs on IT Lab's machine; "Paid" is a commercial service. The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost. A dash (—) means the bench has no data for that cell. Every conclusion is IT Lab's judgment based on its measurements.
Document reading
Extracting a real document into structured data and answering closed questions about it. Article: Documents: extract yes, count no.
| Model | Type | Result (score) | Time | Ed. |
|---|---|---|---|---|
| muse-spark-1.3 | Paid | 100 | 24 s | 1 |
| claude-haiku-4-5 | Paid | 100 | 106 s | 1 |
| mimo-v2.6-flash | Paid | 93 | 17 s | 1 |
| gpt-6-luna | Paid | 93 | 47 s | 1 |
| Qwen3.8-27B-GSQ-RCO | Local | 93 | 68 s | 1 |
| Gemma-4-26B | Local | 93 | 24 s | 1 |
| Qwen3-Coder-30B, Qwopus3.6-35B, Gemma-4-E4B | Local | 90 | 28-33 s | 1 |
| Qwen2.5-Coder-14B, Ministral-3-14B, Gemma-4-12B | Local | 87 | 51-74 s | 1 |
| DeepSeek-Coder-V2-Lite | Local | 76 | 18 s | 1 |
| GPT-OSS-20B | Local | 0 (not parseable) | 96 s | 1 |
Legend: a single pass per model; this family is reported by score. Only two models, both paid, obtained the full score (100); no local model did. In grouped rows, the time is the range for the group. GPU memory ceiling for the family: 15.4 GiB.
Vision
Interpreting 11 images with 31 closed questions. Article: Vision: images and scanned documents.
| Model | Type | Result (score) | Time | Ed. |
|---|---|---|---|---|
| gpt-6-luna | Paid | 100 | 9.2 s | 1 |
| muse-spark-1.3 | Paid | 97 | 38.1 s | 1 |
| claude-haiku-4-5 | Paid | 97 | 19.2 s | 1 |
| Ministral-3-14B | Local | 97 | 14.2 s | 1 |
| Qwen3.8-27B-GSQ-RCO | Local | 97 | 23.7 s | 1 |
| Gemma-4-E4B | Local | 87 | 7.0 s | 1 |
Legend: a single pass per model; score = percentage of correct answers; this family is reported by score. The table includes the six models able to process images; one paid model attached the image but stated it could not see it and was excluded on that statement, not on its score. GPU memory ceiling for the family: 15.4 GiB.
Agentic work
Three episodes with tools and actions. Article: Agentic work.
| Model | Type | Result (score) | Time | Ed. |
|---|---|---|---|---|
| Qwen3.8-27B-GSQ-RCO | Local | 100 | 20.6 s | 1 |
| claude-haiku-4-5 | Paid | 99 | 60.3 s | 1 |
| Qwopus3.6-35B-A3B | Local | 97 | 10.5 s | 1 |
| muse-spark-1.3 | Paid | 96 | 222.6 s | 1 |
| Qwen3-Coder-30B | Local | 95 | 5.1 s | 1 |
| gpt-6-luna | Paid | 76 | 46.5 s | 1 |
| Gemma-4-26B-A4B | Local | 76 | 8.1 s | 1 |
| mimo-v2.6-flash | Paid | 74 | 84.6 s | 1 |
| Gemma-4-12B | Local | 74 | 17.6 s | 1 |
| Qwen2.5-Coder-14B | Local | 73 | 12.9 s | 1 |
| Gemma-4-E4B | Local | 72 | 6.5 s | 1 |
| Ministral-3-14B | Local | 49 | 12.6 s | 1 |
| DeepSeek-Coder-V2-Lite | Local | 42 | 9.5 s | 1 |
| GPT-OSS-20B | Local | failed (5) | — | 1 |
Legend: a single pass per model; pass threshold: 80. Five of fourteen models meet the criterion. The reasoning diagnostic for this family is not an official pass and is reported separately, in the article. GPU memory ceiling for the family: 15.4 GiB.
Code on an existing base
Completing a base with marked gaps (a Rust engine and a Ruby on Rails API), with a judge of six chained steps that stops at the first one that fails. Article: Complete vs create.
| Model | Type | Result (steps passed) | Meets criterion | Time | Ed. |
|---|---|---|---|---|---|
| Gemma-4-26B-A4B | Local | 6 / 6 | Yes | 16.8 s | 1 |
| Qwen3.8-27B-GSQ-RCO | Local | 6 / 6 | Yes | 43.5 s | 1 |
| Qwen2.5-Coder-14B | Local | 6 / 6 | Yes | 29.8 s | 1 |
| Qwopus3.6-35B-A3B | Local | 5 / 6 (stops at the web tests) | No | 17.3 s | 1 |
| Gemma-4-12B | Local | 5 / 6 (stops at the web tests) | No | 39.9 s | 1 |
| Qwen3-Coder-30B | Local | 2 / 6 (stops at the engine contract) | No | 11.6 s | 1 |
| Gemma-4-E4B | Local | 1 / 6 (does not compile) | No | 24.9 s | 1 |
| Ministral-3-14B | Local | 1 / 6 (does not compile) | No | 33.7 s | 1 |
| GPT-OSS-20B | Local | 0 / 6 (no files; output budget exhausted) | No | 92.6 s | 1 |
| DeepSeek-Coder-V2-Lite | Local | 0 / 6 (no files) | No | 6.4 s | 1 |
Legend: a single pass per model; threshold: all six steps. Paid models: —, not evaluated in this family. A trial in team mode produced no conclusive result. GPU memory ceiling for the family: 15.5 GiB.
Code from scratch
Creating an application in two languages with no scaffolding, with a judge of five chained steps. Article: Complete vs create.
| Model | Type | Result (steps passed) | Meets criterion | Time | Ed. |
|---|---|---|---|---|---|
| Qwen2.5-Coder-14B | Local | 2 / 5 (compiles; breaks the engine contract) | No | 9.6 s | 1 |
| Qwen3-Coder-30B, Qwopus3.6-35B, Gemma-4-26B, Qwen3.8-27B-GSQ-RCO, Gemma-4-E4B, Ministral-3-14B | Local | 1 / 5 (does not compile) | No | 20.5-66.5 s | 1 |
| DeepSeek-Coder-V2-Lite, GPT-OSS-20B, Gemma-4-12B | Local | 0 / 5 (leaves no files; two with output budget exhausted) | No | 16.7-395.6 s | 1 |
| gpt-6-luna (reasoning disabled / low) | Paid | 5 / 5 in both configurations | Yes | 49.8 s / 42.5 s | 1 |
| muse-spark-1.3 (reasoning disabled / low) | Paid | 5 / 5 in both configurations | Yes | 144.7 s / 15.9 s | 1 |
| claude-haiku-4-5 | Paid | One pass 5 / 5 and two failures | One pass yes, two no | 186.2 s (passing run); 947.9 s (failure) | 1 |
| mimo-v2.6-flash (reasoning disabled / low) | Paid | 0 / 5, due to delivery format | No | 320.5 s / 12.1 s | 1 |
Legend: a single pass per model and configuration unless stated; threshold: all five steps. In grouped rows, the time is the range for the group. No local model meets the criterion in this family. "Reasoning disabled" is not homogeneous across model families and is not compared between them.
Applied reasoning
40 Chilean small-business cases scored with automatic controls. Article: Reasoning on Chilean business cases.
| Model | Type | Result (passed of 40) | Meets criterion | Time | Average speed | Ed. |
|---|---|---|---|---|---|---|
| Muse-Glimmer-30B | Local | 30 (30/37 without vision) | No | — | ~40 tok/s | 1 |
| muse-spark-1.3 | Paid | 28 | No | — | — | 1 |
| Swift 1.5 | Local | 26 | No | — | 52.3 tok/s | 1 |
| GPT-OSS-20B | Local | 24 | No | — | 140.8 tok/s | 1 |
| Qwen3.8-27B-GSQ-RCO | Local | 24 | No | — | 48.2 tok/s | 1 |
| Gemma-4-12B | Local | 21 (context trimmed by the memory ceiling) | No | — | 47.3 tok/s | 1 |
| gpt-6-luna | Paid | 18 | No | — | — | 1 |
| DeepSeek-Coder-V2-Lite | Local | 9 | No | — | 121.9 tok/s | 1 |
Legend: one full campaign per row. "Passed" means automatic controls passed; some areas with expert review in progress carry a preliminary score. No model passes the full 40-case exam. Reasoning: a case-by-case ladder for Muse-Glimmer-30B, low for the paid models, and not recorded case by case in the local campaigns. Paid models have no comparable speed, so their cell is empty rather than zero. GPU memory ceiling for the family: 15.4 GiB.
Realistic programming
Repositories of 10 to 15 files with interacting faults. Article: Hard programming in real repositories.
| Model | Type | Result (score /10) | Meets criterion | Reasoning level needed | Average speed | Ed. |
|---|---|---|---|---|---|---|
| Qwen3.8-27B-GSQ-RCO | Local | 10 | Yes | 8 at low + 2 at medium | 40.6 tok/s | 1 |
| Muse-Glimmer-30B | Local | 10 | Yes | 8 at low + 2 at high | 38.7 tok/s | 1 |
| Swift 1.5 | Local | 9 | — | 9 at low + 1 at maximum | 41.0 tok/s | 1 |
| GPT-OSS-20B | Local | 9 | — | 5 at low + 2 at medium + 3 at high | 87.5 tok/s | 1 |
| Gemma-4-26B-A4B | Local | 7 | — | a single medium level | 115.3 tok/s | 1 |
Legend: two repetitions, ten runs per model. The bench sets no pass threshold for this family; only 10/10 is marked as met, and the rest are left with a dash. With two repetitions, a 9/10 against 10/10 difference does not establish a winner: some cases change outcome between repetitions. Speed does not predict accuracy: the fastest model in the table received the lowest score. The reasoning ladder steps up only on the cases that fail. Time: —, no comparable figure is available. Paid models: —, not evaluated in this family.
Summary matrix
Each cell shows whether the model meets the family criterion, according to the official pass in the tables above; reading and vision are reported by score, so their cells show the score instead of ✓/✗. It is not a global ranking: the families measure different capabilities and their criteria are not comparable with one another.
Legend: ✓ meets the criterion; ✗ does not meet it; a number is the score (reading and vision); ◐ mixed result (one pass met and two not); — not evaluated, or no explicit data in the bench.
| Model | Type | Reading | Vision | Agentic | Code: existing base | Code: from scratch | Reasoning | Programming |
|---|---|---|---|---|---|---|---|---|
| muse-spark-1.3 | Paid | 100 | 97 | ✓ | — | ✓ | ✗ | — |
| claude-haiku-4-5 | Paid | 100 | 97 | ✓ | — | ◐ | — | — |
| gpt-6-luna | Paid | 93 | 100 | ✗ | — | ✓ | ✗ | — |
| mimo-v2.6-flash | Paid | 93 | — | ✗ | — | ✗ | — | — |
| Qwen3.8-27B-GSQ-RCO | Local | 93 | 97 | — | ✓ | ✗ | — | — |
| Qwen3.8-27B-GSQ-RCO | Local | — | — | ✓ | — | — | — | — |
| Qwen3.8-27B-GSQ-RCO | Local | — | — | — | — | — | ✗ | ✓ |
| Qwopus3.6-35B | Local | 90 | — | — | — | ✗ | — | — |
| Qwopus3.6-35B-A3B | Local | — | — | ✓ | ✗ | — | — | — |
| Qwen3-Coder-30B | Local | 90 | — | ✓ | ✗ | ✗ | — | — |
| Gemma-4-26B | Local | 93 | — | — | — | ✗ | — | — |
| Gemma-4-26B-A4B | Local | — | — | ✗ | ✓ | — | — | — |
| Gemma-4-12B | Local | 87 | — | ✗ | ✗ | ✗ | ✗ | — |
| Gemma-4-E4B | Local | 90 | 87 | ✗ | ✗ | ✗ | — | — |
| Ministral-3-14B | Local | 87 | 97 | ✗ | ✗ | ✗ | — | — |
| Qwen2.5-Coder-14B | Local | 87 | — | ✗ | ✓ | ✗ | — | — |
| DeepSeek-Coder-V2-Lite | Local | 76 | — | ✗ | ✗ | ✗ | ✗ | — |
| GPT-OSS-20B | Local | 0 | — | ✗ | ✗ | ✗ | ✗ | — |
| Swift 1.5 | Local | — | — | — | — | — | ✗ | — |
| Muse-Glimmer-30B | Local | — | — | — | — | — | ✗ | ✓ |
In realistic programming, only a 10/10 score is marked ✓, the only case in which the bench states that the model passes.
Frequently asked questions
What is this results table? It is the public record of the IT Lab Bench results, edition 1. Each row is a model in a test family, and each result comes from the official pass.
Is it updated with new models? Yes: each relevant new model goes through the same bench and is published with its edition number.
What does a dash in a cell mean? A dash (—) means the bench has no data for that cell.
Method note and limitations
The IT Lab Bench measures by task completed without human intervention, using fixed automatic judges and the best possible configuration for each model. The full method is in How IT Lab evaluates a local AI model.
- A single pass per model and case unless the table legend states another number; small differences are noise.
- Each table reports only the official pass; probes and diagnostics are reported separately, in each family's article.
- Thresholds and scales are specific to each family: scores are not compared across tables.
- Paid models are measured per service.
- Rows with "—" are not a negative result: they mean the bench has no data for that cell or the model was not evaluated in that family.
- An IT Lab measurement on a single machine; not a purchase recommendation or a certification.
Measurement sheet
- Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
- Quantization: 3 or 4 bits, depending on the model.
- Passes: per family, stated in each legend.
- IT Lab Bench, edition 1.
Related technical report: Technical report 4: IT Lab Bench methodology
Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk
The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.
Do you need someone to take care of your technology?
Tell us what you need to organize, protect or solve. The first conversation has no cost: we assess whether our model fits you and what the next steps would be.
Let's talk about your business