Why AI benchmarks miss real work
High scores on HumanEval and BigCodeBench did not rank local models by usefulness; IT Lab therefore measures tasks completed without human intervention.
Public coding benchmarks produce high, comparable scores, but in IT Lab's measurements they did not rank local models by usefulness on real tasks. This is why the IT Lab Bench measures tasks completed without human intervention.
- A model with a high HumanEval score dropped once HumanEval+ added its hidden cases; the solution worked, but it was fragile.
- BigCodeBench Hard asks for isolated functions on topics far from a small company's work, and its result depends on the environment.
- A low score on a public benchmark does not prove a model is of little use: it measures something else.
The business question
A Chilean company evaluating local AI is not looking for a universal ranking. It needs to know which model to keep on a normal-sized computer with a single 16 GiB graphics card, for software and document work. Public benchmarks are the usual starting point: they are downloaded, run, and return a percentage. The question is whether that percentage translates into a work decision.
IT Lab ran those tests on local models before defining its own bench. In IT Lab's judgment, the outcome was clear: the scores were high and did not indicate which model to keep.
What each traditional test measures
HumanEval is a set of 164 short programming problems, each with a small set of tests. The reported figure is pass@1: the percentage of problems solved on the first attempt. HumanEval+ uses the same problems with extended tests that add cases the base version does not cover. BigCodeBench Hard is a set of 148 tasks that ask for an isolated function and rely on libraries for scientific plotting, machine learning, time series, web data extraction, spreadsheets and compressed archives, system commands and network connections, cryptography, text recognition in images, and language processing.
These are legitimate exams, useful for comparing models with one another on those skills. The limit appears when they are used to decide what to automate in a company.
First signal: the base version of the exam is the easy one
In IT Lab's run, with one sample per problem and a direct protocol, four local models obtained these results.
| Model | HumanEval | HumanEval+ | Drop |
|---|---|---|---|
| Qwen3.8-27B-GSQ-RCO (direct pass) | 93.3% | 89.0% | 4.3 points |
| Qwen2.5-Coder 14B | 89.6% | 85.4% | 4.2 points |
| DeepSeek-Coder-V2 Lite 16B | 82.9% | 76.2% | 6.7 points |
| gpt-oss 20B | 54.3% | 53.0% | 1.3 points |
One pass per problem (164 problems), measured by IT Lab; the score is the percentage of problems solved and there is no pass threshold.
When the hidden cases are added, the scores fall. A model with a good base-version score ends up lower on the complete version: its solution works on the known cases and fails on others. For a company, that pattern is equivalent to an automation that works in the demonstration and fails on real data.
A variant with targeted retries reached 97.6% and 93.3% on the two versions. IT Lab records it as a diagnostic and does not compare it with the table: a clean pass includes no second chances.
Second signal: the score does not follow the task
Before the public benchmarks, IT Lab compared ten local models on a real software engineering task, with compilation and a test suite. The table also recorded generation speed.
| Model | Tests passed | Generation (tok/s) |
|---|---|---|
| gpt-oss 20B | 8 of 8 | 101.08 |
| Granite 4.2 8B | 2 of 8 | 74.49 |
| Qwen2.5-Coder 14B | 6 of 7 | 50.39 |
| DeepSeek-Coder-V2 Lite 16B | 6 of 6 | 48.96 |
| Devstral Small 2 24B | 0 of 12 | 37.09 |
One pass per model, measured by IT Lab on a single task; speed as recorded by the inference server.
Two readings, in IT Lab's judgment. Speed alone does not decide: a model generating at 74.49 tok/s passed 2 of 8 tests. And the order on this task does not match the order on HumanEval: gpt-oss 20B passed all 8 tests of the task and scored 54.3% on HumanEval, while Qwen2.5-Coder 14B passed 6 of 7 and scored 89.6%. Each measurement answers a different question, and only one resembles the work that was to be delegated. With a single pass, small differences between models are not read as superiority.
Third signal: the exam measures other work
The problem with BigCodeBench Hard is not the score but what the set measures.
- Other work. A score on scientific plotting, machine learning or cryptography tasks does not indicate what fraction of a web application, a form or a database a model can build.
- Isolated functions, not applications. The set asks for standalone units. A small company's work requires a complete application with repository context.
- The environment contaminates the result. Many tasks depend on network access, libraries, external data or artificial requirements. Failing an authentication or the reading of a web table does not support the conclusion that the model cannot develop an application.
The decision: measure completed tasks
On the basis of that evidence, IT Lab set a priority: not the best abstract score, but a dependable partner for real projects. The metric moved from "what percentage of problems pass" to:
- percentage of real tasks completed without manual intervention;
- total time to the correct solution, not only tokens per second;
- later corrections and regressions introduced;
- ability to continue the work in a second iteration.
That definition gave rise to the IT Lab Bench: tests that represent recognizable office and development tasks, passed or failed by fixed automatic judges, with the best possible configuration for each model. The full method is described in How IT Lab evaluates a local AI model and in Technical report 4: IT Lab Bench methodology.
What this means for a Chilean company
In IT Lab's judgment, a public benchmark is useful for ruling models out and for comparing narrow skills, not for deciding which task to delegate. Before adopting local AI, a company should require the evaluation to state which task was measured, how it was passed and on what hardware, and should be wary of a high score presented without those details. A low score does not rule a model out either: it only reports that something else was measured.
Frequently asked questions
What is an AI benchmark and what is it for? A public benchmark gives comparable scores on bounded problems, for example in code. In IT Lab's measurements, those scores did not rank the local models by their usefulness on real tasks.
Why does a high score not guarantee the model will work in my company? A model with a high HumanEval score dropped when the hidden HumanEval+ cases were added: the solution worked, but it was fragile.
How does the IT Lab Bench differ from a public benchmark? The IT Lab Bench measures tasks completed without human intervention, not isolated functions.
Does a low benchmark score mean the model is of little use? No: a low score on a public benchmark does not show that a model is of little use, because it measures something else.
Limitations and scope
- A single pass per model and problem; small differences are noise.
- The targeted-retry variant is a diagnostic and is not compared with the direct pass.
- The engineering task in the second table is a single task; its model order is not a general ranking.
- The public benchmarks were run with IT Lab's direct protocol; other protocols may give other figures.
- The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
- An IT Lab measurement on a single machine; not a purchase recommendation or a certification.
Measurement sheet
- Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
- Quantization: 3 or 4 bits, depending on the model.
- Passes: one per problem or per model.
- IT Lab Bench, edition 1.
Related technical report: Technical report 4: IT Lab Bench methodology
Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk
The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.
Do you need someone to take care of your technology?
Tell us what you need to organize, protect or solve. The first conversation has no cost: we assess whether our model fits you and what the next steps would be.
Let's talk about your business