Hard programming in real repositories with local AI

On repositories of 10 to 15 files with two interacting faults, several local models pass 9 or 10 of 10 in the IT Lab Bench. The differences among them fall within noise, and generation speed does not predict accuracy.

With two repeats, four of the five models measured score 9 or 10 of 10; the gap between 9 and 10 does not establish an order.
The fastest model (115.3 tok/s) scores 7 of 10; the models that reach 10 of 10 generate at about 40 tok/s.
In IT Lab's judgment, maintaining existing code is a task that a local model on a 16 GiB card can already take on, with human review.

The business question

A small or mid-sized company's development team spends much of its week maintaining code that already exists, not writing new applications. The question is whether a model running inside the company, without sending the code to a third party, can locate and fix faults in a repository of realistic size. A short, isolated exercise does not answer that; this test family therefore uses complete repositories.

What the test measures and why it represents real work

The realistic programming family measures whether a model solves, unaided, repositories of 10 to 15 files with interacting faults, in several languages.

This is recognizable maintenance work: understanding a set of files, finding more than one cause, fixing all of them and not breaking what works. The content of the cases is not published. The delivery format is fixed: each modified file is returned in full, labeled with its path.

How a model passes

A fixed automatic judge decides each case: deterministic programs, with no model evaluating another. The judge advances in stages and stops at the first one that fails, so the result shows where the work broke.

The protocol also applies a reasoning ladder. Each case starts at the lowest reasoning level the model offers and moves up to the next only if that case failed; cases already passed are not repeated at higher levels. The rule separates two questions: how many cases a model solves, and how much reasoning, time and tokens it needs to solve them.

Results

Ten cases per model: five repositories, two repeats with a fixed seed. Pass threshold: the staged judge described above, case by case. IT Lab Bench, edition 1. Speed: median of the second repeat, as recorded by the inference server.

Model	Cases passed (of 10)	Reasoning levels used	Speed (tok/s)
Qwen3.8-27B-GSQ-RCO	10	8 at low, 2 at medium	40.6
Muse-Glimmer-30B	10	8 at low, 2 at high	38.7
Swift 1.5	9	9 at low, 1 at the maximum level	41.0
gpt-oss 20B	9	5 at low, 2 at medium, 3 at high	87.5
Gemma-4-26B-A4B	7	a single level (medium)	115.3

Qwen3.8-27B-GSQ-RCO and Muse-Glimmer-30B tie at 10 of 10, and Swift 1.5 and gpt-oss 20B at 9 of 10. With two repeats, several cases change outcome from one to the other, so 9 against 10 is a difference within noise and IT Lab does not name a leader among them.

Swift 1.5 is a separate model and is reported separately.

Speed does not predict accuracy

Gemma-4-26B-A4B generates at 115.3 tok/s, the highest speed in the table, and scores 7 of 10: its result repeats in both repetitions. The models that reach 10 of 10 generate at about 40 tok/s. gpt-oss 20B, also fast, writes between 18,000 and 41,000 tokens per attempt on the hard cases; more tokens did not mean a better solution, only more effort.

The relevant metric for a company is the time to a correct solution and the number of retries it takes, not tokens per second. Total run times are not published in this edition.

The ladder as an operating rule

The levels column of the table shows why the score alone is not enough. Two models with the same score can differ in how much reasoning they consume: one solves almost everything at the low level with short outputs; another needs the high level in more cases. Swift 1.5 reaches 9 of 10 because it needed the maximum level in one case.

For a company, the rule translates as follows: start with low reasoning and move up only when the case demands it. More reasoning corrects failures, but it costs tokens and GPU minutes; using it always multiplies cost without need.

Three classes of failure

The bench classifies each case that does not pass by whose cause it is:

Model error. The model delivers incorrect or incomplete code: for example, it fixes some of the faults and leaves another untouched.
Instrument defect. The case or the judge was miscalibrated. These cases are corrected and rejudged; a result contaminated by the instrument is not used as a conclusion about quality.
Format, limit or transport. The code is correct, but the delivery format is not, or the run hits a limit. When the extracted code passes the tests and the format fails, the bench reports both results separately instead of manufacturing a single figure.

This separation avoids discarding a model over a problem that belonged to the measurement, and lets a company tell a failure of understanding from a failure of presentation.

What this means for a Chilean company

What to automate. In IT Lab's judgment, fixing faults in existing code is a task the local models evaluated handle with high results in this test.
What to review. Every change a model proposes should go through the project's own build and tests, and the format of the response should be validated before it is applied.
What hardware. The measurements correspond to a 16 GiB card, with 3-bit or 4-bit quantization.
How to decide. Measure time to a correct solution and retries, and fix the rule of raising reasoning only when the case demands it.

Frequently asked questions

Can local AI fix faults in a small company's code? In repositories of 10 to 15 files with two interacting faults, several local models score 9 or 10 of 10 in the IT Lab Bench.

Does a faster model code better? No: the fastest model (115.3 tok/s) scores 7 of 10, and those that reach 10 of 10 generate at about 40 tok/s.

What hardware is needed, and is review required? In IT Lab's judgment, maintaining existing code is a task a local model on a 16 GiB card can already take on, with human review.

Limitations and scope

Two repeats per model and case; 9 against 10 is a difference within noise and does not establish an order.
The official pass is kept separate from probes and diagnostics.
GPU memory cap in normal campaigns: 15.5 GiB.
"Reasoning disabled" is not homogeneous across model families: those levels are not compared between families.
The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

Measurement sheet

Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
Quantization: 3 or 4 bits, depending on the model.
GPU memory cap: 15.5 GiB in normal campaigns.
Passes: two repeats with a fixed seed.
IT Lab Bench, edition 1.

Related technical report: Technical report 2: Code

Full method: How IT Lab evaluates a local AI model

Contact: IT Lab evaluates local AI models on the code and data of Chilean organizations. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.