Technical report 2: Local AI for writing code

Code: ITL-IT-03 · IT Lab Bench, edition 1

Summary

This report brings together the three code test families of the IT Lab Bench: code on an existing base, code from scratch and realistic programming. The first two use an application made of an engine in Rust and a web API in Ruby on Rails, and an automatic staged judge that stops at the first gate that fails. The third uses repositories of 10 to 15 files with interacting faults, and is measured over two repetitions.

On a base with marked gaps, three of ten local models pass all six stages in 17 to 43 seconds. From scratch, none of the ten local models meets the criterion, while two paid models do. In realistic programming, two local models score 10 of 10 and two others 9 of 10, differences that fall within the noise of two repetitions; generation speed does not predict accuracy. In IT Lab's judgment, the results support using local models to complete and maintain code with review, and requiring mandatory review when the model creates code from scratch.

1. Introduction and question

A company that develops software wants to know whether a model running on its own machine, without sending code to a third party, can take on programming work and under what conditions. The question has at least three distinct forms, and the IT Lab Bench separates them because the answers do not coincide.

The first is completing a base that already exists: a working system with missing pieces and tests already written. This is maintenance and extension work. The second is creating an application from scratch, with no skeleton, where the model must decide the structure as well as write the logic. The third is fixing real faults in mid-sized repositories, where two defects influence each other and some files must not be touched.

The report answers three business questions. First, what share of the evaluated local models completes an existing base, and how fast. Second, what happens when the skeleton is removed, and how that compares with what a paid model achieves. Third, what determines accuracy in realistic repositories: speed, reasoning level or the way the result is delivered. The bench's general method is described in Technical report 4; the site articles Code: completing an existing base vs creating from scratch and Hard programming in real repositories summarize these conclusions for a business reader.

2. Method

Test families

Code on an existing base. Each model receives a system with an engine in Rust and a web API in Ruby on Rails, with the logic of several marked gaps left unimplemented. The model must complete the missing pieces in both languages. The test files are fixed by the lab and cannot be modified; editing them invalidates the run.

Code from scratch. Each model receives only a brief and no skeleton. It must create the files of the engine and of the web bridge, deliver exactly the requested files and leave a system that compiles and passes the tests. Delivering extra files or editing what the lab provides also invalidates the run.

Realistic programming. Repositories of 10 to 15 files with interacting faults, in several languages. This family rests on an earlier set of smaller realistic cases, which serves as a reference and is reported separately. The model delivers each complete file in a fixed format, and the delivery is judged by its executed content, not by its appearance.

Automatic staged judge

In the first two families the judge is a deterministic program, with no model grading another, and it advances through chained stages (compilation, behavior, web-layer tests and end-to-end test) that stop at the first one that fails. On an existing base there are six stages; from scratch, five.

The design turns a failure into a diagnosis. "Delivered no files" differs from "does not compile", which differs from "compiles but breaks the requested behavior" and from "the engine works but the web layer fails". Each model's result is therefore reported as the number of stages passed and the cause of the first failure, not as a single grade. Output cut off by a limit, or reasoning that consumes the entire generation budget, receives no partial penalty: there is simply no delivery.

The staged judge only allows comparison within the same brief and with the same judge. A pass on this system does not mean "passes in Rust" in general.

Pass criterion

The judge fixes the criterion before the run. In the first two families, a model passes if it clears all stages. In realistic programming, each case is passed or not passed according to visible and hidden tests executed deterministically; the score is the number of cases passed out of the total over the repetitions. The instrument is not modified to favor a model.

Best opportunity for each model

The protocol requires researching each model's documentation and using its best parameters and its best form of instruction. In realistic programming, each model's configuration was also reviewed before the second repetition and confirmed as optimal for that model. The same task is given to all models; the difference between forms of instruction is part of what is measured, not a flaw of the bench.

Reasoning ladder

In realistic programming, each case starts at the lowest reasoning level the model offers and moves to the next level only if that case failed. Cases already passed are not repeated at higher levels. The rule turns the vague question of how capable a model is into two measurable ones: how many cases it solves, and how much reasoning, time and tokens it needs to solve them.

Three classes of failure

Each failure is classified by who is responsible. Class (a) is a genuine model error. Class (b) is an instrument defect, in either the case or the judge. Class (c) concerns format, limit or transport. The classification separates a model that writes incorrect code from an instrument that changes the rules during the measurement.

Instrument quality controls

The bench's general controls apply: GPU memory control and explicit declaration of the number of passes. In these tests the card's memory cap was 15.5 GiB.

3. Results

Code on an existing base, official pass

One pass per model. Pass criterion: six stages cleared. IT Lab Bench, edition 1.

Three of ten local models clear all six stages, in a range of 17 to 43 seconds. Two models stop at the web tests, one gate short of passing; one stops at the engine behavior; two do not compile; and two deliver no files, for different reasons. In the case of GPT-OSS-20B, the judge records output exhausted before any file was written, a budget failure and not a comprehension failure. In the case of DeepSeek-Coder-V2-Lite, the absence of files occurs in 6.4 seconds. The staged judge separates the two; a numeric grade would have merged them.

A team-mode trial, with several coordinated agents, was run with only two local models, since the rest of the batch did not reach delivery. Neither passed: one stopped at the engine behavior and the other did not compile. The trial is not conclusive, because it has one pass per model.

Code from scratch, official pass: local models

One pass per model and configuration. Pass criterion: five stages cleared. IT Lab Bench, edition 1.

None of the ten local models meets the criterion. The most advanced compiles but breaks the behavior requested of the engine: the code exists and does not do what was asked. Among those that leave no files, the one with the longest wall-clock time, 395.6 seconds, clears no gate; the time spent bought no result.

Code from scratch, official pass: paid models

One pass per model and configuration, except the row indicated. Pass criterion: five stages cleared. IT Lab Bench, edition 1. "No reasoning" means reasoning disabled, which is not comparable across model families.

Probe: forced reasoning in local models

Probe, not an official pass. The from-scratch brief was repeated with medium-level reasoning on five local models, in two batches, for a total of ten runs. In all of them the delivered content was zero characters: the reasoning consumed the entire output budget and the model wrote no file. Raising the reasoning did not improve the result; it left it at zero.

Probe: content without the format wrapper

Probe, not an official result. The two non-passing results of mimo-v2.6-flash correspond to the delivery format and not to the code. When the format wrapper was removed and the same content was judged, the model cleared all five stages. The probe does not change the official result in the paid-model table; it shows that there are two ways to fail a code exam and that they are not the same problem: the knowledge can be complete and the delivery broken.

Realistic programming: smaller realistic cases

One pass per model. Pass criterion: case passed by the automatic judge. Realistic cases touching one or two files, preceding the hard series. IT Lab Bench, edition 1.

The four strongest models pass every case; what differs among them is how much reasoning they need, not the score.

Realistic programming: hard series, two repetitions

Two repetitions over five cases, that is, ten runs per model. Pass criterion: case passed on visible and hidden tests. IT Lab Bench, edition 1. The level and speed columns are the aggregates of the second repetition.

Passed and failed cases are reported with the judge's final result. The total time to solution of each model is not published, because the two available sources for that figure do not agree and a figure that cannot be sustained is not reported.

Failure classes observed

Across the four programming phases, transport failures are practically nonexistent; class (c) lives mostly in format. One case is excluded from clean quality conclusions because the instrument defect (b) contaminates it.

There is also a borderline case between classes (a) and (c): two variants of the same model produced an unclosed code block and, when the intact code was extracted, that same code passed the visible and hidden tests. The decision is to keep the contractual format failure and report the code's success separately, instead of producing a single figure. The same result can be read as 1 of 3 or as 2 of 3 depending on what is counted, and the report states it.

4. Discussion

What is signal

Completing beats creating. On an existing base with fixed tests, three of ten local models pass, each in under a minute. From scratch, none of the ten passes and the paid models do. In IT Lab's judgment, the difference between the two families is a robust signal, because it is ten models against zero and repeats in structure (three of ten against zero of ten), even though each family has a single pass per model.

Forced reasoning is not a universal lever. On the from-scratch brief, raising it left the delivered content at zero characters across ten runs. In realistic programming, the ladder uses it only when the case demands it. The operating rule IT Lab derives is to raise the level only when the case demands it and to measure the time and tokens it costs.

Generation speed does not predict accuracy. The model with the highest median speed, 115.3 tok/s, scored 7 of 10, and two models at around 40 tok/s scored 10 of 10. The second fastest, at 87.5 tok/s, writes between 18,000 and 41,000 tokens per attempt on the hard cases at high level and scored 9 of 10: more tokens was not a better solution, only more effort. The relevant metric for a work decision is the time to a correct solution and the number of retries it costs, not tokens per second.

Delivery weighs as much as the code. The model that did not pass because of format cleared all five stages when the wrapper was removed; models with careless formatting lose cases they had solved. A relevant share of the failures an organization would attribute to "the AI" are, in the bench, integration or instrument failures.

What is noise

In realistic programming, 10 of 10 against 9 of 10 does not establish an order. With two repetitions and cases that change outcome between them, IT Lab regards the differences among Qwen3.8-27B-GSQ-RCO, Muse-Glimmer-30B, Swift 1.5 and gpt-oss-20B as within the noise of two repetitions. This report does not rank them and does not name a model chosen by IT Lab. It is a signal, however, that the model with 7 of 10 fails the same case in a stable way, with the same pattern, in both repetitions.

Nor are time differences from a single pass among paid models, or single-gate differences among the local models of the existing-base family, which have a single pass, interpreted as differences in capability.

5. Implications for organizations with limited resources

For completing and maintaining code on an existing base, in IT Lab's judgment there are local models that pass the task in under a minute, with behavior tests included and without the code leaving the company's infrastructure. The condition is that fixed tests exist to decide whether the work is right, and that a person reviews the change as with any other.

For creating modules from scratch, the local result is zero of ten. IT Lab recommends that creation from scratch be a mandatory-review task, or be delegated to a paid model when the data policy allows it. In that case the decision is one of risk and volume: whether the code may leave for a provider, how often the task repeats and what happens if it fails.

For those maintaining realistic multi-file repositories with interacting faults, there are local models that score 10 of 10 on a 16 GiB GPU. Before discarding a model for a poor result, the organization should check whether the problem lay in the code or in the way it was delivered; and before raising reasoning "to improve it", measure what it costs.

Frequently asked questions

Can local AI write code? On a base with marked gaps, three of ten local models pass all six stages in 17 to 43 seconds. From scratch, none of the ten does.

Does generation speed predict accuracy? No: the model with the highest median speed, 115.3 tok/s, scored 7 of 10, and two models at around 40 tok/s scored 10 of 10.

What does IT Lab recommend? Using local models to complete and maintain code with review, and requiring mandatory review when the model creates code from scratch.

How much does local AI cost? The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.

6. Limitations and scope

• Code on an existing base and code from scratch have a single pass per model; small differences, including a single-gate difference, are noise. Realistic programming uses two repetitions, with differences within the noise.
• The official pass is reported apart from probes and diagnostics, labeled as probes in the results section.
• GPU memory cap: 15.5 GiB.
• "Reasoning disabled" is not homogeneous across model families; those levels are not compared across families.
• The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• The staged judge only allows comparison within the same brief and with the same judge; the result is not extrapolated to Rust, to Ruby or to programming in general.
• The total times of the hard series are not published, because they do not agree across the sources.
• The team-mode trial has two models and one pass: it is not conclusive.
• The results are an IT Lab measurement on a single machine; they are not a purchase recommendation or a certification.

7. Conclusions

1. On an existing base with marked gaps, three of ten local models clear all six stages in 17 to 43 seconds. In IT Lab's judgment, completing and maintaining code is viable today with local models on accessible hardware.
2. From scratch, none of the ten local models meets the criterion, and two paid models do.
3. In realistic programming, several models score 9 or 10 of 10 and the differences are within the noise of two repetitions. Generation speed does not predict accuracy.
4. The reasoning ladder, which raises the level only when the case demands it, is the operating rule IT Lab adopts. Forcing reasoning in bulk can leave the result at zero.
5. Failures are classified as model error, instrument defect, and format, limit or transport. An important share of the observed failures is not the model's, and the bench separates them.

Appendix: multi-agent orchestration trials

This appendix is brief, because the evidence comes from exploratory trials with one pass per model.

A hierarchy of roles was trialed on a single local model, with separate histories and budgets: a coordinator that plans and decides at the end, an integrator that breaks the brief into subtasks, a builder that writes the code, a reviewer that looks for defects with severities and a fixer that acts only if the reviewer asks. The flow has seven phases, and the repair is conditional.

The mechanics work when the model answers in a reliable structured format: eight of ten models finish their run and the coordinator accepts the result in five. The bottleneck is not the model's capability but the format. In the early matrices, 14 of the 17 listed failures were decoding errors of the structured format. After validating a deterministic format repair, the listed failures fell from 17 to 13 in batches of the same size: a real improvement, and far from sufficient.

Orchestrating more does not compensate for a model that cannot program. On the from-scratch brief, the only model that completed all 13 of 13 loop phases delivered an empty skeleton, which the judge stopped at the second gate; the others did not reach delivery. On the existing-base family, the two models tried in team mode did not pass. In IT Lab's judgment, team mode is not incorporated into the bench at this level of evidence.

Measurement sheet

• Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
• Quantization: 3 or 4 bits, depending on the local model.
• GPU memory cap: 15.5 GiB.
• Passes: one per model and configuration for code on an existing base and from scratch; two repetitions for realistic programming.
• IT Lab Bench, edition 1.

How to cite

IT Lab, Technical Report ITL-IT-03, IT Lab Bench, edition 1.

Related articles: Code: completing an existing base vs creating from scratch · Hard programming in real repositories · Method: Technical report 4: IT Lab Bench methodology

Contact: IT Lab evaluates local AI models for Chilean organizations using their own code and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.