Local AI for code: completing a base vs creating

In the IT Lab Bench, three of ten local models complete an existing code base with marked gaps and pass all six stages of the judge. Creating the application from scratch, none of the ten local models meets the criterion; four paid models are measured separately.

• On the existing base, three local models pass all six stages, at 16.8 to 43.5 s per run.
• From scratch, all ten local models stop before the last step of the judge; two paid models pass both of their runs.
• In IT Lab's judgment, local models fit completing and maintaining code; creating from scratch requires mandatory review.

The business question

A company with a development team needs to know for which work it can use a model that runs inside its own infrastructure, without sending code to a provider. Maintaining a system and writing a new module are different tasks; the bench measures them separately.

What the test measures and why it represents real work

Both tests use the same application, with an engine in Rust and a web API in Ruby on Rails, so the work crosses two languages.

• Completing an existing base. The system exists and has marked gaps: the model implements the missing engine logic and the missing pieces on the web layer.
• Creating from scratch. There is no skeleton. The model designs and writes the engine and the web bridge, and delivers exactly the files requested.

In both, the test files are fixed by the lab, and the instrument invalidates runs that do not respect the delivery rules. The content of the task is not published.

How a model passes

The judge is a deterministic program, with no model evaluating another, and it works in chained stages (compilation, behavior, web-layer tests and end-to-end test): it stops at the first one that fails. The test on an existing base has six stages; the from-scratch test has five.

The result therefore shows where the work broke: "delivered no files", "does not compile", "compiles but does not meet the requested behavior" and "the engine works and the web layer fails" are distinct failures. A truncated output ceases to exist as a delivery.

Results: completing on an existing base

Ten local models, a single pass per model, pass criterion: all six stages. IT Lab Bench, edition 1.

The two models that delivered nothing fail for different reasons: one exhausted its output budget before writing files, a budget failure and not a comprehension failure; the other returned unusable content. A numeric score would have conflated them.

A team-mode trial, with several coordinated agents, produced no conclusive result.

Results: creating from scratch

One pass per local model; two passes per paid model unless noted. Pass criterion: all five stages. IT Lab Bench, edition 1.

Local models, aggregated.

No local model meets the criterion. The furthest one compiles, but its code does not do what was asked.

Paid models. The two passes of gpt-6-luna and muse-spark-1.3 correspond to one configuration with reasoning disabled and one with low reasoning.

Reasoning probe. In ten local runs with medium reasoning, the delivered content was zero characters, because reasoning consumed the entire output budget. This probe is reported separately and is not part of the official pass.

What this means for a Chilean company

In IT Lab's judgment:

• What to automate. Completing and maintaining code on an existing base with tests already written: some local models pass it in under a minute, without the code leaving the company.
• What to review. Every module created from scratch requires mandatory human review of the deliverables, even if the code compiles. This test showed code that compiles and does not do what was asked.
• What to define beforehand. The delivery format, with explicit files and paths: a failure to pass can come from the packaging and not from the code.
• When to pay. If the code may go to a provider, gpt-6-luna and muse-spark-1.3 passed in both passes; the decision depends on risk and volume.

Frequently asked questions

Can local AI write code? On an existing base with gaps, three of ten local models pass all six stages of the judge. Creating the application from scratch, none of the ten does.

What is it useful for today in a development team? In IT Lab's judgment, for completing and maintaining code; creating from scratch requires mandatory review.

Does the code leave the company? Not with a local model: it runs inside the company's infrastructure, without sending code to a provider.

How fast does it work? The three local models that pass all six stages do so in 16.8 to 43.5 s per run.

Limitations and scope

• A single pass per local model and case, and two per paid model unless noted; small differences are noise.
• The official pass is reported apart from the probe.
• Staged judges only allow comparison within the same task and with the same judge; a result here does not extrapolate to "Rust in general".
• GPU memory cap for the local runs: 15.5 GiB.
• "Reasoning disabled" is not homogeneous across model families and is not compared between them.
• The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

Measurement sheet

• Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
• Quantization: 3 or 4 bits, depending on the local model.
• GPU memory cap: 15.5 GiB.
• Passes: one per local model; two per paid model unless noted.
• IT Lab Bench, edition 1.

General method: How IT Lab evaluates a local AI model

Related technical report: Technical report 2: Code

Contact: IT Lab evaluates local AI models for Chilean organizations using their own code and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.