Agentic work with local AI: finishing the job

In the IT Lab Bench agentic-work family, 5 of 14 models meet the criterion, and the best result came from a local model. The characteristic failure is not ignorance of the answer but failing to close the job. General method: How IT Lab evaluates a local AI model.

• Qwen3.8-27B-GSQ-RCO, a local model, scored 100 in 20.6 s, and three of the five models that meet the criterion run on a 16 GiB card.
• Models that fail tend to find the cause and not deliver.
• In IT Lab's judgment, agentic work still requires human review and a mandatory delivery step in the workflow.

The business question

Delegating processes to an agent requires knowing whether the model finishes the job without intervention. Answering well and completing a sequence of actions are different capabilities; this family measures the second.

What the test measures and why it represents real work

The test has three episodes with real tools and actions, operational and IT in nature, in which the model must act step by step and close the job. Every model receives the same protocol.

The measure is the closed task, not activity: reading every file without delivering a correct result does not count. The content of the episodes is not published.

How a model passes

A fixed automatic judge, with no model grading another, scores the delivered work. The pass threshold is 80 and was set before the run. The official pass is a single run per model, in series, under the same protocol for all.

Results

Official pass: one pass per model; threshold 80; IT Lab Bench, edition 1; reasoning disabled (not comparable across model families).

GPT-OSS-20B aborted after three empty responses.

In IT Lab's judgment, time does not buy score: a paid model took 222.6 s for a score of 96, while a local model scored 95 in 5.1 s. The difference between 100 and 99 in a single pass is within noise and does not declare a better model.

Where the work breaks

Five models explored optimally and still failed the final calculation of the order: the plan was right and the arithmetic was not. The pattern most relevant to a company is another one: models that know but do not finish. One model found the root cause of the incident and repeated the same action without submitting; two used up the turn limit reading, without delivering; another answered empty three times in a row. An agent that does not deliver adds no value in production: the job is not closed.

Diagnostic (outside the official pass)

This is not part of the official pass. It measures how much enabling reasoning corrects and what it costs; it includes extra passes.

IT Lab's reading: reasoning is strictly necessary to pass in only one model. In the two Gemma models, enabling it repairs the calculation and moves the score from 76 to 100 and from 74 to 97, at the price of multiplying tokens and time; in Qwen3.8-27B-GSQ-RCO the score does not change. Time is multiplied by 12 to 39.

"Reasoning disabled" is not homogeneous: a paid model logged 4,910 reasoning tokens in its run without reasoning, so those levels are not compared across families.

Multi-agent orchestration on a single GPU

IT Lab ran a separate exploratory test: a hierarchy of roles on a single local 27B model, with several concurrent context sessions on the same card. A planner makes the final decision, an integrator breaks the assignment into subtasks, and several executors produce and review the pieces.

The mechanics work. With a reliable output format the scheme completes the piece: eight of ten models finish their run, and the concurrent executors add up to about 63.5 tok/s with several concurrent sessions.

The bottleneck is the output format, not intelligence. In the first test matrices, 14 of the 17 failures were format decoding errors. With deterministic format repair they fell from 17 to 13 in batches of the same size: a real improvement, and far from sufficient. Two models do not fit the required context under the memory cap. And the only model that completed all phases of the loop delivered a result with no useful content, which the judge stopped at its second step: more orchestration does not compensate for a model that cannot program. This is evidence from a single exploratory test; its recorded conclusion is "not ready to adopt".

What this means for a Chilean company

In IT Lab's judgment:

• What to automate. Operational tasks with tools, with human review and delivery as a mandatory step with a deadline.
• What to review. Every exact calculation, and demand the result rather than the activity.
• Format and reasoning. An orchestrator needs deterministic format repair and a retry; reasoning is decided per model, with its cost in tokens and time in view.
• Hardware. Three of the five models that meet the criterion run on a 16 GiB card.

Frequently asked questions

What is agentic work? Tasks in which the model uses tools and takes actions. The bench measures them in three episodes.

How reliable is local AI on this kind of work? 5 of 14 models meet the criterion, and the best result came from a local model, with 100 in 20.6 s.

What is the typical failure of these models? Not closing the job: the models that fail usually find the cause but do not deliver.

Can it run unsupervised? In IT Lab's judgment, agentic work still requires human review and a mandatory delivery step in the flow.

Limitations and scope

• A single pass per model and episode; small differences are noise.
• The reasoning diagnostic includes extra passes and is kept separate from the official pass.
• "Reasoning disabled" is not homogeneous across model families; those levels are not compared across families.
• The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

Measurement sheet

• Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
• Quantization: 3 or 4 bits, depending on the model.
• Passes: one per model in the official pass, in series.
• IT Lab Bench, edition 1.

Related technical report: Technical report 1: Local AI office capabilities

Contact: IT Lab evaluates local AI agents with each organization's own processes and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.