Local AI reasoning on Chilean business cases
40 Chilean small-business cases scored with automatic controls: no model passes them all, and extra reasoning fixes only part of the failures.
The IT Lab Bench puts models through 40 Chilean small-business cases and scores them with automatic controls. No model passes them all; the best local result is 30 of 40, with reasoning adjusted case by case.
- No evaluated model passes all 40 cases. The best local result is 30 of 40, and one paid model reaches 28 of 40.
- Extra reasoning fixes part of the failures, but not the structural ones: false premises, source backing, not inventing agreements or rules.
- In IT Lab's judgment, these tasks need human review before being delegated to a local model.
The business question
A company that reads, sorts and summarizes its own paperwork needs to know whether a model can reason about it without adding anything that is not written there. The risk is that the model errs with an air of certainty: a nonexistent rule or an agreement nobody made passes unnoticed in a quick read. The bench measures how often this happens and in which kind of task.
What the test measures and why it represents real work
The applied-reasoning family brings together 40 Chilean small-business cases across varied thematic areas. The material matches what a small company keeps in its files: administrative documents, spreadsheets, email threads, quotations, labor-regulation queries, cybersecurity evidence and meeting minutes. Each case requires reading the material and answering only with what it supports.
IT Lab publishes what capability each area measures and what the result is for; the content of the cases is not published.
How a case is passed
A fixed automatic judge decides each case using deterministic programs, with no model evaluating another. The pass criterion is set before the run and the test material stays frozen, so that any later pass runs on the same material and remains comparable.
The rule that sets this family apart is backing: a case passes only if every statement is supported by the correct source in the material. Citing correctly but from the wrong document counts as a failure. In cases involving regulation, inventing a rule is the most serious fault.
In this piece, "passes" means the case cleared the automatic controls. It is not the same as full human approval: some areas with expert review in progress are not included in the score.
Results
| Model | Where it runs | Passed out of 40 | Mean speed (tok/s) |
|---|---|---|---|
| Muse-Glimmer-30B | local | 30 (30 of 37 without the vision cases) | ~40 |
| Muse Spark 1.3 | paid | 28 | not recorded |
| Swift 1.5 | local | 26 (best pass; the range was 24 to 26) | 52.3 |
| GPT-OSS 20B | local | 24 | 140.8 |
| Qwen3.8-27B-GSQ-RCO | local | 24 | 48.2 |
| Gemma 4 12B | local | 21 (context reduced by the memory limit) | 47.3 |
| GPT-6 Luna | paid | 18 | not recorded |
| DeepSeek-Coder-v2-Lite | local | 9 | 121.9 |
Legend: one pass per model and case; for Swift 1.5 the best of several passes is reported. A case passes if it clears the automatic controls; there is no global pass threshold. Local-model speed was recorded by the inference server; paid models do not run on an IT Lab server, so their speed is not recorded. IT Lab Bench, edition 1.
Note: the reasoning level is not comparable across campaigns. In the local campaigns it was not recorded case by case, and "reasoning disabled" is not homogeneous across model families, so this table does not rank models by that setting.
In IT Lab's judgment, the ceiling is low and speed does not predict the result: the model that generated at 121.9 tok/s passed 9 of 40. Differences between close scores come from a single pass and are treated as noise.
The reasoning ladder: how much it fixes
In the best campaign a reasoning ladder was applied: each case starts at the lowest level and only the failed ones move up a level. At the low level 23 cases passed; the ladder added 7 more, so the count went from 23 to 28 and then to 30. Seven cases stayed failed at the maximum level, and they match the structural failure types the bench records across all campaigns:
- False premises. The model accepts a premise in the statement that the facts contradict.
- Source backing. The statement is plausible but is not supported by the document that corresponds.
- Invented agreements and rules. The model records agreements that no draft contains or cites rules absent from the material, instead of abstaining. Correct abstention exists and is recorded separately, but it is the exception.
- Reading small data. Transcribing an identifier printed in a photographed image remains a weak point.
Raising the level has a cost: it multiplies tokens and time. In a focused test with Swift 1.5, three hard cases failed equally with reasoning disabled and at the low level; the low level generated more tokens and took longer without turning any of them into a pass. In IT Lab's judgment, extra reasoning repairs individual cases but does not make reliable a model that does not back what it states.
Where failures concentrate
The bench counts failures by thematic area across all campaigns with corrected records. The six areas with the most failures are:
| Thematic area | Recorded failures |
|---|---|
| Cybersecurity: classifying evidence | 66 |
| Labor regulation: citing a rule without inventing | 62 |
| Real spreadsheets | 59 |
| Minutes: not inventing agreements | 57 |
| Structured documents with sources | 56 |
| Closing tasks and counts | 50 |
What this means for a Chilean company
In IT Lab's judgment, based on the results of bench edition 1:
- Automatable with supervision: sorting and reconciling data from documents and spreadsheets, classifying evidence and answering queries about specific documents.
- Human review mandatory: counting and aggregating across many documents, citing regulation, drafting minutes and any reading of a small datum in an image. Receipt reading is covered in Local AI: receipts and invoices.
- Hardware: the best local result was obtained on a 16 GiB card. For data that cannot leave the company it is a measured option, always with a person reviewing.
The full method is described in How IT Lab evaluates a local AI model.
Frequently asked questions
Can local AI reason about my company's cases? The bench presented 40 Chilean small-business cases: no model passes them all, and the best local result is 30 of 40.
Does more reasoning fix the failures? It corrects some of them, but not the structural ones: false premises, backing from sources, not inventing agreements or rules.
What is the main risk? That the model is wrong while sounding sure: a nonexistent rule or an agreement nobody made goes unnoticed on a quick read.
Can it be delegated without review? In IT Lab's judgment, these tasks require human review before being delegated to a local model.
Limitations and scope
- A single pass per model and case; small differences are noise.
- "Passes" means automatic controls cleared, not full human approval. Some areas with expert review in progress are not included in the score.
- The reasoning level is not comparable across campaigns or across model families.
- There is no comparable speed for the paid models.
- The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
- An IT Lab measurement on a single machine; not a purchase recommendation or a certification.
Measurement sheet
- Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
- Quantization: 3 or 4 bits, depending on the model.
- GPU memory limit: 15.4 GiB.
- Passes: one per model and case, except as noted in the legend.
- IT Lab Bench, edition 1.
Related technical report: Technical report 1: Office capabilities
Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk
The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.
Do you need someone to take care of your technology?
Tell us what you need to organize, protect or solve. The first conversation has no cost: we assess whether our model fits you and what the next steps would be.
Let's talk about your business