Local AI for a Chilean company: what to automate (1/2)
Which office and development tasks a local model can automate today, which need human review, and what hardware it takes.
The IT Lab Bench measured 14 models on documents and tasks of a Chilean company. In IT Lab's judgment, a local model can take over data extraction and lookup today, while any calculation or count requires human review. This page gathers the decision and links to the evidence for each test family.
- Extracting and locating data in documents can be automated: 12 of the 14 models extracted all 183 cells without error, four of them local; counting or aggregating over what was read requires review.
- Local suits cases where data must not leave the company or where volume is high; today it does not suit building software from scratch or high-precision vision.
- The hardware that suffices is a 16 GiB graphics card running one model at a time; concurrency for several users has not been measured.
The business question
A Chilean company considering local AI has to settle two points: which tasks it can delegate to a model running on its own machines and which ones a person must review. It also has to know the hardware required and the risk it takes on with its data. The IT Lab Bench measures complete tasks; the method is described separately. The figures are laboratory measurements, not a purchase recommendation.
The general rule
In IT Lab's judgment, the rule that sums up the measurements is this: extract and locate, yes; count and aggregate, with review.
In the task of reading a document with 183 cells to extract, 12 of the 14 models extracted all of them without error, including four local ones: Qwen3.8-27B-GSQ-RCO, Gemma-4-26B, Qwen3-Coder-30B and Qwopus3.6-35B. Direct lookup questions were answered correctly in 24 of 24 runs, and those locating a specific value in 22 of 24. By contrast, the strict counting question was answered correctly in only 5 of 24 runs, and no local model answered it correctly in the official pass.
The line does not depend on model size or price, but on whether the task consists of finding a piece of data or of calculating from several.
Task matrix
Each task links to the article with passes, criterion and the detail of the figures.
| Task | Automatable today, locally | Risk | Human review |
|---|---|---|---|
| Extracting fields from a document | Yes | Low | No, if the output format is validated |
| Locating a specific data point | Yes | Low | No |
| Counting, summing or aggregating over what was read | Partly | High | Yes, always |
| Answering business questions about a document | Acceptable with a large model | Medium | Yes, by sampling |
| Photographed receipts | With supervision and IT Lab's own criteria | High | Yes |
| Vision: small text, marks, scans | Nearly (97 of 100 for the best local models) | High on fine text | Yes, for anything critical |
| Agentic work with tools | Yes, with the right model | Medium-high | Yes: only 5 of 14 models meet the criterion |
| Code on an existing base | Yes | Medium | Yes, with the usual code review |
| Code from scratch against a demanding contract | Not locally | High | Mandatory: none of the 10 local models met the criterion |
| Programming in repositories with interacting faults | With review of results | Medium-high | Yes |
| Reasoning about Chilean business cases | As support, not as the final decision | Medium-high | Yes: no model passes every case |
| Delivery format (JSON consumed by another system) | Depends on the model; 14 of 17 failures were wrapper failures, not knowledge failures | High | Yes, if the downstream system is automatic |
IT Lab Bench, edition 1; one pass per model and case unless the linked article states another number. Results per model are collected in IT Lab Bench results.
When local is the better fit
Sensitive data that must not leave. The tax and administrative documents of a Chilean company contain sensitive data. With Chilean data-protection and cybersecurity regulation (its scope is for the organization's legal advisor to determine), control over where data is processed is more than a technical detail. Here the argument is risk: a model running on the company's own machines keeps the document inside its infrastructure.
High volume of repetitive tasks. In document extraction, four local models extracted all 183 cells without error.
Network dependence. A local direct-lookup query resolved in 5 to 28 s and needs no internet connection. The most expensive remote run in the comparison took 947.9 s and ended in failure.
When local is not the better fit today
- Building complex software from scratch. The 10 local models did not meet the criterion; the paid models did. See Code: complete versus create.
- High-precision vision. The only 100-of-100 result came from a paid model; local models reached 97, with fine-reading errors. See Vision.
- Very large contexts at good speed. Generation drops abruptly near 50K tokens; see SMB hardware.
Continues in Part 2.
Contact: IT Lab assesses whether a Chilean organization is ready for local AI on its own documents and data. Let's talk
The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.
Do you need someone to take care of your technology?
Tell us what you need to organize, protect or solve. The first conversation has no cost: we assess whether our model fits you and what the next steps would be.
Let's talk about your business