Technical report 1: office capabilities of local AI

Code: ITL-IT-02 · IT Lab Bench, edition 1

Summary

This report brings together the results of four IT Lab Bench test families that represent office work: document reading, vision, agentic work and reasoning applied to Chilean business cases. Fourteen models were evaluated, ten local and four paid, with one official pass per model and a fixed automatic judge. In reading, twelve of fourteen models extracted all 183 cells without error, but only two models, both paid, obtained the full score (100), because the exam was decided by the counting questions. In vision, five of six eligible models scored 97 or higher and two local models reached 97. In agentic work, five of fourteen passed and the best result came from a local model, with 100. In applied reasoning, no model passed all 40 cases and the best local result was 30 of 40. A section points to the article on receipts and invoices. In IT Lab's judgment, local models extract and read well enough to automate under supervision, while counting, finishing the job, declaring what is unreadable and backing every claim with the right source still require human review.

1. Introduction and question

An organization considering local AI for administrative tasks needs to know which ones it can automate, which ones a person must review and what margin of error to expect. Traditional public benchmark scores do not answer this, because they measure short, isolated problems. The IT Lab Bench, described in Technical report 4: IT Lab Bench methodology, measures whether a model finishes a recognizable task without human intervention.

This report answers a bounded question: what a local model does today, on accessible hardware, across the four capabilities closest to the office work of a Chilean company. Each family has its own explanatory article: document reading, vision, agentic work, reasoning on Chilean business cases and receipts and invoices.

2. Method

Test families

Reading and extraction. A real document with several tables is converted to structured data: 183 cells compared one by one against the reference. The model then answers 12 closed questions about the same document, grouped by type, from direct lookup to several forms of counting, to separate reading from aggregating.

Vision. Eleven real images of varied types with 31 closed questions. The score is the percentage of correct answers.

Agentic work. Three episodes with real tools and actions, operational and IT in nature, under the same protocol for local and paid models.

Applied reasoning. Forty Chilean small-business cases across varied thematic areas: administrative documents, spreadsheets, email threads, quotations, labor regulation, cybersecurity, meeting minutes and others. The material and references are frozen. A case passes only if every claim is backed by the correct source in the material; citing a real but wrong document is a failure.

Judge, criterion and quality controls

The judge is a deterministic program in every family; no model grades another. Each family fixes its criterion before the run: 80 out of 100 in agentic work, a score out of 100 in reading and vision, and number of cases passed out of 40 in applied reasoning. In that family "passes" means the case cleared the automatic controls; some areas with expert review under way are not included in the score. The validators are subject to checks before use.

The protocol requires giving each model its best opportunity, with its best parameters and its best form of instruction. Three instrument controls apply here: format repair is deterministic and separate from the model (an output cut off by a limit is not repaired; the output budget is raised); in vision each image sits next to its questions; and if a model does not fit within the memory cap at its maximum context, the context it supports is measured rather than excluding it. The protocol also separates a model that exhausts its output budget while reasoning from one that cannot solve the task, because from outside they look the same.

The official pass is one per model, run serially, with the same protocol for local and paid models. Diagnostics are reported in labeled subsections and never mixed with the official tables. Local runs were strictly serial, on the machine described in the measurement sheet, with a GPU memory cap of 15.4 GiB.

3. Results

All tables in this section belong to the IT Lab Bench, edition 1. In reading, vision and agentic work the official pass is one per model; the pass threshold of 80 applies to agentic work. Times are wall-clock seconds per run.

Reading and document extraction

Official pass.

Legend: one pass per model; IT Lab Bench, edition 1. Only two models, both paid, obtained the full score (100); no local model did.

Diagnostic: accuracy by question type. The table below splits the exam questions by what they demand, over 24 analyzed runs. It is not part of the official pass.

Failure patterns. Two models did not extract all 183 cells without error: one recorded 181 of 183 and another produced no parseable output. For the rest, extraction barely discriminates; the difference arose in the counting questions, where models returned plausible but wrong numbers. The most demanding question was answered correctly in 5 of the 24 analyzed runs and by no local model in the official pass. In IT Lab's judgment, this indicates comprehension present and aggregation absent, not a formatting or budget problem.

Diagnostic: the model with no result. One local model scored 0 because it exhausted its output cap while reasoning without producing content. Doubling the cap gave the same result and the log showed looping repetition, not a lack of headroom. From outside, "does not reason well" and "had no room to reason" are indistinguishable; the protocol records them separately.

Vision on images and scanned documents

Official pass.

Legend: one pass per model; 11 images and 31 closed questions; IT Lab Bench, edition 1. Five of 6 eligible models score 97 or higher.

The excluded model receives the image but declares it cannot see it; it is dropped for that declaration, not for a low score. The two local models at 97 tie the paid ones, and the fastest local model completes the whole exam in 7.0 s with 87.

Diagnostic: effect of image and question order. When all images are delivered first and the questions afterwards, models confuse the association between image and question. One model went from 77 to 87 when each image was interleaved with its questions, with no question changed. The official format therefore uses the interleaved layout: the difference belongs to the instrument, not to the model.

Failure pattern. Faced with tiny or degraded text, no model read it, and several completed it with a plausible name instead of declaring they could not see it. A classic character reader did not resolve it either. This failure mode differs from a misreading: the model produces a believable answer rather than a signal of uncertainty.

Agentic work

Official pass.

Legend: one pass per model; three episodes; pass threshold 80; IT Lab Bench, edition 1. Five of 14 models pass (36%); three of the five are local.

The best result came from a local model, with 100 in 20.6 s, ahead of a paid model with 99 in 60.3 s. At the other end of the time scale, a paid model took 222.6 s to score 96, while a local model scored 95 in 5.1 s: in this exam, time does not buy score.

Failure patterns. The exam separates exploring from closing. Five models explored optimally, reading the necessary files, and still miscalculated the order total. A failure category that the bench treats as its own also appears: models that know but do not finish. One identified the cause of the incident and looped, repeating the same action without submitting the result; two used up the turn limit reading without delivering; another answered empty on three consecutive turns. The model that failed in reading by exhausting its output cap while reasoning failed here too, with the same diagnosis. The judge treats work not delivered as work not done.

Diagnostic: cost of enabling reasoning. Across the eight eligible models, the change in result when reasoning is enabled was measured outside the official pass; it is not comparable across families. The "disabled" level is not homogeneous: one paid model recorded 4,910 reasoning tokens in its run without reasoning. In two local families, the medium and high levels produced byte-identical output. Enabling it is strictly necessary for only one model; in the others it repairs the arithmetic at the price of multiplying time by 12 to 39, with output tokens 6 to 62 times larger and a score that rises by 0 to 23 points.

Unofficial diagnostic; a single run per cell; not to be mixed with the official table.

Applied reasoning on Chilean business cases

Campaigns by model. Each campaign runs the 40 cases on the same frozen material and reports the number of cases that clear the automatic controls. This family uses per-model campaigns, not a common pass of fourteen models; in the local campaigns the reasoning level was not recorded case by case, except where indicated.

Legend: one pass per campaign unless indicated; times are the sum of recorded generation seconds, not wall-clock time; paid models do not run on an IT Lab server and have no comparable speed; IT Lab Bench, edition 1. "Passed" means automatic controls cleared.

No model passed all 40 cases. The best local result, 30 of 40, corresponds to a configuration with reasoning tuned case by case; the best paid result was 28. The fastest model in the table, at 121.9 tok/s, passed 9 of 40: speed does not predict comprehension.

Diagnostic: the reasoning ladder. In the best campaign, every case starts at the lowest level and only failed cases move up a level. The low level passed 23 cases; the ladder rescued another 7, going through 28 to reach 30. Seven cases stayed failed even at the maximum level. In another focused test with Swift 1.5, three hard cases failed the same way with reasoning disabled and with low reasoning; the low mode generated more tokens and took longer without turning any into a pass. A process limitation: there is not yet a complete comparison between models by reasoning mode, because historical campaigns did not always record it.

Failure patterns. Structural failures repeat across models and do not disappear when reasoning is raised. In general terms: claims not backed by the correct source in the material; false premises in the brief accepted as true; invented agreements or rules instead of abstaining, although honest abstention exists and is recorded separately; evidence misclassified, for example attributed to the wrong document; and miscounts over what had in fact been read. The distribution of failures by area, summed over all campaigns with a corrected record, is as follows.

IT Lab aggregate over the corrected records of roughly 20 campaigns; only the six areas with the most failures.

Tax documents and receipts

IT Lab also measures the reading of photographed Chilean tax documents and receipts, and has its own experience and criteria for this type of document. In general terms, models with vision handle the simplest cases and the most demanding ones need human review. The detail is covered in Local AI for receipts and invoices.

4. Discussion

The readings below are IT Lab's judgment based on its measurements; none rests on a single-pass difference between close models.

Extract yes, aggregate no. In reading, 12 of 14 models extracted all 183 cells without error and extraction stopped discriminating. In the same exam, accuracy falls steadily when the question moves from looking up to counting, from 100% to 20.8%. A parallel pattern appears in agentic work: five models explored optimally and failed in the final calculation. In IT Lab's judgment, reading and exploring are mature capabilities in the local models evaluated; exact aggregation over what was read is not.

Finishing the job is distinct from knowing. Interpreting images is largely solved, with five of six eligible models at 97 or higher, while delivering a multi-step job is not: in agentic work five of fourteen models passed. The agentic failures are not failures of knowledge: one model identified the cause and did not submit it, others used up their turns reading, another answered empty. In production, work that is not closed cannot be reviewed or used, and the judge treats it as not done.

Inventing versus declaring unreadable. In vision and in reading tax documents, the highest-risk failure mode is not misreading but producing a believable answer where uncertainty should be declared. A classic character reader fails on the same side with degraded text, but returns noise rather than a plausible guess; the model adds that risk and, in exchange, resolves context judgments the classic reader cannot. That is why the judge punishes invention and not declared illegibility, and why any extraction flow needs an explicit "not legible" output.

Reasoning fixes part, not the structural. In agentic work, enabling reasoning repairs the arithmetic of some models at 12 to 39 times the time, and is necessary for only one. In applied reasoning, the ladder goes from 23 to 30 cases, but seven stay failed at the maximum level and are structural: source backing, false premises, misclassified evidence, invented rules and agreements. In IT Lab's judgment, extra reasoning is a lever for correcting individual cases, with a measurable cost, not a way to make a model reliable when it does not back what it says; the ladder, which raises the level only when the case requires it, is the operating rule that best balances result and cost.

Speed and noise. Speed does not predict accuracy: the fastest model in applied reasoning passed 9 of 40, and in agentic work time is unrelated to score. Differences of one or two points between models in the same group come from a single pass and do not establish that one outperforms the other.

5. Implications for organizations with limited resources

IT Lab's judgment on its measurements, valid for the machine described.

• Automate under supervision: extraction of tables and data, search within documents, general reading of screenshots, logos and scanned pages, and reading of legible tax documents. Local models matched paid ones in vision and scored between 87 and 93 in reading, which allows documents to stay inside the organization.
• Always review: any count or total computed over what was read; small or degraded text; and any citation of a rule or claim that must be backed by the correct source.
• Design the close: in multi-step flows, submitting the result must be a mandatory event with a deadline, and the flow must demand the result rather than the activity.
• Require an uncertainty output: a "not legible" field the model can use and rule-based validation of fields with known structure before accepting the reading.
• Reasoning with judgment: apply it by ladder rather than by default, because it multiplies tokens and time and does not change structural failures.
• Hardware: three of the five models that passed agentic work are local and fit on a 16 GiB card.

Frequently asked questions

Which office tasks did the bench measure? Document reading, vision, agentic work and reasoning applied to Chilean company cases, with fourteen models: ten local and four paid.

How reliable is local AI at reading documents? Twelve of fourteen models extracted all 183 cells of a document without error; counting over what was read requires review.

What can be automated without review? In IT Lab's judgment, data extraction and general image reading can be automated; counts, fine text and agentic work require human review.

How much does local AI cost? The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.

6. Limitations and scope

• One pass per model in reading, vision and agentic work; small differences are noise. Applied reasoning uses per-model campaigns.
• The official pass is kept apart from diagnostics: accuracy by question type, image and question order, cost of enabling reasoning and the reasoning ladder.
• In applied reasoning, "passes" means automatic controls cleared; some areas with expert review under way are not included in the score.
• The reasoning level of local models was not recorded case by case in those campaigns, so there is no complete comparison by mode; paid models have no comparable speed and the sum of generation seconds is not wall-clock time.
• GPU memory cap of 15.4 GiB: one model took part with reduced context in applied reasoning and two received the maximum context they supported in agentic work.
• "Reasoning disabled" is not homogeneous across model families: those levels are not compared across families.
• The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• IT Lab measurement on one machine; not a purchase recommendation or a certification.

7. Conclusions

1. Document reading and extraction are not today the factor separating local models from paid ones: 12 of 14 models extracted all 183 cells without error.
2. Counting and aggregating over what was read still requires review: only two models, both paid, obtained the full score in reading.
3. In vision, two local models matched the paid ones with 97; tiny or degraded text and the tendency to complete with something plausible are the limit.
4. In agentic work five of fourteen models passed and the best result was local, with 100; the characteristic failure is not closing the job.
5. In applied reasoning no model passed all 40 cases; the best local result was 30 of 40 with reasoning tuned case by case, which fixes part of the failures and not the structural ones.
6. For tax documents and receipts, IT Lab keeps human review as the rule and applies its own criteria, detailed in the dedicated article.

Measurement sheet

• Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
• Quantization: 3 or 4 bits, depending on the model.
• Context: the maximum each model supports within the memory cap; models that did not fit received the measured maximum.
• GPU memory cap: 15.4 GiB.
• Passes: one per model in reading, vision and agentic work; per-model campaigns in applied reasoning.
• Judge: deterministic program; pass threshold 80 in agentic work; reading and vision are reported by score.
• IT Lab Bench, edition 1.

How to cite

IT Lab, Technical Report ITL-IT-02, IT Lab Bench, edition 1.

Related technical report: Technical report 4: IT Lab Bench methodology

Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.