Documents with local AI: extract yes, count no

In the IT Lab Bench, turning a real document into structured data no longer separates the models; aggregating or counting over what was read does. In IT Lab's judgment, extraction and lookup can be automated, and every count should be reviewed.

12 of 14 models extracted all 183 cells of the document with no error, among them several that run on a 16 GiB card without the document leaving the company.
Only two models, both paid, obtained the full score (100); no local model did, and the difference was decided by the questions that require counting.
A model that exhausts its output budget while reasoning looks the same as one that does not know; the protocol keeps the two cases apart.

The business question

A company that wants to turn invoices, orders, spreadsheets or contracts into organized data needs two answers: whether a local model can transcribe without errors, and whether it can answer questions about what it transcribed. The first decides whether the work is automated; the second, how much human review it requires. This piece answers both with the document-reading test family of the IT Lab Bench, whose method is set out in how IT Lab evaluates a local AI model.

What the test measures and why it represents real work

Every office turns documents into data every day. The test uses a real document with several tables and dozens of positions, and sets a two-part task:

Extract. Move the content into an exact structured format, cell by cell: 183 cells compared one by one against the original, with required fields and a required format.
Answer. Twelve closed questions about the same document, mixing direct lookup and counting.

Results are measured by task completed, not by how convincing the answer looks. IT Lab publishes what capability the test measures and what the result is for; the content of the document and of the questions is reserved.

How a model is scored

A fixed automatic judge computes the score: a deterministic program, with no model evaluating another. The score combines extraction and questions, out of 100. Every model, local and paid, received the same protocol: a single pass, run in series, with no second chances. Fourteen models took part, ten local and four paid.

Results

Official pass of the reading family, IT Lab Bench, edition 1. One pass per model.

Model	Where it runs	Score	Time
muse-spark-1.3	paid	100	24.1 s
claude-haiku-4-5	paid	100	105.8 s
mimo-v2.6-flash	paid	93	16.6 s
gpt-6-luna	paid	93	47.1 s
Qwen3.8-27B-GSQ-RCO	local	93	67.6 s
Gemma-4-26B	local	93	24.3 s
Qwen3-Coder-30B	local	90	28.4 s
Qwopus3.6-35B	local	90	28.7 s
Gemma-4-E4B	local	90	32.6 s
Qwen2.5-Coder-14B	local	87	51.3 s
Ministral-3-14B	local	87	51.6 s
Gemma-4-12B	local	87	74.0 s
DeepSeek-Coder-V2-Lite	local	76	17.9 s
GPT-OSS-20B	local	0 (no interpretable content)	95.7 s

Only two models, both paid, obtained the full score (100); no local model did. Differences of a few points among the local models, measured in a single pass, fall within noise and do not establish an order among them.

Extraction no longer separates the models

Twelve of the 14 models copied all 183 cells with no error. The only one with real transcription faults recorded 181 of 183. For turning a table into structured data, almost any modern model is sufficient, including those that run on the company's own machine.

Accuracy by question type

The exam was decided by the questions. The table groups accuracy by type over the 24 analyzed runs of this family.

Question type	Correct	Share
Direct lookup	24 of 24	100 %
Finding a specific value	22 of 24	92 %
Counting (simple)	17 of 24	71 %
Counting (group total)	12 of 24	50 %
Counting with a variant to exclude	8 of 24	33 %
Counting, strict	5 of 24	21 %

Accuracy falls at the point where the task moves from finding to counting. The models found what there was to find; the final aggregation was missing. The failure is not one of format or budget: in IT Lab's reading, comprehension is present and arithmetic over what was read is absent.

Protocol control: the reasoning budget

GPT-OSS-20B ended with a score of 0. The cause was not a lack of knowledge: it spent the whole output budget on reasoning and produced no content. The run was repeated with the ceiling doubled and the outcome was the same; the server log shows a repetition loop, not exhaustion from too little room.

This case is not an anecdote but the reason for a control. From the outside, "does not reason well" and "had no room to reason" produce the same score. The protocol therefore gives reasoning its own budget, separate from the content of the answer, and records how much of the output was spent thinking. Without that separation, what gets measured is the budget and not the model. With it, the 0 is classified as a reasoning loop and not as an inability to read; the official score stays at 0.

What this means for a Chilean company

In IT Lab's judgment, given these results:

Automate. Extracting tables, spreadsheets and documents into structured data, and direct lookup of values. Almost any model in the test does it, and the local option is justified when the document cannot leave the organization.
Review. Every count, sum or aggregation over what was read. In this family only two models, both paid, obtained the full score and no local model did. Reading a contract and summarizing what it says is viable; counting in how many of several contracts a clause appears requires human review.
Hardware. Local runs were measured on a 16 GiB card, with times of 17.9 to 95.7 s per run.

Frequently asked questions

Can local AI turn invoices, spreadsheets or contracts into structured data? For extraction, yes: 12 of 14 models extracted all 183 cells of a real document without error. Several run on a 16 GiB card without the document leaving the company.

Where does local AI fail with documents? When counting or aggregating over what was read. No local model obtained the full score, and the difference was decided on the questions that require counting.

Does the model's output need review? In IT Lab's judgment, extraction and lookup can be automated, and every count must be reviewed.

What hardware is needed? Several of the models that extracted all 183 cells without error run on a 16 GiB graphics card.

Limitations and scope

A single pass per model; small score differences are noise and do not declare a superior model.
The official pass is kept separate from probes and diagnostics.
GPU memory cap of the configuration: 15.4 GiB.
The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
An IT Lab measurement on a single machine and a single document; not a purchase recommendation or a certification.

Measurement sheet

Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
Quantization: depends on the local model.
GPU memory cap: 15.4 GiB.
Passes: one per model (14 models); accuracy by question type over 24 analyzed runs.
IT Lab Bench, edition 1.

Related technical report: Technical report 1: local AI office capabilities

Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and data. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.