Vision: images and scanned documents

In the IT Lab Bench, reading screenshots, logos, prices and visible text is a task that local models now handle at the level of paid ones. The limit is tiny or degraded text, where a model may fill in something plausible instead of reading it.

• Five of six eligible models score 97 or higher; two local models reach 97.
• On tiny text engraved on an object, no model read it correctly and classic OCR did not either; several models answered with something credible instead of stating that they could not see it.
• In IT Lab's judgment, general image reading can be automated; fine text, brand marks and numbers require human review and an explicit "not sure" output.

The business question

Many companies receive information as images: error screenshots, product photographs, logos, scanned pages of a PDF. The question is how much of that reading can be delegated to a model running inside the company, without sending files to a third party, and in which cases a person must step in. The bench measures that boundary with real images and questions that have a checkable answer.

What the test measures and why it represents real work

The test uses 11 real images of varied types. Over them, 31 closed questions are asked about specific data points in what is visible. It is the work a person does when looking at a file and saying what it contains.

The vision family measures whether the model extracts the correct data point from an image without human intervention. It does not measure the quality of a open-ended description. IT Lab publishes what capability the test measures and what the result is for; the content of the images and of the questions is not published.

How a model is scored

The score is the percentage of correct answers. It is applied by a fixed automatic judge: a deterministic program with an acceptance list of valid answers, with no model evaluating another. The instrument is never modified to favor a model. Any adjustment to the judge must apply equally to all runs, because a correction that does not hold for every model is a patch for one.

The protocol is the same for models running on the 16 GiB card and for paid ones, with a single pass per model. One paid model was left out of the table because it attached the image but stated that it could not see it; the exclusion follows that statement, not a low score.

Results

One pass per model. IT Lab Bench, edition 1.

Five of the six eligible models score 97 or higher. The two best local models match the paid ones at 97. The fastest local model completed the whole test in 7.0 s with 87. Differences of one or two correct answers among the models at the top of the table fall within the noise of a single pass and do not establish that one is superior to another.

Where vision fails: tiny text engraved on an object

The most demanding question in the test asked for reading tiny text engraved on a photographed object. No model read it correctly. Three of them, instead of stating that they could not make it out, answered with a credible name that fit the category of the object but not the data point. Ground truth was fixed by manually enlarging a crop of the photograph, not by any automatic means.

The classic OCR used by the bench could not read it either. From that case, IT Lab draws three conclusions.

1. The threshold for small text on physical objects remains open for both models and OCR. When a process depends on engraved text captured in photographs, it needs a person or a better photograph.
2. A model may fill in something plausible. This is a failure mode OCR does not have: OCR returns noise or nothing, while a model can produce an answer that looks right. In a workflow, that is a process risk.
3. A model contributes what OCR cannot. Deciding what is and is not a relevant document is a context judgment. OCR hands over plain text and leaves that decision to a person. The comparison is developed for Chilean tax documents in Local AI for receipts and invoices.

Quality control on the instrument: each image next to its questions

In the official format of the test, each image is presented next to the questions that belong to it. Sending all images first and the questions afterwards induces association errors, in which the model answers about one image using another. The protocol requires the interleaved format so that the score reflects the ability to see, not the ability to keep an index. It is a control on the instrument: the test must measure the model, not the setup.

What this means for a Chilean company

In IT Lab's judgment, and based on these measurements:

• Automatable today with good scores: extracting on-screen text, identifying logos and colors, reading visible prices and figures, describing error screenshots and classifying scanned documents.
• With a person reviewing: small text engraved or printed on objects, photographed in poor light, and any data point that cannot tolerate a guess.
• Process requirement: if the system can produce a credible answer instead of stating that it cannot see, the workflow should include a mandatory "not sure" field and a review of the cases that use it. Without that exit, the model fills gaps with an appearance of confidence.
• Hardware: the two local models at 97 run on a 16 GiB card, a configuration within reach of a small company.

The detail of the vision family, alongside document reading, agentic work and applied reasoning, is in Technical report 1: office capabilities. The general method is described in How IT Lab evaluates a local AI model.

Frequently asked questions

Can local AI read images, screenshots and scanned documents? Yes: reading screenshots, logos, prices and visible text is handled by local models at the level of the paid ones. Five of six eligible models score 97 or more.

When does image reading fail? With tiny or degraded text, where the model may fill in something plausible instead of reading.

Can it be trusted without review? In IT Lab's judgment, general image reading can be automated; fine text, marks and numbers require human review.

Does a classic OCR read tiny text better? Not in this test: on tiny text engraved on an object, no model read it correctly and neither did classic OCR.

Limitations and scope

• A single pass per model and question; small differences are noise.
• The test consists of closed questions over 11 images. It does not measure open-ended descriptions or every type of image a company may receive.
• The official pass is kept separate from probes and diagnostics.
• The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

Measurement sheet

• Equipment: measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.
• Quantization: 3 or 4 bits, depending on the model.
• Passes: one per model.
• IT Lab Bench, edition 1.

Related technical report: Technical report 1: office capabilities

Contact: IT Lab evaluates local AI models for Chilean organizations using their own documents and images, without taking them out of their infrastructure. Let's talk

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.