ITLab Bench 02.5: 10 AI models across 49 real-world cases

Eight local AI models and two paid ones worked through the same set of real tasks. A local model, running on a single consumer graphics card, tops the overall score; the best paid model finishes second, two points behind.

• A statistical tie at the top. Swift Flash Next IQ3_S scores 94.6, followed by Claude Haiku 5.5 (92.6), Qwen 3.8 Flash Next SC117 IQ2_XS (92.5) and Swift SC117 IQ3_XXS (89.1). First and fourth are 5.5 points apart, and second and third just 0.1: with one run per case, those gaps don't separate anyone.
• The best model for office work is local. Swift 27B IQ3_S passed more checks than any other model in analysis and office work (49 of 54; 90.7), one more than Claude Haiku 5.5 and Qwen 3.8 Flash Next SC117 IQ3_XXS.
• Paying doesn't buy you the top spot. The two paid models finished second and ninth. What splits the field most is the Rust and Rails software project: five models complete it fully and five don't.

What the ITLab Bench is

The ITLab Bench is ITLab's in-house test for evaluating AI models on real office, development and analysis work. This article uses ITLab Bench, version 02.5, the current version of the instrument: a single test that keeps being refined and serves as the baseline for every new model we measure. Earlier studies, such as Swift 27B versus two Flash Next models, belong to a previous testing stage and are not mixed with these figures.

We measured 10 models on 49 cases each, with a single final run per model and case. The eight local models ran on one 16 GiB consumer graphics card, in quantized (compressed) builds so they would fit in its memory, at temperature 1.0 with reasoning set to medium. The two paid models, Claude Haiku 5.5 and GPT-6 Luna, were called with each provider's native settings.

Why it matters: a local model can work through contracts, spreadsheets or source code without that data ever leaving the organization. The question this bench answers is how much you give up, if anything, by working that way on hardware a mid-sized Chilean company can actually afford.

The cases fall into six series that count toward the total:

• N, analysis and office work (35%): 10 exercises with three cases each (a, b and c) and one with a single case; 31 cases in all.
• C, programming (20%): 3 Rust exercises checked by automated tests.
• M, software project (20%): 2 projects combining Rust and Rails.
• A, agentic (10%): 1 exercise with simulated tools, over three episodes.
• Vision (10%): 1 test of 31 questions about images and 1 photographed-receipt exercise with three cases.
• L, reading (5%): 1 table-reading exercise.

Three more exercises (N13, N14 and M08) are run and reported separately, but don't count toward the total because they only check part of the task. GPT-OSS 20B doesn't accept images, so it sits out the Vision series: it doesn't get a zero; its total is calculated over the other five series instead.

How the score works

In series C, M, A, Vision and L, the score is built from the bottom up with the weights set by the test's authors, on four levels: check, case, exercise and series. At every level, what matters more in real work weighs more. Series N is the exception: there, every check counts the same (point 6).

1. Checks. Each case is graded by several automated checks (is the total added up correctly? does it cite the source? does it pass the test?). Each check has a weight within its exercise, set by what it measures and how much damage that mistake would do in a real office. Each check also carries a label: critical, important or normal.
2. Critical failures. If a critical check fails, the case scores 0, even if everything else is right. Examples of critical checks: inventing a month that isn't in the data, counting the same receipt twice, letting a code injection through, or making a false legal statement.
3. Format gates. Some checks only verify that the answer is usable: that it can be parsed, that it has the requested structure, or that the code compiles. These gates add no points, but if one fails, everything that depends on it scores 0.
4. Checks not run. If a model's mistake prevents a check from running (for example, the code doesn't compile so the tests never execute), that check scores 0: it follows from the model's error, not from an omission by the grader.
5. Cases and exercises. An exercise's score is the average of its cases a, b and c.
6. Series N, by checks passed. In N, every check counts the same: the score is the number of checks passed divided by 54, times 100. There are no per-exercise or per-check weights, no partial credit, and a critical failure doesn't void the case. An exercise with more checks weighs more simply because it has more of them. Format gates add no points, and checks not run or indeterminate count toward the total as not passed.
7. Exercises within the series. In series C, M and Vision, each exercise carries the weight set by the authors (the tables for each series, below, show it); A and L have a single exercise.
8. Series in the total. The series weigh N 35%, C 20%, M 20%, A 10%, Vision 10% and L 5%, based on the work that matters most to ITLab.

Three exercises use partial credit per check, with no pass threshold: reading (L01), the 31-question vision test (V01) and the agentic exercise (A01). In them, each check contributes its share even if others fail.

A minimal example in N. If an office exercise has 4 checks and the model passes 3, those 3 passes and the 4 checks are added to the rest of the series; the N score is the total passed over the series' 54 checks.

Transparency. The test's authors set the weights already knowing the results of this round. So that isn't a blank check, every weight is justified by what the check measures and what its error costs, and we also publish the ranking with all weights equal (the "Equal weights" column in the overall ranking). The weights are recorded in a file whose SHA256 fingerprint starts with `e0669dcfc517`¹; any change to the weights changes that fingerprint.

The series weights are ITLab's call, not a universal truth. An organization that mostly uses AI to read documents or to write code can reweight them and will get a different order.

Overall ranking

The top four are 5.5 points apart, and second and third 0.1: with one run per case, that's a statistical tie. All four tie on programming, the software project, vision and reading; what separates them is office work and the agentic series, where Claude Haiku 5.5 and Swift SC117 IQ3_XXS score 72.3 and 76.0, against 99.3 for the best. There are 3.2 points between fourth and fifth, driven by the software project, though Swift 27B IQ3_S wins some of it back with its lead in office work: none of the models from fifth place down, except GPT-6 Luna, completed it fully.

With equal weights the same four models stay on top, and first, third, fifth, seventh, ninth and tenth stay put. Claude Haiku 5.5 drops from second to fourth and Swift SC117 IQ3_XXS climbs from fourth to second; Gemma 26B climbs from eighth to sixth, because with equal weights its many partial successes on the calculator project count for more, and Qwen 3.8 Flash Next SC117 IQ3_XXS drops from sixth to eighth.

Some names look alike but are different models. The Flash Next variants differ by quantization, shown in the name; Swift 27B IQ3_S is a separate model from Swift Flash Next IQ3_S.

Results by series

Each series gets its own table, ranked from best to worst; models with the same score share a rank. "Checks passed" counts the substantive checks fully passed in the series, excluding format gates; in N it is also the basis of the score. With a single run per case, gaps of one or two points are noise.

N: analysis and office work (35% of the total)

Tests whether a model handles office tasks on real material and answers only with what the material says. It's the series closest to a company's daily work: office documents, business, customer service, legal support, systems maintenance and a small application. In N, every check counts the same: the score is the percentage of the 54 checks passed, and an exercise with more checks weighs more simply because it has more of them.

The "Critical failures" column counts the cases in which a check labeled critical by the test's authors failed; it is for information only and doesn't change the N score.

Swift 27B IQ3_S leads with one check more than Claude Haiku 5.5 and Qwen 3.8 Flash Next SC117 IQ3_XXS, and two more than Swift Flash Next IQ3_S: the four sit within 3.7 points, a margin a single run can't order with confidence. Fifth through last are six checks apart; Gemma 26B finishes last, with 38 of 54.

C: programming (20% of the total)

Tests whether the model's Rust code passes automated tests it never sees. Useful for choosing a coding assistant, including for secure coding.

Seven models solve all three exercises without errors. GPT-6 Luna, one of the two paid models, finishes last in this series.

M: software project (20% of the total)

Tests whether a model can carry a full two-language project, in Rust and Rails, whose parts have to fit together and pass tests. Useful for choosing a model for multi-part projects, not just standalone functions.

This is the series that splits the field most. Everyone solves the installment engine (M09); the difference is the calculator (M10). Gemma 26B gets nearly all of the calculator engine right but doesn't reach the Rails console or the end-to-end tests; the bottom four score no checks on it.

A: agentic (10% of the total)

Tests whether the model uses tools in the right order and completes the task across three episodes: diagnosing an incident, computing a total from several sources and deploying in line with a policy. Useful for choosing a model that operates tools on its own. It's a single exercise (A01) with partial credit: in each episode, the delivered result weighs 70, economy of steps 15 and protocol (well-formed turns) 15.

Four models sit at 99.3, the highest score reached: none got to 100 because all of them used more steps than needed in the first episode (the optimum is four), and the process criterion gives partial credit. Checks passed don't follow the same order as the score, because they don't all weigh the same: the delivered result is worth 70 and the process and the protocol 15 each, with partial credit. That is why GPT-OSS 20B, with 4 of 9 fully passed, sits at 90.7, and Claude Haiku 5.5, with 7 of 9, at 72.3: it failed one of the heaviest checks. With a single exercise and one pass, read it as a trend, not a verdict.

Vision (10% of the total)

Covers reading images and photographed documents. Useful for choosing a model that reads receipts, document photos, charts or screens.

Critical failures: cases in this series voided by a critical failure. GPT-OSS 20B does not take part.

Seven models tie: they get all 31 image questions right and miss the document reading on the same receipt case. GPT-6 Luna and Gemma 26B miss further checks and drop about 20 points. GPT-OSS 20B doesn't accept images: it has no vision score, isn't penalized, and its total is calculated over the other five series.

L: reading (5% of the total)

Tests whether the model accurately extracts the data from a set of tables and answers questions about it. Useful for choosing a model that reads reports, listings or org charts. It's a single exercise (L01) with partial credit: extracted data weighs 60 and answers 40.

Seven models get everything right. The other three miss one to three checks, all of them questions, which weigh more than a single cell. This series barely tells models apart, which is why it carries the lowest weight.

For information only: N13, N14 and M08

These three exercises are run and scored under the same rules, but they don't count toward their series or the total, because they only check part of the task. N13 measures translation into Chilean Spanish while keeping figures, register and ambiguities; N14, rewriting a customer message in a formal tone without changing facts, amounts or degree of certainty; M08, a web admin interface, of which we only measure that it follows the brief's security rules and integrates into the repository.

They are ordered from the model that solves the three best to the one that solves them worst, by the average of the three; ties go to the model that solved them faster. Time is the sum of inference time over the seven cases (three of N13, three of N14 and M08).

Which model for which job

• Balanced general use: Swift Flash Next IQ3_S, first overall, with top marks in programming, the software project and reading, and among the best in agentic and vision. Claude Haiku 5.5, Qwen 3.8 Flash Next SC117 IQ2_XS and Swift SC117 IQ3_XXS are in a statistical tie with it.
• Office work and analysis (N): Swift 27B IQ3_S, with the most checks passed, although Claude Haiku 5.5, Qwen 3.8 Flash Next SC117 IQ3_XXS and Swift Flash Next IQ3_S are one or two checks behind. Note that it didn't solve the calculator in the software project, so it isn't the best pick if you also need multi-part software projects.
• Rust programming (C): seven models with no errors; any of them will do.
• Multi-part Rust and Rails projects (M): the five that completed it fully: three local (Swift Flash Next IQ3_S, Qwen 3.8 Flash Next SC117 IQ2_XS and Swift SC117 IQ3_XXS) and both paid models.
• Agentic (A): four models reach 99.3 and three sit between 72.3 and 76.0; since it's a single exercise, the gap between them isn't conclusive.
• Vision and reading: seven models tie at the top of each; these series have few exercises and shouldn't decide the choice on their own.
• Systems maintenance: currently covered by N07 (a single case), N08 and partly A01. That isn't enough to recommend a model; a dedicated exercise is planned.

Failure patterns

The same mistakes show up in local and paid models alike:

• Output format. Broken data structures, or fields with the wrong data type, that an automated integration can't consume.
• Figures. Monthly totals that are slightly off, near-duplicate rows handled incorrectly, or months added that don't exist in the source.
• Unsafe reading of ambiguity. Faced with a contradictory requirement, several models chose the interpretation that allowed an invalid state instead of blocking it.
• Security finding classification. The most common miss was labeling a finding as real, or as a false positive, when the evidence was insufficient to decide.
• Minutes and traceability. Agreements that weren't in the transcript, and inaudible or cut-off audio that went unreported.
• Photographed document numbers. Reference numbers read with extra or missing digits.

Speed, time and tokens

These tables use the final passes of each model's 49 cases, the ones behind its score. When a case was rerun, only the final valid pass counts. Time is inference time (how long the model takes to respond), excluding loading and grading; in the agentic case it is the sum across turns. GPT-OSS 20B totals 45 cases because it doesn't accept images and didn't run the four vision cases.

Tokens generated

A token is the unit of text a model works with, on average a bit less than a word. Output tokens are everything the model writes when it answers; for models that reason, they normally include the reasoning done before answering. More tokens means more text or more reasoning, not a better answer. For a local model they are time and energy; for a paid service, cost. The table runs from the model that generated the most to the one that generated the least.

Tokens follow each engine's or provider's own accounting and don't compare work across tokenizers. For paid services, that accounting may not include all internal reasoning, so their figures are not equivalent to the local models'.

Response times

The table runs from fastest to slowest by total time.

Paid-service times include the network and the provider's queue, so they aren't comparable with the local models, which ran on a single consumer graphics card. We publish them because the total time of a full battery is also a practical figure for anyone choosing.

Reading and generation speed (local models)

The table runs from the model that reads fastest to the one that reads slowest.

These are consolidated speeds: the engine counters across all final passes are added up (tokens processed or generated over the corresponding time), without averaging averages. Reading is the speed at which the model processes the prompt, and generation is the speed at which it writes the answer. Paid services (Claude Haiku 5.5 and GPT-6 Luna) don't provide these counters, so they are not in this table; their times and tokens are in the two tables above.

More speed doesn't mean better answers: the fastest model at generation, GPT-OSS 20B, finishes last overall.

ITLab's take

This is ITLab's professional judgment based on our own measurements; it is not a certification or a buying recommendation.

1. Choose by type of work, not by rank. For analysis involving figures, sources and policy, Swift 27B IQ3_S was the most reliable; for multi-part software projects, pick a model from the group that fully completed the Rust and Rails project.
2. Take local seriously. First, third and fourth are local models, on a single consumer graphics card, without data leaving the organization; the best paid model, in second, is in a statistical tie with them.
3. Don't assume paying guarantees quality. The two paid services we measured finished second and ninth.
4. Always validate the format before using structured data generated by a model.
5. Keep a human in the loop for figures, regulations, security and any text that commits you to something with a customer.
6. Test with your own cases and, if your use differs from ours, reweight the series before deciding.

Limitations and scope

• One run per model and case. A difference of one or two cases can be noise; there's no statistical significance and no universal winner.
• Weights set with knowledge of the results. In C, M, A, Vision and L, the test's authors defined the weights after seeing this round. That's why every weight is justified by what it measures and we publish the equal-weights ranking, which leaves the same group at the top.
• Series weights set by ITLab. The total depends on what each series is worth; the mix of units across series makes that choice unavoidable.
• Few substantive checks in series N. Several office exercises are decided by one or two checks per case; refining them is a pending improvement to the bench.
• Human review pending for N10 and N13. Their results come from automated verification only.
• Single-exercise series. Reading and agentic each rest on one exercise, so their scores swing more.
• Memory ceiling. Local models ran in compressed builds to fit a 16 GiB card; that ceiling determines which version of each model can be measured.
• Reasoning. Reasoning modes aren't equivalent across model families and are not compared with each other.
• Paid versus local. Paid-service times and tokens aren't comparable with local ones, and we can't certify which weights sit behind a paid service's name.

Frequently asked questions

Can a local model replace a paid one? Largely, yes. In this round, first place went to a local model and third place sits 0.1 points behind the best paid model, and the top model for office work was also local. We still recommend human review and testing with your own cases.

How are checks and exercises weighted? In series N, every check counts the same: the score is the percentage of checks passed. In the other series, each check has a weight within the exercise based on what it measures and the damage its error would cause, and if a critical check fails, the case scores 0; in addition, each exercise carries the weight set by the authors, and each series has a weight in the total. How long a model takes has no effect on the score.

Do the weights decide the ranking? No. With all weights equal, the top four are still the same four models; Claude Haiku 5.5 and Swift SC117 IQ3_XXS swap places (second and fourth), as do Gemma 26B and Qwen 3.8 Flash Next SC117 IQ3_XXS (sixth and eighth).

Why doesn't GPT-OSS 20B have a vision score? Because it doesn't accept images. It isn't penalized; its total is calculated over the other five series.

Can I use different series weights? Yes. The weights reflect ITLab's needs; an organization with a different use case can reweight them and get a different order.

What happens with new models? They'll be evaluated with ITLab Bench, version 02.5, and the same weights, so their results are comparable with the ones in this article.

Contact

If your organization is considering local AI with limited resources, ITLab can help you choose a model for your type of work and design tests built on your real cases. Contact us.

¹ SHA256 of the weights file: `e0669dcfc5172b8b8526ab8f3e6f264686ca79c7d5bbcff44f85fec38dddd92a`.

The ITLab Bench is a living instrument that we keep refining. The conclusions reflect ITLab's judgment based on its own measurements.