Unsloth and Avalon IQ3_KS on the ITLab Bench
We tested Unsloth UD-IQ4_XS and Avalon IQ3_KS with Swift as a reference: Unsloth ranks first of 17, Avalon ninth. Tables by series and what changed in the test.
We measured two new builds of Qwen3.8 Flash Next on the 49 cases of the ITLab Bench, version 02.5, on the same machine and with Swift Flash Next as a reference. Unsloth UD-IQ4_XS scores 95.9 points and ranks first of 17 configurations; Avalon IQ3_KS scores 90.5 and ranks ninth. With one run per case, the top places are a statistical tie.
- Unsloth, first. It scores 95.9 points: 0.3 above GPT-6.1 Sol + Pi (95.6) and 1.2 above Swift on the same engine (94.7).
- Avalon, ninth because of one case. It ties Unsloth on office work (49 of 54) but fails C03, worth 6 points of the total: its Rust code doesn't compile because it calls `String::reverse`. With that case passed it would score 96.5.
- Same weights. The 54 office checks, the 418 substantive checks and the weights match the earlier reports. We explain below what changed in the test.
- The engines, separately. What it took to get each package running is in Avalon and Strata on 16 GB: our experience.
What we measured
Three configurations of Qwen3.8 Flash Next ran the full battery: 49 tasks covering reasoning, code, vision, long context and agentic work. Each task ran once and was evaluated before moving on. The agents under test don't see the rubrics.
Each row is a package of model plus engine. The IQ3_KS model only loads in Avalon, so these results don't separate the effect of one from the other.
| Swift + Strata | Unsloth + Strata | Avalon IQ3_KS + Avalon | |
|---|---|---|---|
| Model | Swift 1.5 Flash Next GSQ-RCO abliterated IQ3_S | Unsloth UD-IQ4_XS V3.1 | Flash Next Avalon IQ3_KS |
| Engine | Strata 0.2.10 | Strata 0.2.10 | Avalon, commit 16de68d28 |
| Context | 262,144 | 262,144 | 262,144 |
| Reasoning | medium | medium | medium |
| Temperature / top_p / top_k | 1.0 / 0.9 / 20 | 1.0 / 0.95 / 20 | 1.0 / 0.9 / 20 |
| Speculative decoding (MTP) | 3 drafts | 3 drafts | 2 drafts |
| KV cache | int8 | int8 | q8_0 / q4_0 |
| Vision | Own projector, CPU | Unsloth projector, CPU | Unsloth projector, CPU |
The machine is the same as in earlier measurements: RX 9070 XT with 16 GB, Ryzen 7 5700X, 94 GB of RAM and NVMe, on Arch Linux. Only one model was on the card at a time, with the CPU capped at 80% and the desktop in normal use. All three configurations finished the battery with no infrastructure errors.
Scoring follows the same rules as the earlier reports:
- Series N (analysis and office work): every check counts the same; the score is the percentage of the 54 passed.
- Series C, M, A, Vision and L: scored with the per-exercise and per-check weights set by the test's authors.
- Total: N 35%, C 20%, M 20%, A 10%, Vision 10% and L 5%.
The full method is in the 10-model report.
What changed in the test
This version adds 8 checks to the battery, which goes from 534 to 542. All of them are format gates, which check whether the delivery is usable and add no points:
- N14: 3 delivery gates, one per case. It's an informational exercise that doesn't count toward the total, and we already noted this rule in the GPT-6.1 Terra note and the Pi agent note.
- C01, C02, C03, M09 and M10: one delivery gate each, separate from the compile gate.
That's why the tables can be read together: they're still the same 54 office checks and the same 418 substantive checks, with the same weights. The three new configurations passed all 8 gates, and recalculating their scores without them gives the same result.
The 14 earlier rows keep their published scores. They have the N14 gates, but the 5 C and M gates weren't applied to their responses, except the M10 gate on Swift + Pi + Ponytail.
There's one more difference in conditions. The new configurations ran with a system instruction asking the model to finish reasoning and deliver a visible final answer. Nine of the earlier rows keep 38 to 41 cases answered before that instruction existed. We didn't measure its effect.
Overall ranking
| Rank | Model | Type | Total | Measured |
|---|---|---|---|---|
| 1 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 95.9 | November |
| 2 | GPT-6.1 Sol + Pi | Paid, with Pi | 95.6 | October |
| 3 | GPT-6.1 Sol | Paid | 95.0 | October |
| 4 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 94.7 | November |
| 5 | Swift Flash Next IQ3_S | Local | 94.6 | October |
| 6 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 94.3 | October |
| 7 | Claude Haiku 5.5 | Paid | 92.6 | October |
| 8 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 92.5 | October |
| 9 | Avalon IQ3_KS + Avalon | Local | 90.5 | November |
| 10 | Swift SC117 IQ3_XXS | Local | 89.1 | October |
| 11 | Swift 27B IQ3_S | Local | 85.9 | October |
| 12 | GPT-6.1 Terra | Paid | 84.5 | October |
| 13 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 82.7 | October |
| 14 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 82.5 | October |
| 15 | Gemma 26B | Local | 81.2 | October |
| 16 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 76.8 | October |
| 17 | GPT-6 Luna | Paid | 74.9 | October |
| — | GPT-OSS 20B | Local | No comparable total (doesn't accept images) | — |
| — | GPT-OSS 20B + Pi | Local, with Pi | no comparable total (70.1 without Vision) | — |
Rows in bold are the two models tested; Swift + Strata 0.2.10 is the reference. The "Measured" column separates this round's rows from those published in October.
The top five are within 1.3 points: with one run per case, there's no clear winner among them. Swift appears twice, at 94.7 and 94.6. It's the same model on another version of the engine and the battery, and it changed on several checks without our being able to pin that on one cause.
Avalon IQ3_KS ranks ninth because of programming: at 70.0 in C it gives up 6 points to a model with a perfect score. In the other series it's at or near the top.
Another possible order: every check counts the same
This order counts the checks passed across all series, as if each were worth the same. It takes in the 418 substantive checks of the six series. Format gates and the informational exercises N13, N14 and M08 stay out. GPT-OSS 20B records 375 because it has no Vision, so it has no rank.
| Rank | Configuration | Type | Checks passed (%) | Passed of recorded |
|---|---|---|---|---|
| 1 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 98.3 | 411 of 418 |
| 2 | GPT-6.1 Sol | Paid | 98.1 | 410 of 418 |
| 2 | GPT-6.1 Sol + Pi | Paid, with Pi | 98.1 | 410 of 418 |
| 4 | Claude Haiku 5.5 | Paid | 97.8 | 409 of 418 |
| 4 | Swift Flash Next IQ3_S | Local | 97.8 | 409 of 418 |
| 4 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 97.8 | 409 of 418 |
| 7 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 97.6 | 408 of 418 |
| 7 | Avalon IQ3_KS + Avalon | Local | 97.6 | 408 of 418 |
| 9 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 96.9 | 405 of 418 |
| 10 | Swift SC117 IQ3_XXS | Local | 96.4 | 403 of 418 |
| 11 | GPT-6.1 Terra | Paid | 94.5 | 395 of 418 |
| 12 | Gemma 26B | Local | 91.1 | 381 of 418 |
| 12 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 91.1 | 381 of 418 |
| 14 | GPT-6 Luna | Paid | 89.7 | 375 of 418 |
| 15 | Swift 27B IQ3_S | Local | 81.3 | 340 of 418 |
| 16 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 79.9 | 334 of 418 |
| 17 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 78.9 | 330 of 418 |
| — | GPT-OSS 20B + Pi (no Vision) | Local, with Pi | 89.1 | 334 of 375 |
| — | GPT-OSS 20B (no Vision) | Local | 72.8 | 273 of 375 |
Avalon moves up from ninth to seventh because C03 accounts for only 3 of the 418 checks. Unsloth stays first, one check ahead of GPT-6.1 Sol.
Results by series
Each table runs from highest to lowest score; ties share a rank. "Checks passed" counts the substantive checks passed in the series, without format gates.
N: analysis and office work (35% of the total)
Measures whether the model handles office tasks with real material and answers only from what the material says. Every check counts the same.
| Rank | Model | Type | Series score | Checks passed | Critical failures |
|---|---|---|---|---|---|
| 1 | Swift 27B IQ3_S | Local | 90.7 | 49 of 54 | 1 |
| 1 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 90.7 | 49 of 54 | 1 |
| 1 | Avalon IQ3_KS + Avalon | Local | 90.7 | 49 of 54 | 1 |
| 4 | Claude Haiku 5.5 | Paid | 88.9 | 48 of 54 | 1 |
| 4 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 88.9 | 48 of 54 | 1 |
| 4 | GPT-6.1 Sol | Paid | 88.9 | 48 of 54 | 3 |
| 4 | GPT-6.1 Sol + Pi | Paid, with Pi | 88.9 | 48 of 54 | 1 |
| 8 | Swift Flash Next IQ3_S | Local | 87.0 | 47 of 54 | 1 |
| 8 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 87.0 | 47 of 54 | 1 |
| 10 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 85.2 | 46 of 54 | 2 |
| 11 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 81.5 | 44 of 54 | 2 |
| 11 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 81.5 | 44 of 54 | 2 |
| 11 | GPT-6.1 Terra | Paid | 81.5 | 44 of 54 | 1 |
| 14 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 79.6 | 43 of 54 | 3 |
| 15 | Swift SC117 IQ3_XXS | Local | 77.8 | 42 of 54 | 2 |
| 15 | GPT-OSS 20B + Pi | Local, with Pi | 77.8 | 42 of 54 | 3 |
| 17 | GPT-6 Luna | Paid | 75.9 | 41 of 54 | 2 |
| 18 | GPT-OSS 20B | Local | 74.1 | 40 of 54 | 3 |
| 19 | Gemma 26B | Local | 70.4 | 38 of 54 | 2 |
Critical failures: cases where a check labeled critical by the test's authors failed; it's informational and doesn't change the N score.
Unsloth and Avalon tie at the top with Swift 27B IQ3_S. Swift + Strata ranks tenth: three of its eight unpassed checks didn't run because of a format error in a single case.
C: programming (20% of the total)
Measures whether Rust code passes automated tests the model doesn't see.
| Rank | Model | Type | Series score | Checks passed |
|---|---|---|---|---|
| 1 | GPT-6.1 Sol | Paid | 100 | 34 of 34 |
| 1 | Swift Flash Next IQ3_S | Local | 100 | 34 of 34 |
| 1 | Claude Haiku 5.5 | Paid | 100 | 34 of 34 |
| 1 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 100 | 34 of 34 |
| 1 | Swift SC117 IQ3_XXS | Local | 100 | 34 of 34 |
| 1 | Swift 27B IQ3_S | Local | 100 | 34 of 34 |
| 1 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 100 | 34 of 34 |
| 1 | Gemma 26B | Local | 100 | 34 of 34 |
| 1 | GPT-6.1 Sol + Pi | Paid, with Pi | 100 | 34 of 34 |
| 1 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 100 | 34 of 34 |
| 1 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 100 | 34 of 34 |
| 1 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 100 | 34 of 34 |
| 1 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 100 | 34 of 34 |
| 14 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 87.1 | 25 of 34 |
| 15 | Avalon IQ3_KS + Avalon | Local | 70.0 | 31 of 34 |
| 16 | GPT-6.1 Terra | Paid | 60.0 | 24 of 34 |
| 16 | GPT-OSS 20B | Local | 60.0 | 24 of 34 |
| 16 | GPT-OSS 20B + Pi | Local, with Pi | 60.0 | 24 of 34 |
| 19 | GPT-6 Luna | Paid | 44.3 | 13 of 34 |
This is where Avalon IQ3_KS falls behind. C03 is worth 30% of the series, and its delivery doesn't compile. Swift passed that same case, using 156,128 tokens and 39 minutes.
M: software project (20% of the total)
Measures whether the model can sustain a full Rust and Rails project whose parts must integrate and pass tests.
| Rank | Model | Type | Series score | Checks passed |
|---|---|---|---|---|
| 1 | GPT-6.1 Sol | Paid | 100 | 83 of 83 |
| 1 | Swift Flash Next IQ3_S | Local | 100 | 83 of 83 |
| 1 | Claude Haiku 5.5 | Paid | 100 | 83 of 83 |
| 1 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 100 | 83 of 83 |
| 1 | Swift SC117 IQ3_XXS | Local | 100 | 83 of 83 |
| 1 | GPT-6.1 Terra | Paid | 100 | 83 of 83 |
| 1 | GPT-6 Luna | Paid | 100 | 83 of 83 |
| 1 | GPT-6.1 Sol + Pi | Paid, with Pi | 100 | 83 of 83 |
| 1 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 100 | 83 of 83 |
| 1 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 100 | 83 of 83 |
| 1 | Avalon IQ3_KS + Avalon | Local | 100 | 83 of 83 |
| 1 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 100 | 83 of 83 |
| 13 | GPT-OSS 20B + Pi | Local, with Pi | 76.7 | 73 of 83 |
| 14 | Gemma 26B | Local | 73.8 | 69 of 83 |
| 15 | Swift 27B IQ3_S | Local | 50.0 | 12 of 83 |
| 15 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 50.0 | 12 of 83 |
| 15 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 50.0 | 12 of 83 |
| 15 | GPT-OSS 20B | Local | 50.0 | 12 of 83 |
| 19 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 25.0 | 60 of 83 |
All three configurations complete the project.
A: agentic (10% of the total)
Measures whether the model uses simulated tools in the right order. It's a single exercise with partial credit.
| Rank | Model | Type | Series score | Checks passed |
|---|---|---|---|---|
| 1 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 100 | 9 of 9 |
| 2 | Swift Flash Next IQ3_S | Local | 99.3 | 8 of 9 |
| 2 | Swift 27B IQ3_S | Local | 99.3 | 8 of 9 |
| 2 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 99.3 | 8 of 9 |
| 2 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 99.3 | 8 of 9 |
| 2 | GPT-6.1 Terra | Paid | 99.3 | 8 of 9 |
| 2 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 99.3 | 8 of 9 |
| 8 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 98.7 | 8 of 9 |
| 8 | Avalon IQ3_KS + Avalon | Local | 98.7 | 8 of 9 |
| 10 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 97.7 | 7 of 9 |
| 11 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 97.0 | 8 of 9 |
| 12 | GPT-6.1 Sol | Paid | 96.3 | 8 of 9 |
| 12 | GPT-6.1 Sol + Pi | Paid, with Pi | 96.3 | 8 of 9 |
| 14 | Gemma 26B | Local | 96.0 | 7 of 9 |
| 15 | GPT-OSS 20B | Local | 90.7 | 4 of 9 |
| 16 | Swift SC117 IQ3_XXS | Local | 76.0 | 7 of 9 |
| 17 | GPT-6 Luna | Paid | 75.3 | 7 of 9 |
| 18 | Claude Haiku 5.5 | Paid | 72.3 | 7 of 9 |
| 19 | GPT-OSS 20B + Pi | Local, with Pi | 37.3 | 1 of 9 |
Swift + Strata is the only row with a perfect score.
Vision (10% of the total)
Measures reading images (V01) and photographed receipts (N02).
| Rank | Model | Type | Series score | Checks passed | Critical failures |
|---|---|---|---|---|---|
| 1 | GPT-6.1 Sol + Pi | Paid, with Pi | 98.4 | 42 of 43 | 0 |
| 1 | Avalon IQ3_KS + Avalon | Local | 98.4 | 42 of 43 | 0 |
| 1 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 98.4 | 42 of 43 | 0 |
| 4 | GPT-6.1 Sol | Paid | 92.5 | 42 of 43 | 0 |
| 4 | Swift Flash Next IQ3_S | Local | 92.5 | 42 of 43 | 0 |
| 4 | Claude Haiku 5.5 | Paid | 92.5 | 42 of 43 | 0 |
| 4 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 92.5 | 42 of 43 | 0 |
| 4 | Swift SC117 IQ3_XXS | Local | 92.5 | 42 of 43 | 0 |
| 4 | Swift 27B IQ3_S | Local | 92.5 | 42 of 43 | 0 |
| 4 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 92.5 | 42 of 43 | 0 |
| 4 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 92.5 | 42 of 43 | 0 |
| 4 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 92.5 | 42 of 43 | 0 |
| 13 | GPT-6.1 Terra | Paid | 90.9 | 41 of 43 | 0 |
| 13 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 90.9 | 41 of 43 | 0 |
| 13 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 90.9 | 41 of 43 | 0 |
| 16 | GPT-6 Luna | Paid | 74.2 | 39 of 43 | 1 |
| 17 | Gemma 26B | Local | 72.6 | 38 of 43 | 1 |
| — | GPT-OSS 20B | Local | Not supported | — | — |
| — | GPT-OSS 20B + Pi | Local, with Pi | Not supported | — | — |
Critical failures: cases in the series where a critical check failed.
All three pass 42 of 43, but Unsloth scores lower because its miss falls on the receipts, an exercise with fewer checks and the same weight as V01.
L: reading (5% of the total)
Measures whether the model accurately extracts data from a five-table org chart and answers questions about it.
| Rank | Model | Type | Series score | Checks passed |
|---|---|---|---|---|
| 1 | GPT-6.1 Sol | Paid | 100 | 195 of 195 |
| 1 | Swift Flash Next IQ3_S | Local | 100 | 195 of 195 |
| 1 | Claude Haiku 5.5 | Paid | 100 | 195 of 195 |
| 1 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 100 | 195 of 195 |
| 1 | Swift SC117 IQ3_XXS | Local | 100 | 195 of 195 |
| 1 | Swift 27B IQ3_S | Local | 100 | 195 of 195 |
| 1 | GPT-6.1 Terra | Paid | 100 | 195 of 195 |
| 1 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 100 | 195 of 195 |
| 1 | Gemma 26B | Local | 100 | 195 of 195 |
| 1 | GPT-6.1 Sol + Pi | Paid, with Pi | 100 | 195 of 195 |
| 1 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 100 | 195 of 195 |
| 1 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 100 | 195 of 195 |
| 1 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 100 | 195 of 195 |
| 1 | Avalon IQ3_KS + Avalon | Local | 100 | 195 of 195 |
| 1 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 100 | 195 of 195 |
| 16 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 96.7 | 194 of 195 |
| 16 | GPT-OSS 20B + Pi | Local, with Pi | 96.7 | 194 of 195 |
| 18 | GPT-OSS 20B | Local | 93.3 | 193 of 195 |
| 19 | GPT-6 Luna | Paid | 90.0 | 192 of 195 |
For information only: N13, N14 and M08
These three exercises are run and scored but don't count toward the total, because they only check part of the task: N13, translation into Chilean Spanish; N14, a formal rewrite of a message to a client; M08, security rules for an admin interface.
| Rank | Model | Type | N13 | N14 | M08 | Average | Total time (min:s) |
|---|---|---|---|---|---|---|---|
| 1 | GPT-6.1 Sol + Pi | Paid, with Pi | 100 | 100 | 100 | 100 | 2:38 |
| 2 | GPT-6.1 Sol | Paid | 100 | 100 | 100 | 100 | 2:39 |
| 3 | GPT-6.1 Terra | Paid | 66.7 | 100 | 100 | 88.9 | 0:41 |
| 4 | Swift SC117 IQ3_XXS | Local | 66.7 | 100 | 100 | 88.9 | 4:52 |
| 5 | Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 66.7 | 100 | 100 | 88.9 | 5:28 |
| 6 | Swift 27B IQ3_S | Local | 66.7 | 100 | 100 | 88.9 | 6:22 |
| 7 | Swift Flash Next IQ3_S | Local | 66.7 | 100 | 100 | 88.9 | 6:46 |
| 8 | Gemma 26B | Local | 66.7 | 100 | 100 | 88.9 | 7:07 |
| 9 | Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 66.7 | 100 | 100 | 88.9 | 13:06 |
| 10 | Swift Flash Next IQ3_S + Pi (initial run) | Local, with Pi | 100 | 66.7 | 100 | 88.9 | — |
| 11 | GPT-6 Luna | Paid | 66.7 | 66.7 | 100 | 77.8 | 0:35 |
| 12 | GPT-OSS 20B | Local | 33.3 | 100 | 100 | 77.8 | 1:56 |
| 13 | Claude Haiku 5.5 | Paid | 100 | 33.3 | 100 | 77.8 | 2:52 |
| 14 | Swift Flash Next IQ3_S + Pi + Ponytail (combined) | Local, with Pi | 66.7 | 66.7 | 100 | 77.8 | 4:10 |
| 15 | Avalon IQ3_KS + Avalon | Local | 100 | 33.3 | 100 | 77.8 | 5:33 |
| 16 | Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 66.7 | 66.7 | 100 | 77.8 | 7:18 |
| 17 | Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 66.7 | 66.7 | 100 | 77.8 | 9:07 |
| 18 | GPT-OSS 20B + Pi | Local, with Pi | 66.7 | 66.7 | 100 | 77.8 | — |
| 19 | Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 66.7 | 100 | 0 | 55.6 | 6:20 |
They're sorted by the average of the three; ties go to the fastest. Time is the sum of inference over the seven cases; two Pi configurations have no comparable time, shown as "—".
Speed, time and tokens
Earlier rows keep their published figures. For the three new rows, tokens and time are the sum of the engine's counters over the 49 tasks. Accounting differs between campaigns, so the overall order is only a guide.
Tokens generated
Output tokens normally include the reasoning before the answer. The table runs from the fewest generated to the most.
| Model | Type | Cases | Output tokens | Average per case |
|---|---|---|---|---|
| GPT-6 Luna | Paid | 49 | 22,661 | 462 |
| GPT-6.1 Terra | Paid | 49 | 24,942 | 509 |
| GPT-6.1 Sol | Paid | 49 | 40,527 | 827 |
| GPT-6.1 Sol + Pi | Paid, with Pi | 49 | 45,666 | 932 |
| GPT-OSS 20B | Local | 45 | 116,633 | 2,592 |
| GPT-OSS 20B + Pi | Local, with Pi | 45 | 122,586 | 2,724 |
| Swift Flash Next IQ3_S + Pi + Ponytail (original campaign) | Local, with Pi | 49 | 148,570 | 3,032 |
| Swift 27B IQ3_S | Local | 49 | 183,442 | 3,744 |
| Swift Flash Next IQ3_S | Local | 49 | 192,293 | 3,924 |
| Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 49 | 192,729 | 3,933 |
| Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 49 | 199,410 | 4,070 |
| Swift SC117 IQ3_XXS | Local | 49 | 207,099 | 4,227 |
| Claude Haiku 5.5 | Paid | 49 | 277,780 | 5,669 |
| Gemma 26B | Local | 49 | 321,057 | 6,552 |
| Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 49 | 344,117 | 7,023 |
| Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 49 | 361,764 | 7,383 |
| Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 49 | 376,319 | 7,680 |
| Avalon IQ3_KS + Avalon | Local | 49 | 458,538 | 9,358 |
Swift spent 156,128 tokens on C03 alone. Avalon IQ3_KS generated more tokens than Swift in 30 of the 49 cases.
Response times
The table runs from fastest to slowest by total time.
| Model | Type | Cases | Total response time | Average per case |
|---|---|---|---|---|
| GPT-6 Luna | Paid | 49 | 0:04:51 | 5.9 s |
| GPT-6.1 Terra | Paid | 49 | 0:06:29 | 7.9 s |
| GPT-OSS 20B | Local | 45 | 0:12:58 | 17.3 s |
| GPT-OSS 20B + Pi | Local, with Pi | 45 | 0:14:18 | 19.1 s |
| GPT-6.1 Sol | Paid | 49 | 0:16:21 | 20.0 s |
| GPT-6.1 Sol + Pi | Paid, with Pi | 49 | 0:17:25 | 21.3 s |
| Claude Haiku 5.5 | Paid | 49 | 0:21:23 | 26.2 s |
| Swift Flash Next IQ3_S + Pi + Ponytail (original campaign) | Local, with Pi | 49 | 0:50:28 | 61.8 s |
| Qwen 3.8 Flash Next SC117 IQ3_XXS | Local | 49 | 0:50:58 | 62.4 s |
| Swift SC117 IQ3_XXS | Local | 49 | 0:51:33 | 63.1 s |
| Gemma 26B | Local | 49 | 0:52:21 | 64.1 s |
| Swift Flash Next IQ3_S | Local | 49 | 0:53:58 | 66.1 s |
| Swift 27B IQ3_S | Local | 49 | 0:56:11 | 68.8 s |
| Qwen3.8-27B-GSQ-RCO IQ3_S | Local | 49 | 1:01:36 | 75.4 s |
| Qwen 3.8 Flash Next SC117 IQ2_XS | Local | 49 | 1:16:26 | 93.6 s |
| Swift Flash Next IQ3_S + Strata 0.2.10 | Local | 49 | 1:36:40 | 118.4 s |
| Unsloth UD-IQ4_XS + Strata 0.2.10 | Local | 49 | 2:04:16 | 152.2 s |
| Avalon IQ3_KS + Avalon | Local | 49 | 2:05:15 | 153.4 s |
Paid-service times include the provider's network and queue; they aren't comparable with local ones.
Reading and generation speed (local models)
The table runs from the fastest reader to the slowest.
| Model | Reading (tok/s) | Generation (tok/s) |
|---|---|---|
| GPT-OSS 20B | 4,658.3 | 153.5 |
| GPT-OSS 20B + Pi | 4,614.8 | 152.0 |
| Qwen 3.8 Flash Next SC117 IQ2_XS | 1,055.4 | 82.7 |
| Swift Flash Next IQ3_S + Strata 0.2.10 | 951.8 | 66.3 |
| Swift SC117 IQ3_XXS | 943.1 | 78.2 |
| Qwen 3.8 Flash Next SC117 IQ3_XXS | 915.7 | 73.7 |
| Swift Flash Next IQ3_S | 844.9 | 69.3 |
| Swift Flash Next IQ3_S + Pi + Ponytail | 737.0 | 68.5 |
| Unsloth UD-IQ4_XS + Strata 0.2.10 | 613.5 | 49.8 |
| Gemma 26B | 499.4 | 110.6 |
| Qwen3.8-27B-GSQ-RCO IQ3_S | 292.9 | 61.5 |
| Swift 27B IQ3_S | 280.7 | 62.7 |
| Avalon IQ3_KS + Avalon | 223.0 | 65.5 |
Avalon's reading drops because of the four image cases: without them it reads at 551.6 tok/s, against 869.8 for Swift and 556.0 for Unsloth. On images, the engines don't read the same number of tokens and vision runs on the CPU, so that figure doesn't measure the same thing. Swift generates 33% faster than Unsloth on the same engine, although Unsloth uses a different `top_p` and a hand-built pack. The speed detail, with the control tests, is in the engine note.
What it means for choosing a model
- None of the three wins everywhere. Unsloth does best on office work, Swift on agentic work and speed on its engine, and Avalon ties on office work but loses one programming case.
- They fail different things. All three have 6 failures in scored cases, and Swift and IQ3_KS overlap on only two. One case can move several points.
- These results apply to this machine. Test with your own cases and keep a human in the loop for figures, regulations and security.
Limitations
- One run, with temperature above zero. Gaps of one or two points can be noise.
- Rows from two rounds. The 14 earlier rows ran before, on another version of the battery and, in some cases, of the engines.
- Unsloth pack. It uses `--compat-bf16`, which according to Strata's notes can reduce quality on AMD. We didn't measure that effect. The model is no longer on the machine, so repeating its figures means downloading it again.
- Refusals. The battery doesn't measure them, even though Swift is an abliterated build.
- Weights. The series weights are ITLab's call, and the weights in C, M, A, Vision and L were set by the test's authors.
Frequently asked questions
Did Unsloth beat GPT-6.1 Sol? In this table it's 0.3 points above Sol + Pi and 0.9 above Sol, with one run and in different rounds. That's a statistical tie, not a proven lead.
Why does Avalon IQ3_KS rank ninth if it ties at the top on office work? Because of C03, worth 6 points of the total. Its Rust code doesn't compile. With that case passed it would score 96.5.
Did the test change from the earlier reports? It adds 8 format gates, which add no points. The 54 office checks, the 418 substantive checks and the weights are the same.
Which is better, Strata or Avalon? This data doesn't say, because IQ3_KS only loads in Avalon. Our experience with both is in this note.
Sources
- Strata, the engine used for the Swift and Unsloth configurations (version 0.2.10).
- Avalon, the engine used with IQ3_KS (commit 16de68d28).
- Qwen3.8 Flash Next Avalon GGUF, the repository for the IQ3_KS model.
- Unsloth, Qwen3.8 Flash Next GGUF, the repository for UD-IQ4_XS and the vision projector.
Contact
If your organization is considering AI, local or paid, ITLab can help you choose a model for your type of work and test it on your real cases. Contact us.
The ITLab Bench is a living instrument that we keep refining. The conclusions reflect ITLab's judgment based on its own measurements.
Do you need someone to take care of your technology?
Tell us what you need to organize, protect or solve. The first conversation has no cost: we assess whether our model fits you and what the next steps would be.
Let's talk about your business