GPT-6.1 Terra joins ITLab Bench 02.5

We ran GPT-6.1 Terra through the same 49-case battery as our 10-model report. It finishes ninth of the 14 configurations measured, with 84.5 points: it fully completes the Rust and Rails project and nearly all of the agentic work, but misses a good part of the programming. GPT-6.1 Sol entered in the same round and, inside the Pi agent, tops the overall ranking.

• Terra, ninth. It scores 84.5, between Swift 27B IQ3_S (85.9) and Qwen 3.8 Flash Next SC117 IQ3_XXS (82.7).
• Terra's best and worst. Top marks on the software project and reading, 99.3 on agentic work; 60.0 on programming, where it passed 24 of 34 checks.
• Fast and frugal. It got through all 49 cases in 6 min 29 s and generated 24,942 tokens; only GPT-6 Luna was faster.
• GPT-6.1 Sol at the top. It scores 95.6 with the Pi agent and 95.0 without; Swift Flash Next IQ3_S, the best local model, is third with 94.6. With a single run per case, that's a statistical tie.

What we measured

GPT-6.1 Terra and GPT-6.1 Sol are paid models and were called through the provider's native API with default settings, just like Claude Haiku 5.5 and GPT-6 Luna in the previous report. They ran the 49 cases of ITLab Bench, version 02.5, with one run per case and the same scoring rules:

• Series N (analysis and office work): every check counts the same; the score is the percentage of the 54 checks passed.
• Series C, M, A, Vision and L: scored with the per-exercise and per-check weights set by the test's authors; a critical failure sets the case to 0, and A01, L01 and V01 give partial credit.
• Total: N 35%, C 20%, M 20%, A 10%, Vision 10% and L 5%.

The full method is explained in the 10-model report.

What goes into these rankings. Each table includes the 11 models in their base configuration (local models on the 16 GiB graphics card, paid models through their native API) and the four configurations in which a model worked inside the Pi agent, all with their actual rank and marked "+ Pi" in the name. Caveats: Swift Flash Next + Pi + Ponytail is a combined result with selective substitutions, not a full run; Swift's initial Pi run had attachment problems, later fixed, and isn't in the token and time tables; and GPT-OSS 20B, with or without Pi, doesn't accept images and has no overall rank. The agent analysis is in Does a model improve when it works inside an agent?.

Overall ranking

The ten models from the previous report keep their series scores; adding Sol and Terra only changes the ranks. The top five are within 3.0 points, and the top four within 1.3: with one run per case, there's no clear winner among them.

Terra lands ninth because of programming: a 60.0 in C costs it eight points of the total against a model with full marks. On the software project, which sinks several local models, Terra scores 100.

Another possible order: every check counts the same

ITLab Bench has its own weighting: weights per series and, except in N, per-exercise and per-check weights set by the test's authors. But every individual check is recorded, and with them you can look at the full results from another angle: count every check passed, across all series, as if each were worth the same.

This table takes in the 418 substantive checks in the six series (N, C, M, A, Vision and L, with N02 counted under Vision). It leaves out format gates, which aren't substantive checks, and the informational exercises N13, N14 and M08. Checks not run or indeterminate count as not passed. Swift with Pi and Ponytail records 420 because in its M10 rerun the grading was broken down into two more checks; GPT-OSS 20B records 375 because it has no Vision, so its percentage isn't comparable and it gets no rank.

The top five are the same five that lead the weighted ranking (95.6, 95.0, 94.6, 94.3 and 92.6), with Sol, with and without Pi, tied at the top. The differences show up further down, and they come from how the checks are distributed: reading (L01) contributes 195 of the 418 and the calculator (M10) 72, while the office series contributes only 54. So on this basis Swift 27B IQ3_S goes from eighth in the weighted ranking to twelfth in this table (it's the best at office work but fails the calculator), while Terra, Gemma 26B and GPT-6 Luna climb because they solve the calculator. Neither order is "the true one": ITLab's weighting reflects which work matters most; the simple count shows the volume of correct results. We publish both so every reader can judge with the full data.

Results by series

Each table runs from highest to lowest score; ties share a rank. "Checks passed" counts the substantive checks passed in the series, excluding format gates.

N: analysis and office work (35% of the total)

Tests whether the model handles office tasks on real material and answers only with what the material says: spreadsheets, emails, quotes, a small Rails app, slide decks, backups, security findings, reviews, legal questions, explanations and meeting minutes. In N, every check counts the same; an exercise with more checks weighs more simply because it has more of them.

Critical failures: cases in which a check labeled critical by the test's authors failed; for information only, it doesn't change the N score.

Terra sits in the middle group, five checks behind the leader. Sol ties for second, though with three cases containing a critical failure.

C: programming (20% of the total)

Tests whether Rust code passes automated tests the model never sees: distributing the width of a table, cleaning user text without letting dangerous markup through, and porting a Ruby function with exact parity.

This is Terra's weak spot: it shares 13th place with GPT-OSS 20B, with and without Pi, the smallest local model in the round.

M: software project (20% of the total)

Tests whether the model can carry a full project in Rust and Rails whose parts have to fit together and pass tests: an installment engine (M09) and a scientific calculator built from scratch (M10).

All four paid models complete the project. Terra, which stumbles on standalone programming tasks, does put together a multi-part project correctly.

A: agentic (10% of the total)

Tests whether the model uses simulated tools in the right order across three episodes: diagnosing an incident, computing a total from several sources and deploying in line with a policy. It's a single exercise (A01) with partial credit.

Terra ties for the highest score reached. Because the exercise gives partial credit and weighs its components differently, checks passed don't follow the score order exactly.

Vision (10% of the total)

Covers reading images (V01, 31 questions about 11 images) and photographed receipts (N02, three cases).

Critical failures: cases in this series in which a critical check failed.

L: reading (5% of the total)

Tests whether the model accurately extracts the data from a five-table org chart and answers questions about it.

For information only: N13, N14 and M08

These three exercises are run and scored but don't count toward the total, because they only check part of the task: N13, translation into Chilean Spanish; N14, a formal rewrite of a customer message; M08, the security rules of an admin interface. N14 now has a new rule: if the answer adds notes outside the requested message, it fails the delivery gate and its content goes ungraded. That lowered Claude Haiku 5.5's N14 score compared with the previous report.

They're ordered by the average of the three; ties go to the faster model. Time is the sum of inference time over the seven cases; for two Pi configurations there is no comparable time for these cases, shown as "—".

Speed, time and tokens

The figures come from the runs behind each model's score. Time is inference time, excluding loading and grading. GPT-OSS 20B totals 45 cases because it didn't run the four vision cases. For the Pi configurations, the figures come from their original campaign, without adding reruns; the time for Swift with Pi and Ponytail includes pauses to tune the graphics card's memory, and Swift's initial Pi run isn't in these tables.

Tokens generated

A token is the unit of text a model works with, on average a bit less than a word; output tokens normally include the reasoning done before answering. Since this is practical rather than creative work, fewer tokens for a correct answer is better: less time, less cost and less padding. The table runs from the most efficient model (the one that generated the least) to the one that generated the most.

Tokens follow each engine's or provider's own accounting. For paid services it may not include all internal reasoning, so their figures aren't equivalent to the local models' and their place in this table should be read with caution.

Response times

The table runs from fastest to slowest by total time.

Paid-service times include the network and the provider's queue; they aren't comparable with the local models, which ran on a single consumer graphics card.

Reading and generation speed (local models)

Sol and Terra are paid services and don't provide these counters. It includes the two local Pi configurations. The table runs from the model that reads fastest to the one that reads slowest.

What it means for choosing a model

• Terra suits agentic tasks, multi-part projects and reading, where it performs at the level of the best, with low token use and fast answers.
• It isn't the best pick for standalone Rust programming: eight models, six of them local, pass every check in that series.
• Sol and the best local model are tied at the top. If you can send data to an external service, Sol is the most even option; if your data must stay inside the organization, Swift Flash Next IQ3_S delivers an equivalent result on a consumer graphics card.
• Keep a human in the loop for figures, regulations and security, and test with your own cases before deciding.

Limitations

• One run per model and case. Gaps of one or two points can be noise.
• Weights. The series weights are ITLab's call, and the weights in C, M, A, Vision and L were set by the test's authors with knowledge of earlier results.
• Paid versus local. Paid-service times and tokens aren't comparable with local ones, and we can't certify which exact model sits behind a service's name.
• N14 has a new delivery rule. It's informational and doesn't change the total.

Frequently asked questions

Where does GPT-6.1 Terra rank? Ninth of 14 configurations, with 84.5 points. Its strengths are agentic work, the software project and reading; its weak spot is Rust programming.

Why does Terra rank below several local models? Because of the programming series, worth 20% of the total: it passed 24 of 34 checks, while six local models passed them all.

Is GPT-6.1 Sol better than the best local model? By 0.4 points, a gap a single run per case can't support. In practice they're tied.

How should the Pi agent configurations be read? Like any other row, with their rank, but with their caveats: Swift + Pi + Ponytail is a combined result and Swift's initial Pi run had attachment problems. We cover them in Does a model improve when it works inside an agent?.

Why doesn't GPT-OSS 20B have an overall rank? Because it doesn't accept images and didn't complete the Vision series; without it, its total isn't comparable with the others'.

Contact

If your organization is considering AI, local or paid, ITLab can help you choose a model for your type of work and test it on your real cases. Contact us.

The ITLab Bench is a living instrument that we keep refining. The conclusions reflect ITLab's judgment based on its own measurements.