Does a model improve when it works inside an agent?

We had three models work through ITLab Bench 02.5 inside Pi, an agent that gives them tools to read files, run commands and hand in their results. The short answer: an agent doesn't improve a model on its own. When the integration is done right, the result stays the same or gets slightly better; when it isn't, it gets much worse.

• GPT-6.1 Sol: a slight gain. It goes from 95.0 to 95.6 points, using 13% more tokens. Almost all of the gain comes from the Vision series, and the setup changed in more than one way, so it can't all be credited to the agent.
• Swift Flash Next IQ3_S: it comes down to the integration. Its first run with Pi dropped to 76.8 because of attachment problems. With the integration fixed and Ponytail enabled, the combined result is 94.3, against 94.6 without the agent: practically the same, with 23% fewer tokens.
• GPT-OSS 20B: not comparable. It rises from 68.5 to 70.1, but it doesn't accept images, and its drop on the agentic series shows the integration still doesn't suit it.

What we tested

Pi is a coding agent: instead of receiving the task and answering in one go, the model works in turns, opens files, runs commands and hands in its result through tools. Ponytail is a set of instructions (skills) for Pi that pushes the model toward the simplest solution that works. We used Pi 1.4.0 and Ponytail 5.3.2; the local models ran on the same 16 GiB consumer graphics card as in the 10-model report, with the stable llama.cpp build we use for the bench.

We compared three models, each with and without Pi, on the same 49 cases of ITLab Bench, version 02.5, under the same rules: in series N every check counts the same; series C, M, A, Vision and L use the weights set by the test's authors; and the total weighs N 35%, C 20%, M 20%, A 10%, Vision 10% and L 5%.

• GPT-6.1 Sol (paid): direct through the native API versus Pi.
• Swift Flash Next IQ3_S (local): no agent versus two Pi setups, an initial one and another with the integration fixed and Ponytail enabled.
• GPT-OSS 20B (local): no agent versus Pi.

Results

The 94.3 for Swift with Pi and Ponytail is a combined result with selective substitutions, not a full run: after fixing the bench and the adapter, five authorized cases were rerun and their results replaced those of the original run, which had scored 78.6 on the test authors' scale.

With a single run per case, gaps under one point don't let us declare one setup better than another.

Overall ranking with every configuration

The Pi configurations are integrated with their actual rank and marked "+ Pi" in the name; GPT-OSS 20B, with and without Pi, has no comparable total because it doesn't accept images. The per-series rankings for every model are in GPT-6.1 Terra joins ITLab Bench 02.5.

By series

Series N is scored by checks passed; the others use the test authors' weights. Critical failures: cases in which a check labeled critical failed; for information only, it doesn't change the score.

Sol. With and without Pi it passes the same 48 office checks, though with Pi it has two fewer cases with a critical failure. The difference in the total comes from Vision (from 92.5 to 98.4). The Pi run and the direct run don't differ only in the agent: the way the model is called and other session settings change too. That's why we don't credit the whole gain to Pi.

Swift Flash Next. The first Pi run lost the calculator in the software project (M drops to 25.0) and four office checks. Digging in, we found that attachments arrived with different names and in a different order than expected, that image transport was failing, and that this setup had skills disabled. With that fixed and Ponytail enabled, Swift gets back all 47 office checks and the full project. It ends up practically level with its no-agent result.

GPT-OSS 20B. With Pi it gains on office work, the software project and reading, but drops on the agentic series from 90.7 to 37.3. Actions in that series are simulated, and inside the agent the model tended to look for real files instead of using the exercise's tools; we fixed that integration and reran A01, and the result still came out low. Since it doesn't accept images, its total is calculated without Vision and isn't comparable with the other two.

For information only: N13, N14 and M08

These don't count toward the total. The new N14 delivery rule (notes outside the requested message fail the gate) affected some Pi answers.

Another possible order: every check counts the same

ITLab Bench has its own weighting: weights per series and, except in N, per-exercise and per-check weights set by the test's authors. But every individual check is recorded, and with them you can look at the full results from another angle: count every check passed, across all series, as if each were worth the same.

This table takes in the 418 substantive checks in the six series (N, C, M, A, Vision and L, with N02 counted under Vision). It leaves out format gates, which aren't substantive checks, and the informational exercises N13, N14 and M08. Checks not run or indeterminate count as not passed. Swift with Pi and Ponytail records 420 because in its M10 rerun the grading was broken down into two more checks; GPT-OSS 20B records 375 because it has no Vision, so its percentage isn't comparable and it gets no rank.

The top five are the same five that lead the weighted ranking (95.6, 95.0, 94.6, 94.3 and 92.6), with Sol, with and without Pi, tied at the top. The differences show up further down, and they come from how the checks are distributed: reading (L01) contributes 195 of the 418 and the calculator (M10) 72, while the office series contributes only 54. So on this basis Swift 27B IQ3_S goes from eighth in the weighted ranking to twelfth in this table (it's the best at office work but fails the calculator), while Terra, Gemma 26B and GPT-6 Luna climb because they solve the calculator. Neither order is "the true one": ITLab's weighting reflects which work matters most; the simple count shows the volume of correct results. We publish both so every reader can judge with the full data.

For the agent question, the simple count is clear: Sol passes exactly the same 410 checks with and without Pi, and Swift with Pi and Ponytail ends up practically level with its no-agent result. GPT-OSS 20B rises from 273 to 334 checks with Pi, almost all of it from the calculator, though it loses the agentic series.

Cost in tokens and time

Times and tokens come from each configuration's original campaign; reruns aren't added in. Swift's initial Pi run isn't included.

• Sol with Pi uses 12.7% more tokens and takes 6.6% longer.
• Swift with Pi and Ponytail generates 22.7% fewer tokens and takes 6.5% less time, though its time includes pauses to tune the graphics card's memory. The five selective reruns added 21,018 tokens and 5 min 47 s; counting the original run plus those reruns, the total spend was 169,588 tokens and 56 min 15 s.
• GPT-OSS with Pi uses 5.1% more tokens and takes 10.2% longer.

Reading and generation speed (local models)

Generation speed barely moves: the agent neither speeds up nor slows down the model. What changes is how much it writes. For the Swift setup we tried several graphics-memory reservations and settled on 544 MiB.

When tools and skills help

What separated a Pi run that dropped 17.8 points from one that held level was the integration, not the model:

1. Files have to arrive intact. Attachment names and order must match what the task asks for; otherwise the model works on the wrong file.
2. Images have to reach the model. Broken image transport sinks vision even if the model can read images.
3. Skills have to be enabled. A setup with skills disabled never gets the instructions that are supposed to improve it.
4. The agent's tools mustn't compete with the exercise's. If the task calls for simulated actions and the agent offers real file access, the model may go looking in the wrong place.
5. Memory has to suffice. On a 16 GiB card, the memory reserved for the agent and the engine affects whether the battery completes.

ITLab's take

• Don't adopt an agent expecting it to improve the model. In our tests, the best result with an agent was practically the same as without one.
• Adopt it if the work calls for it, for example to operate on real files and repositories, and measure the result before using it in production.
• Check the integration before blaming the model: attachments, images, skills and tools.
• Factor in the cost: with Pi, Sol used more tokens; Swift with Ponytail used fewer.

Limitations

• One run per configuration and case. Gaps under one point are noise.
• Confounded variables. For Sol, the way the model is called changes too; for Swift, the bench and the adapter were fixed together. No comparison isolates the agent's effect alone.
• Combined result. Swift's 94.3 with Pi and Ponytail combines the original run with five authorized selective reruns; it isn't another full battery.
• GPT-OSS 20B isn't comparable. It doesn't accept images and its total is calculated over five series.
• N14 is informational. A new delivery rule in N14 affected some Pi answers, but N14 doesn't count toward the total.

Frequently asked questions

Does an agent like Pi improve an AI model? Not on its own. Sol gained 0.6 points with other setup changes in play; Swift Flash Next held level with the integration fixed, and did much worse without the fix.

What is Ponytail? A set of skills for Pi that steers the model toward the simplest solution that works. On Swift Flash Next it coincided with 23% fewer generated tokens.

Why isn't Swift's 94.3 with Pi a full run? Because, after fixing the bench and the adapter, five authorized cases were rerun and their results replaced the originals. It's a combined result.

Why isn't GPT-OSS 20B compared like the others? Because it doesn't accept images. Its total is calculated without the Vision series and isn't comparable with models that completed it.

Where are the full rankings? In GPT-6.1 Terra joins ITLab Bench 02.5, with every model in its base configuration.

Contact

If your organization is considering AI agents with local or paid models, ITLab can help you integrate them and measure whether they actually improve your work. Contact us.

The ITLab Bench is a living instrument that we keep refining. The conclusions reflect ITLab's judgment based on its own measurements.