Local AI for a Chilean company: hardware and limits (2/2)

Part 2 of 2. Read Part 1.

What hardware it takes

Measurements were taken on a machine that IT Lab specified and built specifically for testing: a Ryzen 7 5700X processor, 31 GiB of RAM and an AMD Radeon RX 9070 XT graphics card with 16 GiB, running Linux. The card was chosen as the most affordable option today with a good cost-benefit ratio that a small company can buy. The models evaluated have versions optimized for NVIDIA cards, so the general pattern is expected to be comparable on that hardware; the published figures correspond to this machine.

It is a high-end desktop computer, not a server. Of the card's nominal 15.922 GiB, the bench's memory budget is 14.242 GiB usable.

With that budget it allows:

• Quantized models from 12B to 35B parameters, with a working context of 64K to 131K.
• Useful speed with short context of 40 to 172 tok/s, depending on the model.

And today it does not allow:

• Serving several users at once. Everything was measured serially, with one model loaded; concurrency was not measured and the bench offers no service-level agreements.
• Huge context at good speed. The same model falls from 36-43 tok/s with short context to 7.5-14 tok/s with 80K-110K of context filled, and filling 110K takes about 34 minutes. A model accepting 128K does not mean it works at good speed with 128K.

Configuration of the inference engine changes the result as much as the choice of model: see Configuring a local model matters as much as choosing it. Full sizing is in SMB hardware for local AI with 16 GB.

What IT Lab does not promise

• Cost figures. The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.
• That any local model reads a document and does the arithmetic. It reads well; it counts poorly.
• That a computer replaces a developer. No local model delivered code from scratch that met the criterion.
• That a 128K or 200K context runs as fast as a short one. The theoretical ceiling is not performance, and configuring 64K does not show that the model processes or understands that whole window.
• Market prices.
• A purchase recommendation. These are laboratory measurements.

Frequently asked questions

How much does local AI cost? The bench does not measure costs. Running local AI carries real costs (energy, upfront hardware investment, maintenance and time) and, in IT Lab's judgment, current prices of paid services may not reflect their real cost.

What hardware does a small company need? In the measurements a 16 GiB graphics card with one model at a time was enough; concurrency across several users is not measured.

What can be automated today? Extracting and locating data in documents; counting or aggregating over what was read requires human review.

Is it suitable for sensitive data? A local model keeps the document inside the company's own infrastructure; the scope of each legal obligation is for its legal advisor to determine.

Limitations and scope

• A single pass per model and case unless another number is stated; small differences are noise. The official pass is kept separate from probes and diagnostics.
• Concurrency and human time are not measured.
• An IT Lab measurement on a single machine; not a purchase recommendation or a certification.

Measurement sheet

• Equipment: as described in the previous section.
• Quantization: 3 or 4 bits, depending on the model.
• Card memory: 14.242 GiB usable of 15.922 GiB.
• IT Lab Bench, edition 1.

Contact: IT Lab assesses whether a Chilean organization is ready for local AI on its own documents and data. Let's talk

---

The IT Lab Bench is a living instrument under continuous refinement. Some measurements come from earlier versions of the instrument and from previous generations of models, and some figures may contain inaccuracies that are corrected in each new edition. The conclusions express IT Lab's judgment based on its own measurements.