What hardware an AI model needs on your premises

The second figure follows from this: memory bandwidth. It states how fast the card can read its own memory and thereby sets the speed of output. Capacity decides whether it runs. Bandwidth decides how fast.

How much space a parameter occupies depends on how many bits are used to store it. Reducing that precision is called quantisation. The rule behind it is primary-school arithmetic:

Parameters in billions × bytes per parameter = memory requirement in gigabytes

The loss in quality is not proportional to the saving. For business applications, 8-bit is regarded as practically equivalent to 16-bit and is the usual operating point. 4-bit costs measurable accuracy and is used where memory would otherwise not suffice.

Alongside the weights, the system keeps a working store for every live conversation holding the exchange so far. It grows with every token, that is, with every fragment of text the model reads or writes, and it exists separately per conversation. There is a formula for this too:

Memory per token = 2 × layers × key-value heads × head width × bytes per value

For an openly documented model in the 9-billion class (36 layers, 8 key-value heads, head width 128, 16-bit) this yields 144 kibibytes per token; for a model in the 27 to 32-billion class (64 layers), 256 kibibytes. At 8,000 tokens per conversation, meaning a medium-length dialogue with attached documents, that is 1.2 and 2.1 gigabytes respectively. Per user.

Calculated with 8-bit weights and a 20 per cent allowance for intermediate results and the runtime environment. Allowing longer contexts, say 32,000 tokens instead of 8,000, quadruples the history column.

A device on which the model starts and serves one user is not the same as a device for a department. The difference shows up in speed as well as memory. Adults read non-fiction at around 238 words per minute, that is, some 5 to 6 tokens per second; beyond 20 tokens per second virtually nobody keeps up. Below reading speed, output feels sluggish however good the total looks. Our own measurement on the DGX Spark (July 2026): the 9-billion model delivers 14 tokens per second for a single user at 16-bit precision; with 16 concurrent requests, aggregate throughput rises to 208 tokens per second, leaving around 13 per user. The delivered 4-bit version reads only a quarter of the bytes per parameter, so arithmetically the speed is correspondingly higher. The 27-billion model reads three times as much and is correspondingly slower: enough for a single team; for whole departments we recommend the larger device class.

The reason is batching: modern serving software combines several requests and so keeps the card better occupied. That raises aggregate throughput considerably, but distributes it across everyone involved. For procurement this means: settle how many people are to work simultaneously, and how fast the answer has to appear, before choosing a device.

Compact desktop systems with shared memory. The NVIDIA DGX Spark provides 128 GB of memory shared between processor and graphics unit. Its list price was raised in February 2026 from 3,999 to 4,699 US dollars; German trade quotes sit around 5,000 euros net depending on supplier. Capacity is not the issue in this class; speed is bounded by the 273 GB/s of bandwidth, which is sufficient for our model sizes and small teams.

A single professional graphics card in a workstation. The RTX PRO 6000 Blackwell carries 96 GB of memory at 600 W. Its list price rose from 8,565 US dollars (March 2025) to 13,250 US dollars (June 2026), driven by the shortage of GDDR7 memory. The smaller RTX PRO 5000 with 48 GB lists at 6,999 US dollars with street prices of 5,200 to 6,500 US dollars. This class has considerably more bandwidth than a compact desktop system and accordingly serves more people promptly.

Data-centre technology in a tower. The NVIDIA DGX Station with GB300 superchip offers 748 GB of coherent memory at up to 1,600 W. NVIDIA quotes no list price; a partner system was listed at just under 97,000 US dollars in early 2026. This class addresses several teams, or several models on one device.

An ordinary workplace graphics card is ruled out for the large tier. The GeForce RTX 5090, listed in German retail between roughly 3,050 and 4,500 euros depending on model, has 32 GB. A 27-billion model at 8-bit occupies 27 GB in weights alone. That leaves 5 GB for histories on paper; in practice it means two to three concurrent conversations and no reserve whatsoever. For the middle tier the same card is usable as long as user numbers stay small.

Also insufficient: running models on main memory and processor, coupling several small cards of 16 GB each, or buying capacity without regard to bandwidth. And 96 GB on one card is not the same as two times 48 GB, because splitting a model across two cards introduces communication and therefore latency.

Peak draw is on the data sheet: 240 W for the compact desktop system, 600 W for the large professional card alone, up to 1,600 W for the tower variant. At idle the compact system draws 22 to 25 W following a software update.

Commercial customers paid around 27 ct/kWh in February 2026 at a consumption of 10,000 kWh per year; larger industrial customers around 16 ct/kWh. At round-the-clock operation and 27 ct/kWh this gives:

The mixed load of 100 W and the system figure of 900 W are estimates; the peak values come from manufacturer data. Electricity is therefore secondary to the purchase price, though no longer a footnote for the tower variant, and cooling plus circuit protection belong in the planning.

Four figures suffice for the shortlist: memory capacity in gigabytes, memory bandwidth in gigabytes per second, power draw in watts, and whether the memory sits on one unit or is split. Set the capacity against your row in the second table. Anything below it is not an option, whatever the price.

We supply the fine-tuned model, not the metal. Tier M is based on Qwen3.5-9B with 9 billion parameters, tier L on Qwen3.8-27B with 27 billion. You procure the hardware yourself, and we deliberately do not tie you to a particular device: availability and prices currently fluctuate sharply, as the 55 per cent rise in the professional card within 16 months shows. Committing to today's model would be a mistake within a year.

What we ship is a setup program. It inspects your hardware, selects the appropriate path for your model and puts it into service together with the AnythingLLM interface. As orientation we name the DGX Spark for most cases and the DGX Station where entire departments are to work on it. Which device you buy is your decision.