Guides · September 20, 2026

The largest AI models that fit on your GPU

Ask how large a model your graphics card can run and the honest answer has two halves: every consumer card stops at roughly the same size, and what changes is how much of the model survives.

All four cards top out around 35 billion parameters, because that is where today's popular mixture-of-experts models sit. What separates a 32 GB card from a 16 GB one is the number of bits each weight gets:

What bits per weight means

A model is a pile of numbers, published at 16 bits each. Quantizing rounds them into smaller slots: 4-bit quantization stores each weight in about half a byte, plus small scale factors that let the runtime reconstruct each block — which is why the real figure lands on 4.8 rather than exactly 4.

Down to about 4 bits the loss is mild and most people never notice it. At 3 bits it shows: more repetition, weaker reasoning, worse code. Below that, models degrade badly. So a 16 GB card can load a 35B model — it is running a more damaged copy than a 32 GB card is, and often a smaller model at 5 bits is the better trade.

The file names encode the scheme: Q4 is the nominal width, _K spends more bits on the parts that matter most, _S/_M/_L/_XL are variants of that trade, IQ squeezes lower with a smarter codebook, and UD- is Unsloth's dynamic mix.

The biggest models each card can hold

Largest download first, as in the chart, with at least 4K tokens of context left over. Note that the biggest file is not always the biggest model: a 35B model at 3 bits is a smaller download than a 14B one at 8.

RTX 5090 — 32 GB

RTX 4090 — 24 GB

RTX 3090 — 24 GB

RTX 5080 — 16 GB

Why mixture-of-experts models get so far

A 35B mixture-of-experts model holds all 35B parameters in memory but only runs a few billion of them for each token, so it behaves like a much smaller model while thinking, and like a large one in your VRAM. That is why the lists above are full of 35B-A3B names: 35 billion parameters, about 3 billion active per token.

The consequence is that memory, not speed, is what stops you. A card that can hold the file will usually run it at a reasonable pace.

Related

• The most downloaded AI models that fit on your GPU
• The largest open-weight AI models, by parameter count
• Best AI models for a 24 GB GPU (RTX 4090, RTX 3090)
• Which AI models can I run on my GPU? — every card, including Macs and data-centre GPUs