Can I run flint on an RTX 5060 Ti 16 GB?
Yes, with a GGUF version
The published weights are too big, but the UD-Q2_K_XL file from unsloth/flint-GGUF (13.6 GB) fits, with room for ≈ 27K tokens of context. That file is under 3 bits per weight, so expect noticeably lower quality.
Full analysis of flint
Everything that runs on RTX 5060 Ti 16 GB
Get the UD-Q2_K_XL file on Hugging Face
ollama run hf.co/unsloth/flint-GGUF:UD-Q2_K_XLFiles and context on RTX 5060 Ti 16 GB
Each file of flint (and of GGUF versions the site has analyzed), and how much context fits next to it, with the context cache in 16-bit or 8-bit.
| File | Size | On RTX 5060 Ti 16 GB | Context (16-bit) | Context (8-bit) |
|---|---|---|---|---|
| As published | 79.6 GB | Doesn't fit | — | — |
| BF16 | 79.6 GB | Doesn't fit | — | — |
| Q8_0 | 42.3 GB | Doesn't fit | — | — |
| UD-Q6_K_XL | 36.3 GB | Doesn't fit | — | — |
| UD-Q5_K_XL | 29.9 GB | Doesn't fit | — | — |
| UD-Q4_K_XL | 25.1 GB | Doesn't fit | — | — |
| UD-Q3_K_XL | 18.5 GB | Doesn't fit | — | — |
| UD-IQ3_XXS | 14.5 GB | Doesn't fit | — | — |
| UD-Q2_K_XL | 13.6 GB | Tight | 27K | 56K |
| UD-IQ2_XXS | 11.9 GB | Tight | 85K | 172K |
| UD-IQ1_M | 11.4 GB | Fits | 103K | 206K |