Can I run flint on an RTX 3090?
Yes, with a GGUF version
The published weights are too big, but the UD-Q3_K_XL file from unsloth/flint-GGUF (18.5 GB) fits, with room for ≈ 105K tokens of context.
ollama run hf.co/unsloth/flint-GGUF:UD-Q3_K_XLFiles and context on RTX 3090
Each file of flint (and of GGUF versions the site has analyzed), and how much context fits next to it, with the context cache in 16-bit or 8-bit.
| File | Size | On RTX 3090 | Context (16-bit) | Context (8-bit) |
|---|---|---|---|---|
| As published | 79.6 GB | Doesn't fit | — | — |
| BF16 | 79.6 GB | Doesn't fit | — | — |
| Q8_0 | 42.3 GB | Doesn't fit | — | — |
| UD-Q6_K_XL | 36.3 GB | Doesn't fit | — | — |
| UD-Q5_K_XL | 29.9 GB | Doesn't fit | — | — |
| UD-Q4_K_XL | 25.1 GB | Doesn't fit | — | — |
| UD-Q3_K_XL | 18.5 GB | Tight | 105K | 210K |
| UD-IQ3_XXS | 14.5 GB | Fits | 239K | 256K |
| UD-Q2_K_XL | 13.6 GB | Fits | 256K | 256K |
| UD-IQ2_XXS | 11.9 GB | Fits | 256K | 256K |
| UD-IQ1_M | 11.4 GB | Fits | 256K | 256K |