Can I run flint on a Mac with 96 GB?
Yes, with a GGUF version
The published weights are too big, but the Q8_0 file from unsloth/flint-GGUF (42.3 GB) fits, with room for ≈ 256K tokens of context.
ollama run hf.co/unsloth/flint-GGUF:Q8_0Files and context on Mac with 96 GB
Each file of flint (and of GGUF versions the site has analyzed), and how much context fits next to it, with the context cache in 16-bit or 8-bit.
| File | Size | On Mac with 96 GB | Context (16-bit) | Context (8-bit) |
|---|---|---|---|---|
| As published | 79.6 GB | Doesn't fit | — | — |
| BF16 | 79.6 GB | Doesn't fit | — | — |
| Q8_0 | 42.3 GB | Fits | 256K | 256K |
| UD-Q6_K_XL | 36.3 GB | Fits | 256K | 256K |
| UD-Q5_K_XL | 29.9 GB | Fits | 256K | 256K |
| UD-Q4_K_XL | 25.1 GB | Fits | 256K | 256K |
| UD-Q3_K_XL | 18.5 GB | Fits | 256K | 256K |
| UD-IQ3_XXS | 14.5 GB | Fits | 256K | 256K |
| UD-Q2_K_XL | 13.6 GB | Fits | 256K | 256K |
| UD-IQ2_XXS | 11.9 GB | Fits | 256K | 256K |
| UD-IQ1_M | 11.4 GB | Fits | 256K | 256K |
flint on other hardware
- NVIDIA H100Yes, with a GGUF version
- RTX 5090Yes, with a GGUF version
- RTX 4090Yes, with a GGUF version
- RTX 3090Yes, with a GGUF version
- RTX 5060 Ti 16 GBYes, with a GGUF version
- RTX 3060 12 GBNo
- RTX 4060No
- Mac with 32 GBYes, with a GGUF version
- Mac with 64 GBYes, with a GGUF version
- Mac with 96 GBYes, with a GGUF version
- Mac with 128 GBYes, just