Can I run clef on an RTX 4090?
Yes, with a GGUF version
The published weights are too big, but the Q5_K_M file from bartowski/Cloudflare_clef-GGUF (20.6 GB) fits, with room for ≈ 12K tokens of context.
ollama run hf.co/bartowski/Cloudflare_clef-GGUF:Q5_K_MFiles and context on RTX 4090
Each file of clef (and of GGUF versions the site has analyzed), and how much context fits next to it, with the context cache in 16-bit or 8-bit.
| File | Size | On RTX 4090 | Context (16-bit) | Context (8-bit) |
|---|---|---|---|---|
| As published | 54.7 GB | Doesn't fit | — | — |
| BF16 | 54.1 GB | Doesn't fit | — | — |
| Q8_0 | 28.8 GB | Doesn't fit | — | — |
| Q6_K_L | 24.0 GB | Doesn't fit | — | — |
| Q6_K | 23.3 GB | Doesn't fit | — | — |
| Q5_K_M | 20.6 GB | Tight | 12K | 24K |
| Q5_K_S | 19.6 GB | Tight | 28K | 56K |
| Q4_K_L | 18.6 GB | Tight | 42K | 85K |
| Q4_1 | 17.7 GB | Tight | 56K | 111K |
| Q4_K_M | 17.7 GB | Tight | 57K | 113K |
| Q4_K_S | 16.6 GB | Fits | 72K | 145K |
| IQ4_NL | 16.3 GB | Fits | 77K | 153K |
| Q4_0 | 16.2 GB | Fits | 78K | 157K |
| IQ4_XS | 15.6 GB | Fits | 88K | 175K |
| Q3_K_L | 15.2 GB | Fits | 94K | 188K |
| Q3_K_M | 14.5 GB | Fits | 104K | 208K |
| IQ3_M | 13.8 GB | Fits | 114K | 229K |
| Q3_K_S | 13.6 GB | Fits | 117K | 234K |
| Q2_K | 11.7 GB | Fits | 145K | 256K |
| BF16 | 54.1 GB | Doesn't fit | — | — |
| Q8_0 | 28.7 GB | Doesn't fit | — | — |
| Q4_K_M | 19.2 GB | Tight | 33K | 66K |