Can I run clef on an RTX 5060 Ti 16 GB?
Yes, with a GGUF version
The published weights are too big, but the IQ3_M file from bartowski/Cloudflare_clef-GGUF (13.8 GB) fits, with room for ≈ 7K tokens of context.
ollama run hf.co/bartowski/Cloudflare_clef-GGUF:IQ3_MFiles and context on RTX 5060 Ti 16 GB
Each file of clef (and of GGUF versions the site has analyzed), and how much context fits next to it, with the context cache in 16-bit or 8-bit.
| File | Size | On RTX 5060 Ti 16 GB | Context (16-bit) | Context (8-bit) |
|---|---|---|---|---|
| As published | 54.7 GB | Doesn't fit | — | — |
| BF16 | 54.1 GB | Doesn't fit | — | — |
| Q8_0 | 28.8 GB | Doesn't fit | — | — |
| Q6_K_L | 24.0 GB | Doesn't fit | — | — |
| Q6_K | 23.3 GB | Doesn't fit | — | — |
| Q5_K_M | 20.6 GB | Doesn't fit | — | — |
| Q5_K_S | 19.6 GB | Doesn't fit | — | — |
| Q4_K_L | 18.6 GB | Doesn't fit | — | — |
| Q4_1 | 17.7 GB | Doesn't fit | — | — |
| Q4_K_M | 17.7 GB | Doesn't fit | — | — |
| Q4_K_S | 16.6 GB | Doesn't fit | — | — |
| IQ4_NL | 16.3 GB | Doesn't fit | — | — |
| Q4_0 | 16.2 GB | Doesn't fit | — | — |
| IQ4_XS | 15.6 GB | Doesn't fit | — | — |
| Q3_K_L | 15.2 GB | Doesn't fit | — | — |
| Q3_K_M | 14.5 GB | Doesn't fit | — | — |
| IQ3_M | 13.8 GB | Tight | 7K | 14K |
| Q3_K_S | 13.6 GB | Tight | 10K | 19K |
| Q2_K | 11.7 GB | Tight | 38K | 76K |
| BF16 | 54.1 GB | Doesn't fit | — | — |
| Q8_0 | 28.7 GB | Doesn't fit | — | — |
| Q4_K_M | 19.2 GB | Doesn't fit | — | — |