Can I run clef on an RTX 6000 Ada?
Yes, with a GGUF version
The published weights are too big, but the Q8_0 file from bartowski/Cloudflare_clef-GGUF (28.8 GB) fits, with room for ≈ 212K tokens of context.
ollama run hf.co/bartowski/Cloudflare_clef-GGUF:Q8_0Files and context on RTX 6000 Ada
Each file of clef (and of GGUF versions the site has analyzed), and how much context fits next to it, with the context cache in 16-bit or 8-bit.
| File | Size | On RTX 6000 Ada | Context (16-bit) | Context (8-bit) |
|---|---|---|---|---|
| As published | 54.7 GB | Doesn't fit | — | — |
| BF16 | 54.1 GB | Doesn't fit | — | — |
| Q8_0 | 28.8 GB | Fits | 212K | 256K |
| Q6_K_L | 24.0 GB | Fits | 256K | 256K |
| Q6_K | 23.3 GB | Fits | 256K | 256K |
| Q5_K_M | 20.6 GB | Fits | 256K | 256K |
| Q5_K_S | 19.6 GB | Fits | 256K | 256K |
| Q4_K_L | 18.6 GB | Fits | 256K | 256K |
| Q4_1 | 17.7 GB | Fits | 256K | 256K |
| Q4_K_M | 17.7 GB | Fits | 256K | 256K |
| Q4_K_S | 16.6 GB | Fits | 256K | 256K |
| IQ4_NL | 16.3 GB | Fits | 256K | 256K |
| Q4_0 | 16.2 GB | Fits | 256K | 256K |
| IQ4_XS | 15.6 GB | Fits | 256K | 256K |
| Q3_K_L | 15.2 GB | Fits | 256K | 256K |
| Q3_K_M | 14.5 GB | Fits | 256K | 256K |
| IQ3_M | 13.8 GB | Fits | 256K | 256K |
| Q3_K_S | 13.6 GB | Fits | 256K | 256K |
| Q2_K | 11.7 GB | Fits | 256K | 256K |
| BF16 | 54.1 GB | Doesn't fit | — | — |
| Q8_0 | 28.7 GB | Fits | 213K | 256K |
| Q4_K_M | 19.2 GB | Fits | 256K | 256K |
clef on other hardware
- NVIDIA H100Yes
- RTX 5090Yes, with a GGUF version
- RTX 4090Yes, with a GGUF version
- RTX 3090Yes, with a GGUF version
- RTX 5060 Ti 16 GBYes, with a GGUF version
- RTX 3060 12 GBNo
- RTX 4060No
- RTX 6000 AdaYes, with a GGUF version
- Mac with 32 GBYes, with a GGUF version
- Mac with 64 GBYes, with a GGUF version
- Mac with 128 GBYes