NVIDIA graphics card
What AI models can the NVIDIA GeForce RTX 4090 24 GB run?
At 4-bit quantization with an 8K-token conversation, 60 of the 146 open-weight models in our directory fit entirely in the NVIDIA GeForce RTX 4090 24 GB's 24 GB of video memory (about 23 GB usable), up to Qwen3.5 35B-A3B. With 32 GB of dual-channel DDR5-5600, 12 more run partly from system RAM, more slowly. Among models of 7B parameters or more, Laguna XS 2.1 is the fastest, at an estimated 312 to 404 tokens per second.
| Model | Parameters | Memory needed | Estimated speed |
|---|---|---|---|
| Runs entirely in fast memory (60) | |||
| Qwen3.5 35B-A3B Alibaba (Qwen) | 36.0B 4.7B active | 23 GB 22 GB + 0.2 GB cache | 199–257 tokens/s Fast, little memory to spare |
| Qwen3.6 35B-A3B Alibaba (Qwen) | 36.0B 4.7B active | 23 GB 22 GB + 0.2 GB cache | 199–257 tokens/s Fast, little memory to spare |
| Nex-N2.5-Mini NEX AGI | 35.1B 3.9B active | 22 GB 21 GB + 0.2 GB cache | 229–296 tokens/s Fast, little memory to spare |
| Laguna XS 2.1 Poolside | 33.4B 2.2B active | 21 GB 20 GB + 0.4 GB cache | 312–404 tokens/s Fast |
| Qwen3 VL 32B Instruct Alibaba (Qwen) | 33.4B | 23 GB 20 GB + 2.1 GB cache | 35–45 tokens/s Fast, little memory to spare |
| Qwen2.5 Coder 32B Instruct Alibaba (Qwen) | 32.8B | 23 GB 20 GB + 2.1 GB cache | 35–46 tokens/s Fast, little memory to spare |
| Qwen3 32B Alibaba (Qwen) | 32.8B | 23 GB 20 GB + 2.1 GB cache | 35–46 tokens/s Fast, little memory to spare |
| Nemotron 3.5 Lightning 30B A3B NVIDIA | 31.6B | 20 GB 19 GB + 0.4 GB cache | Speed not estimated for this architecture |
| Nemotron 3 Nano 30B A3B NVIDIA | 31.6B | 20 GB 19 GB + 0.1 GB cache | Speed not estimated for this architecture |
| Gemma 4 31B IT Google DeepMind | 31.3B | 22 GB 19 GB + 2.2 GB cache | 37–47 tokens/s Fast, little memory to spare |
| GLM-4.7-Flash Zhipu AI (Z.ai) | 31.2B 5.2B active | 20 GB 19 GB + 0.4 GB cache | 180–233 tokens/s Fast |
| Qwen3 VL 30B A3B Thinking Alibaba (Qwen) | 31.1B 3.9B active | 20 GB 19 GB + 0.8 GB cache | 210–271 tokens/s Fast |
| Qwen3 VL 30B A3B Instruct Alibaba (Qwen) | 31.1B 3.9B active | 20 GB 19 GB + 0.8 GB cache | 210–271 tokens/s Fast |
| Qwen3 30B A3B Instruct 2507 Alibaba (Qwen) | 30.5B 3.4B active | 20 GB 19 GB + 0.8 GB cache | 230–298 tokens/s Fast |
| Qwen3-Coder 30B-A3B Instruct Alibaba (Qwen) | 30.5B 3.4B active | 20 GB 19 GB + 0.8 GB cache | 230–298 tokens/s Fast |
| Qwen3 30B A3B Alibaba (Qwen) | 30.5B 3.4B active | 20 GB 19 GB + 0.8 GB cache | 230–298 tokens/s Fast |
| Qwen3 30B A3B Thinking 2507 Alibaba (Qwen) | 30.5B 3.4B active | 20 GB 19 GB + 0.8 GB cache | 230–298 tokens/s Fast |
| Hy-MT2-30B-A3B Tencent (Hunyuan) | 30.1B 3.5B active | 20 GB 18 GB + 0.8 GB cache | 226–292 tokens/s Fast |
| Muse Glimmer 30B Meta | 29.8B | 19 GB 18 GB + 0.2 GB cache | 40–52 tokens/s Fast |
| Qwen3.8 27B Alibaba (Qwen) | 27.8B | 18 GB 17 GB + 0.5 GB cache | 43–55 tokens/s Fast |
| Qwen3.6 27B Alibaba (Qwen) | 27.8B | 18 GB 17 GB + 0.5 GB cache | 43–55 tokens/s Fast |
| Qwen3.5 27B Alibaba (Qwen) | 27.8B | 18 GB 17 GB + 0.5 GB cache | 43–55 tokens/s Fast |
| Gemma 3 27B IT Google DeepMind | 27.4B | 17 GB weights only | Speed not estimated (architecture not public) |
| Gemma 2 27B Google DeepMind | 27.2B | 17 GB weights only | Speed not estimated (architecture not public) |
| Gemma 4 26B A4B IT Google DeepMind | 25.8B | 17 GB 16 GB + 0.5 GB cache | Speed not estimated for this architecture |
| Voxtral Small 24B 2507 Mistral AI | 24.3B | 17 GB 15 GB + 1.3 GB cache | 47–61 tokens/s Fast |
| Uncensored Cognitivecomputations | 24.0B | 17 GB 15 GB + 1.3 GB cache | 48–62 tokens/s Fast |
| Mistral Small 3.1 24B Instruct Mistral AI | 24.0B | 17 GB 15 GB + 1.3 GB cache | 48–62 tokens/s Fast |
| LFM2.5-24B-A2B Liquid | 24.0B | 16 GB 15 GB + 0.2 GB cache | Speed not estimated for this architecture |
| Cydonia 24B V4.1 Thedrummer | 23.6B | 16 GB 14 GB + 1.3 GB cache | 48–63 tokens/s Fast |
| Mistral Small 3 Mistral AI | 23.6B | 16 GB 14 GB + 1.3 GB cache | 48–63 tokens/s Fast |
| Cydonia 24B V4.1 Thedrummer | 23.6B | 16 GB 14 GB + 1.3 GB cache | 48–63 tokens/s Fast |
| Safety GPT OSS 20B OpenAI | 21.5B 4.8B active | 14 GB 13 GB + 0.2 GB cache | 196–254 tokens/s Fast |
| GPT OSS 20B OpenAI | 20.9B 4.2B active | 14 GB 13 GB + 0.2 GB cache | 216–280 tokens/s Fast |
| Reka Flash 3 Rekaai | 20.9B | 15 GB 13 GB + 1.1 GB cache | 54–70 tokens/s Fast |
| Qwen3 14B Alibaba (Qwen) | 14.8B | 11 GB 9.0 GB + 1.3 GB cache | 73–94 tokens/s Fast |
| Phi 4 Microsoft | 14.7B | 11 GB 8.9 GB + 1.7 GB cache | 72–94 tokens/s Fast |
| Ministral 3 14B Mistral AI | 13.9B | 11 GB 8.5 GB + 1.3 GB cache | 77–99 tokens/s Fast |
| Gemma 3 12B IT Google DeepMind | 12.2B | 8.2 GB weights only | Speed not estimated (architecture not public) |
| Llama Guard 4 12B Meta | 12.0B | 8.1 GB weights only | Speed not estimated (architecture not public) |
| Qwen3.5 9B Alibaba (Qwen) | 9.7B | 6.9 GB 5.9 GB + 0.3 GB cache | 111–144 tokens/s Fast |
| Moondream 3.1 Moondream | 9.3B | 6.4 GB weights only | Speed not estimated (architecture not public) |
| Moondream 3 Preview Moondream | 9.3B | 6.4 GB weights only | Speed not estimated (architecture not public) |
| Ministral 3 8B Mistral AI | 8.9B | 7.3 GB 5.4 GB + 1.1 GB cache | 112–144 tokens/s Fast |
| Granite 4.2 8B IBM | 8.8B | 7.5 GB 5.3 GB + 1.3 GB cache | 111–144 tokens/s Fast |
| Qwen3 VL 8B Instruct Alibaba (Qwen) | 8.8B | 7.3 GB 5.3 GB + 1.2 GB cache | 113–146 tokens/s Fast |
| Qwen3 VL 8B Thinking Alibaba (Qwen) | 8.8B | 7.3 GB 5.3 GB + 1.2 GB cache | 113–146 tokens/s Fast |
| UI-TARS 7B Bytedance | 8.3B | 6.3 GB 5.0 GB + 0.5 GB cache | 125–161 tokens/s Fast |
| Qwen3 8B Alibaba (Qwen) | 8.2B | 7.0 GB 5.0 GB + 1.2 GB cache | 119–154 tokens/s Fast |
| Llama 3.1 8B Instruct fp8 Meta | 8.0B | 5.7 GB weights only | Speed not estimated (architecture not public) |
| Hy-MT2-7B Tencent (Hunyuan) | 8.0B | 6.7 GB 4.9 GB + 1.1 GB cache | 122–158 tokens/s Fast |
| Qwen2.5 7B Instruct Alibaba (Qwen) | 7.6B | 5.9 GB 4.6 GB + 0.5 GB cache | 133–173 tokens/s Fast |
| Gemma 3 4B IT Google DeepMind | 4.3B | 3.4 GB weights only | Speed not estimated (architecture not public) |
| Nemotron 3.5 Content Safety NVIDIA | 4.3B | 3.7 GB 2.6 GB + 0.3 GB cache | 210–272 tokens/s Fast |
| Ministral 3 3B Mistral AI | 3.8B | 4.0 GB 2.3 GB + 0.9 GB cache | 209–271 tokens/s Fast |
| Llama 3.2 3B Instruct Meta | 3.2B | 2.7 GB weights only | Speed not estimated (architecture not public) |
| Granite 4.0 Micro IBM | 3.2B | 2.8 GB 1.9 GB + 0.1 GB cache | 267–346 tokens/s Fast |
| Hy-MT2-1.8B Tencent (Hunyuan) | 2.0B | 2.6 GB 1.2 GB + 0.5 GB cache | 319–413 tokens/s Fast |
| Hy-MT2-1.8B Tencent (Hunyuan) | 2.0B | 2.6 GB 1.2 GB + 0.5 GB cache | 319–413 tokens/s Fast |
| Llama 3.2 1B Instruct Meta | 1.2B | 1.5 GB weights only | Speed not estimated (architecture not public) |
| Runs, partly from system RAM (slower) (12) | |||
| Qwen3-Next 80B-A3B Instruct Alibaba (Qwen) | 81.3B 5.5B active | 51 GB 49 GB + 0.8 GB cache | 41–57 tokens/s Fast, 28 GB in system RAM |
| Qwen3-Next 80B-A3B (Thinking) Alibaba (Qwen) | 81.3B 5.5B active | 51 GB 49 GB + 0.8 GB cache | 41–57 tokens/s Fast, 28 GB in system RAM |
| Hunyuan A13B Instruct Tencent (Hunyuan) | 80.4B | 51 GB 49 GB + 1.1 GB cache | 27 GB in system RAM; Speed not estimated for this architecture |
| Qwen3 Coder Next Alibaba (Qwen) | 79.7B 3.9B active | 50 GB 48 GB + 0.8 GB cache | 46–63 tokens/s Fast, 27 GB in system RAM |
| Qwen2.5-VL 72B Instruct Alibaba (Qwen) | 73.4B | 48 GB 45 GB + 2.7 GB cache | 1.2–1.7 tokens/s Too slow for chat, 25 GB in system RAM |
| Qwen2.5 72B Instruct Alibaba (Qwen) | 72.7B | 48 GB 44 GB + 2.7 GB cache | 1.2–1.7 tokens/s Too slow for chat, 24 GB in system RAM |
| Magnum v4 72B Anthracite ORG | 72.7B | 48 GB 44 GB + 2.7 GB cache | 1.2–1.7 tokens/s Too slow for chat, 24 GB in system RAM |
| Hermes 3 70B Instruct Nousresearch | 70.6B | 46 GB 43 GB + 2.7 GB cache | 1.3–1.8 tokens/s Too slow for chat, 23 GB in system RAM |
| Hermes 3 70B Instruct Nousresearch | 70.6B | 46 GB 43 GB + 2.7 GB cache | 1.3–1.8 tokens/s Too slow for chat, 23 GB in system RAM |
| Llama-3.1-70B-Instruct Meta | 70.6B | 44 GB weights only | 20 GB in system RAM; Speed not estimated (architecture not public) |
| Llama-3.3-70B-Instruct Meta | 70.6B | 44 GB weights only | 20 GB in system RAM; Speed not estimated (architecture not public) |
| Skyfall 36B V2 Thedrummer | 36.9B | 25 GB 22 GB + 2.1 GB cache | 10–14 tokens/s Comfortable to read, 1.9 GB in system RAM |
74 larger models do not fit. Change the quantization, conversation length or system RAM in the tool.
Compared with similar cards
- NVIDIA GeForce RTX 5080 16 GB: 16 GB, 28 models fit entirely (-32); about 4% slower on models that fit both (960 vs 1008 GB/s).
- AMD Radeon RX 7900 XTX 24 GB: same memory, so the same models fit; about 14% slower on models that fit both (960 vs 1008 GB/s, and AMD cards run llama.cpp less efficiently than NVIDIA's).
- NVIDIA GeForce RTX 3090 24 GB: same memory, so the same models fit; about 6% slower on models that fit both (936 vs 1008 GB/s).
- Intel Arc Pro B60 24 GB: same memory, so the same models fit; about 67% slower on models that fit both (456 vs 1008 GB/s, and Intel cards run llama.cpp less efficiently than NVIDIA's).
- NVIDIA GeForce RTX 5090 32 GB: 32 GB, 61 models fit entirely (+1); about 28% faster on models that fit both (1792 vs 1008 GB/s).
Frequently asked questions
What is the largest AI model the NVIDIA GeForce RTX 4090 24 GB can run?
At 4-bit quantization with an 8K-token conversation, the largest model in our directory that fits entirely in its 24 GB of video memory is Qwen3.5 35B-A3B (23 GB). With 32 GB of dual-channel DDR5-5600 for the overflow, larger models such as Qwen3-Next 80B-A3B Instruct also run, more slowly, partly from system RAM.
Can the NVIDIA GeForce RTX 4090 24 GB run GPT OSS 20B?
Yes. At 4-bit, GPT OSS 20B needs about 14 GB and fits entirely in the NVIDIA GeForce RTX 4090 24 GB's memory at an estimated 216 to 280 tokens per second.
Can the NVIDIA GeForce RTX 4090 24 GB run Qwen3 32B?
Yes. At 4-bit, Qwen3 32B needs about 23 GB and fits entirely in the NVIDIA GeForce RTX 4090 24 GB's memory at an estimated 35 to 46 tokens per second.
Can the NVIDIA GeForce RTX 4090 24 GB run OpenAI GPT 120B?
No. OpenAI GPT 120B needs about 72 GB at 4-bit, more than the NVIDIA GeForce RTX 4090 24 GB can hold even with 32 GB of dual-channel DDR5-5600 for overflow.
These are estimates, not measurements: memory from each model's parameter count and published architecture, speed from memory bandwidth, calibrated on the AIPCs test bench. Drivers, software and prompt length move real results. How the estimate works.