06-22-2026, 10:46 AM
Update from running local models for a few months now:
The VRAM situation has genuinely improved. A few months ago 24GB was the sweet spot for decent 30B models. Now with better quantisation (IQ4_XS, Q4_K_M improvements in llama.cpp) you can get solid results from 30B quants on 16GB.
For people on 8GB VRAM - Qwen3 7B Q6_K is now genuinely useful. Not as capable as the bigger models but fast, and it handles code questions and summarisation well.
CPU inference via llama.cpp has also improved a lot. If you have a modern CPU with AVX2/AVX512 and 32GB+ RAM, 7B models at Q8 run at tolerable speeds (5-10 tokens/sec). Not for real-time conversation but fine for batch tasks.
For Apple Silicon users: the Metal backend in Ollama/llama.cpp is excellent. An M3 Pro with 36GB unified memory runs 70B models faster than most RTX 4090 setups due to memory bandwidth.
The VRAM situation has genuinely improved. A few months ago 24GB was the sweet spot for decent 30B models. Now with better quantisation (IQ4_XS, Q4_K_M improvements in llama.cpp) you can get solid results from 30B quants on 16GB.
For people on 8GB VRAM - Qwen3 7B Q6_K is now genuinely useful. Not as capable as the bigger models but fast, and it handles code questions and summarisation well.
CPU inference via llama.cpp has also improved a lot. If you have a modern CPU with AVX2/AVX512 and 32GB+ RAM, 7B models at Q8 run at tolerable speeds (5-10 tokens/sec). Not for real-time conversation but fine for batch tasks.
For Apple Silicon users: the Metal backend in Ollama/llama.cpp is excellent. An M3 Pro with 36GB unified memory runs 70B models faster than most RTX 4090 setups due to memory bandwidth.