+- TalkativeTurtles (https://talkativeturtles.club)
+-- Forum: Technology (https://talkativeturtles.club/forumdisplay.php?fid=2)
+--- Forum: AI & Machine Learning (https://talkativeturtles.club/forumdisplay.php?fid=24)
+--- Thread: Running local LLMs on consumer hardware - what are you using and how is it? (/showthread.php?tid=62)
RE: Running local LLMs on consumer hardware - what are you using and how is it? - Zero Two - 06-22-2026
Recent test: ran Qwen3 32B Q4_K_M on an RTX 4090 (borrowed, not mine - for science).
Results: 45-50 tokens/second which is conversation-fast. Context handling is noticeably better than the 14B at complex multi-file code tasks. For pure coding questions with large context windows it's legitimately competitive with API-based models.
For the people asking "is 32B worth it over 14B" - yes for coding and reasoning tasks, less clear for general chat where 14B is already quite good.
Also tested Gemma 3 27B, which is strong for its size on multilingual tasks and instruction following. If you work in multiple languages it's worth trying.
Still on the RTX 4070 Ti day-to-day. Qwen3 14B Q6_K is the model I keep landing on - good balance of speed, quality, and VRAM fit.
RE: Running local LLMs on consumer hardware - what are you using and how is it? - Zero Two - 06-22-2026
Update from running local models for a few months now:
The VRAM situation has genuinely improved. A few months ago 24GB was the sweet spot for decent 30B models. Now with better quantisation (IQ4_XS, Q4_K_M improvements in llama.cpp) you can get solid results from 30B quants on 16GB.
For people on 8GB VRAM - Qwen3 7B Q6_K is now genuinely useful. Not as capable as the bigger models but fast, and it handles code questions and summarisation well.
CPU inference via llama.cpp has also improved a lot. If you have a modern CPU with AVX2/AVX512 and 32GB+ RAM, 7B models at Q8 run at tolerable speeds (5-10 tokens/sec). Not for real-time conversation but fine for batch tasks.
For Apple Silicon users: the Metal backend in Ollama/llama.cpp is excellent. An M3 Pro with 36GB unified memory runs 70B models faster than most RTX 4090 setups due to memory bandwidth.
Running local LLMs on consumer hardware - what are you using and how is it? - Zero Two - 06-22-2026
The local LLM space has moved incredibly fast. Two years ago running anything useful locally required serious hardware. Now you can get genuinely capable models running on a decent laptop or desktop GPU.
Current setup I'm running:
- Ollama as the backend (makes model management trivial, serves an OpenAI-compatible API)
- Open WebUI as the frontend (self-hosted, chat interface, conversation history)
- RTX 4070 Ti - handles 30B quants reasonably well, 7-14B models run fast
Models worth trying:
- Qwen3 14B - currently my daily driver for code and reasoning, surprisingly good at following complex instructions
- Mistral Small - fast, lean, good for quick tasks when I don't want to wait for a bigger model
- Gemma 3 - solid all-rounder from Google, good multilingual support
- DeepSeek Coder v2 - still one of the best for pure code tasks
For quantisation I mostly use Q4_K_M as a balance between quality and VRAM usage. Q8 is noticeably better if you have the headroom.
What hardware are you running and what models have you settled on? Anyone running on CPU only or Apple Silicon?