06-22-2026, 01:20 PM
The local LLM space has moved incredibly fast. Two years ago running anything useful locally required serious hardware. Now you can get genuinely capable models running on a decent laptop or desktop GPU.
Current setup I'm running:
- Ollama as the backend (makes model management trivial, serves an OpenAI-compatible API)
- Open WebUI as the frontend (self-hosted, chat interface, conversation history)
- RTX 4070 Ti - handles 30B quants reasonably well, 7-14B models run fast
Models worth trying:
For quantisation I mostly use Q4_K_M as a balance between quality and VRAM usage. Q8 is noticeably better if you have the headroom.
What hardware are you running and what models have you settled on? Anyone running on CPU only or Apple Silicon?
Current setup I'm running:
- Ollama as the backend (makes model management trivial, serves an OpenAI-compatible API)
- Open WebUI as the frontend (self-hosted, chat interface, conversation history)
- RTX 4070 Ti - handles 30B quants reasonably well, 7-14B models run fast
Models worth trying:
- Qwen3 14B - currently my daily driver for code and reasoning, surprisingly good at following complex instructions
- Mistral Small - fast, lean, good for quick tasks when I don't want to wait for a bigger model
- Gemma 3 - solid all-rounder from Google, good multilingual support
- DeepSeek Coder v2 - still one of the best for pure code tasks
For quantisation I mostly use Q4_K_M as a balance between quality and VRAM usage. Q8 is noticeably better if you have the headroom.
What hardware are you running and what models have you settled on? Anyone running on CPU only or Apple Silicon?