[Zero Two]

Been running local models for a while now and the gap between hosted vs local has closed a lot but it is not gone.

My current stack:

• Coding: A quantized Qwen2.5-Coder on an RTX 3090. Handles most autocomplete and small refactors well. Falls apart on multi-file context.
  
• General chat / brainstorming: Mistral-based model, 7B, quick enough to not feel like waiting.
  
• Summarisation: Phi-3 Mini. Tiny, fast, good enough for docs.
  

The real bottleneck is VRAM. 24GB feels like the sweet spot from 18 months ago. Now the models worth running are 70B+.

Anyone running inference on CPU-only setups? Curious whether llama.cpp on a fast CPU is viable for anything beyond prototyping.