My system for the local model is a Strix Halo (Framework Desktop) with 128gb vram.
This new Qwen model does not needed to be fully loaded onto VRAM. It can outsource it's ngram embeddings onto normal ...
It hurts thinking about the capex of a framework desktop, but it sure looks like a viable option for ai hardware 😭
Maybe I can convince my wife of using the next tax return to pay for it 😝
Maybe I can convince my wife of using the next tax return to pay for it 😝
💀1