considering buying hardware to run everything locally. Would should I buy? #asknostr
What’s your budget? Ryzen AI Max+ 395 APUs offer UMA, which you’ll need to a decent model. I like Framework’s desktop offering. A bit more expensive than some chinesium builds, but you’re going to get solid firmware and driver support in linux - and that is king.
Set 1 or more up as an inference “appliance” and that’s all it does. Have everything else run on a different machine.
Stick with Ubuntu Server to start with - just easier support. Go ROCm + llama.cpp first, then fall back to vulkan if there’s issues. Can go Ollama when things are looking good.
I aim to build a Nix port once it’s all stable, making rebuilds of these “appliances” simple.
Set 1 or more up as an inference “appliance” and that’s all it does. Have everything else run on a different machine.
Stick with Ubuntu Server to start with - just easier support. Go ROCm + llama.cpp first, then fall back to vulkan if there’s issues. Can go Ollama when things are looking good.
I aim to build a Nix port once it’s all stable, making rebuilds of these “appliances” simple.
1❤️1