Damus
Open in Damus
Moon profile picture
Moon
@Moon
What options exist for one to host-self an AI model AND the associated compute for inference? What is required? What is the cost (generally / directionally)?

#AskNostr
8
Moon · 2w
nostr:npub1zgxfjvfyt9j59rnpxfay666njxa7wl4aye8d25jj2wmj9zp0m68q2gx9uk
ABH3PO · 2w
It works fine for me on an NVIDIA 4080 RTX, you need ollama to host download and host the models and provide an api for inference, then you need a UI to imteract which could be openClaw or openWebUI or your own vibecoded think, qwen3:14b works well for agentic stuff, gpt-oss: 20b woeks a little bet...
Scott · 2w
Top of the line: several hundred GB of ram running locally. $20k-$100k of hardware before cost for beefing up electrical wiring setup at home. Middle of the road: maybe a few grand on high end consumer grade GPUs Shitty: 1 word per second or less and shit context but can run on a middle of the road...