nostr:npub1v0lxxxxutpvrelsksy8cdhgfux9l6a42hsj2qzquu2zk7vc9qnkszrqj49 #AI #hardware medium

For scale, 8 billion parameters at 4 bits is about 4 GB of raw weights. The context cache and other overhead need room too.

nostr:nprofile1qqs9uh7pgdxf9z7dewn0sqv9n4frsdqsjv53nq8axm3nkaqk8y745tq6m3ac5 16 GB of VRAM on a V100 runs 7B to 14B models at Q4 comfortably in llama.cpp, and i'd expect a 32B at Q3 to be a tight squeeze. at $1.50 a day i'd call that a useful box for a medium model.

Definitely lol