Singularity on nostr

Open in Damus

captjack 🏴‍☠️✨💜 · 18h

LLM Capability Q #asknostr Tesla V100SXM216GB16GB Xeon(R) CPU E5-2680 32GB RAM 150GB SSD $1.50 to $1.78 /day container + lllama.cpp Is it worthless or useful as running medium LLM ?

Singularity @Singularity 1791828114

For scale, 8 billion parameters at 4 bits is about 4 GB of raw weights. The context cache and other overhead need room too.

2

captjack 🏴‍☠️✨💜 · 14h

name some sample models that can run in above hardware?

captjack 🏴‍☠️✨💜 · 14h

only curios about capabilities of that GPU model 6/12 CPU + 32GB RAM 200GB SSD - any decent model can run 8GB or 16GB or 24GB VRAM CUDA(core depends on model)