Damus
Open in Damus
gojiberra · 3d
it says "Nvidia GeForce RTX 3050 (4 GB)". i do have an old dell optiplex server that runs Start9, so maybe i could add the GPU to that. i didn't realize that's all it needs. but i think i skimped on ...
 Raison d'État profile picture
Raison d'État @Raison d'État
You know, that is probably enough to run the smaller Granite models purely in vRAM :) 350M for sure, 1B as well but it might need to be quantised, depending on how many tokens you're feeding it.

Which operating system? Do you have CUDA running on it?
1
gojiberra · 3d
CUDA? now i'm seriously out of my depth. oh, it's a StartOS server, so i think it's their OS. actually, i don't know if i can run anything else on it. or customize the operating system like add CUDA, there's "FreeGPT" which can run models on the Start9 server. so i assume if i put a grpahics ca...