You know, that is probably enough to run the smaller Granite models purely in vRAM :) 350M for sure, 1B as well but it might need to be quantised, depending on how many tokens you're feeding it.
Whic...
CUDA? now i'm seriously out of my depth.
oh, it's a StartOS server, so i think it's their OS. actually, i don't know if i can run anything else on it. or customize the operating system like add CUDA,
there's "FreeGPT" which can run models on the Start9 server. so i assume if i put a grpahics card in there, it could do it. but i think i would have to get more ROM as well.
maybe a 20$ monthly claude subscription is all i need.
oh, it's a StartOS server, so i think it's their OS. actually, i don't know if i can run anything else on it. or customize the operating system like add CUDA,
there's "FreeGPT" which can run models on the Start9 server. so i assume if i put a grpahics card in there, it could do it. but i think i would have to get more ROM as well.
maybe a 20$ monthly claude subscription is all i need.