Can I help? In theory should work easily on 96GB, even with 256k context window.
Yes sir I will ask your help tmr, currently only have 2 GPU because I need some PSU crap for the other two. It was due to vllm docker image not being compatible with Gemma