Since we proxy to openrouter for this particular model, you can use any parameters that OR accepts and force 8 bit quantization if you like. We could be better about advertising this feature though.
...
Thank you for confirming :).
The default sorting algorithm is the same as the one you see in my screenshot. It goes to 4-bit quantization.
Yes, we can force 8 bit quantization, but it is more expensive. Do users want that? That's what I'm trying to figure out.
The default sorting algorithm is the same as the one you see in my screenshot. It goes to 4-bit quantization.
Yes, we can force 8 bit quantization, but it is more expensive. Do users want that? That's what I'm trying to figure out.
1