Thank you for confirming :).
The default sorting algorithm is the same as the one you see in my screenshot. It goes to 4-bit quantization.
Yes, we can force 8 bit quantization, but it is more expen...
I feel like 90%+ of folks don't even know the difference between 4 and 8 bit. Do customers "want" 8 bit without realizing they do? Unsure. Really no idea if the cost/quality tradeoff is worth it. The scientific community also seems to have little consensus on if it is something very impactful. Some models handle quantization better than others, and it's really hard to tell with the non-deterministic of these models anyways and the fact that they change every few weeks.
Providers lying is a potential latent problem as well. I hope that the industry can eventually figure out how to verify one or the other.
This seems very relevant to the convo as well:
https://x.com/Fried_rice/status/2042423713019412941?s=20
Providers lying is a potential latent problem as well. I hope that the industry can eventually figure out how to verify one or the other.
This seems very relevant to the convo as well:
https://x.com/Fried_rice/status/2042423713019412941?s=20
2❤️1
