If the enclave turns off completely, people lose their funds, corr?

Yea I think it would send a long prompt to a model that can't support it... Has than been an issue on your side?

No, the three defaults are not the same.

I upvoted it but now that I check again my upvote is gone! Wtf!

I feel like 90%+ of folks don't even know the difference between 4 and 8 bit. Do customers "want" 8 bit without realizing they do? Unsure. Really no idea if the cost/quality tradeoff is worth it. The scientific community also seems to have little consensus on if it is something very impactful. Some models handle quantization better than others, and it's really hard to tell with the non-deterministic of these models anyways and the fact that they change every few weeks.

Providers lying is a potential latent problem as well. I hope that the industry can eventually figure out how to verify one or the other.

This seems very relevant to the convo as well:

https://x.com/Fried_rice/status/2042423713019412941?s=20

Since we proxy to openrouter for this particular model, you can use any parameters that OR accepts and force 8 bit quantization if you like. We could be better about advertising this feature though.

I’m not sure if the standard
Flow is 4 or 8 bit though. It may still be 8.

I assume you could, haven't tried that flow yet though.

We have free model mode if you want to try that.

We are adding memory in soon. Can you like experience? I'd like to take a look.