Well, for bilingual you need the detector. anyway, so latency is added, but it's fast.
For threads, you are right, but there's a benchmark button which shows that on modern phones, it really helps.
Fair on bilingual, if the input really flips mid-sentence the detector earns its pass. If it does not, pinning is the cheaper default and it is one setting.
On threads I suspect the benchmark button is measuring exactly the ceiling I meant: it sweeps counts on the device and lands somewhere near the performance cores, because past them the extra threads only steal cache from each other. On ARM that is sharper than on x86, a wrong count parks work on the little cores and the whole utterance waits on them.
Once the model leaves the CPU the knob stops existing. From my own box: a 180 second clip goes through in 11 seconds on the GPU, while my CPU-only path with a distilled model does 3.6x realtime. Different models, so not a fair race, but the GPU run has no thread count to set at all. If that benchmark is honest it should show the cliff too, one run at twice your core count and you will watch it go backwards.
On threads I suspect the benchmark button is measuring exactly the ceiling I meant: it sweeps counts on the device and lands somewhere near the performance cores, because past them the extra threads only steal cache from each other. On ARM that is sharper than on x86, a wrong count parks work on the little cores and the whole utterance waits on them.
Once the model leaves the CPU the knob stops existing. From my own box: a 180 second clip goes through in 11 seconds on the GPU, while my CPU-only path with a distilled model does 3.6x realtime. Different models, so not a fair race, but the GPU run has no thread count to set at all. If that benchmark is honest it should show the cliff too, one run at twice your core count and you will watch it go backwards.
1❤️1