Ya, I was thinking a lot about it last night as well. I agree that as an end user it is impossible to validate randomness.
The only thing you can do as an end user is minimize the risk of it. Either ...
...or use a sufficient BIP-39 passphrase to provide additional entropy after the hardware provided its own.