I think that's unfortunate there is so much confusion about what silent payments is and what it can/can't do. It's about making address reuse not only undesirable but outright *impossible*, and doing it with good UX otherwise it won't catch up.
What it's not doing is magically making you immune to other chain analysis heuristics, spyware, surveillance etc.
That may sound underwhelming but address reuse is the one, single worst offender about bitcoin privacy. Solving that is maybe not enough, but it's absolutely necessary if we want to achieve acceptable privacy at some point.
We will still need some kind of coin history obfuscation, something like coinjoins. And also getting people out of exchanges and other KYC services would help a lot.
As for the scanning server seeing your outputs, it's exactly the same privacy model as electrum, and the good thing is that you have the choice, you can use another wallet that relies on blindbit like @Dana wallet, scanning happens client side so slightly more bandwidth/cpu cost for you, but the server doesn't learn anything.