I'm guessing that's part of how it works. Perhaps we can't consider DMs private? Perhaps the contents are though, did you try to read a DM?
The rest, yeah, those are public events, so yes, if they have your public npub, they can view all the events as if they were you, but they can't sign anything...