Yes, that's true. I try not to forget to take my Yubikey in my pocket when I'm in a public place and have to leave my laptop unattended for a few moments, but I must admit that I tend to leave it connected at all times.
However, I always lock the session.