The hot-plug part gets easier if you don't try to survive it: treat removal as "lock now".
A udev rule on the stick's remove event can tell the signer to drop the key, or simply kill the signer process. Killing it is the more reliable option: in most languages you can't be sure every copy of the ke...