Turns out: Even though I wasn't prompted to enter an Admin PIN the counter was set.to 0.
So... next step: Resetting the GPG part of my Key.
Because something that should take 5 Minutes has to take at least 5 hours and requires wiping a currently running setup...
Remind me again why people think ...