I ssh in to home server run that in tmux, a watch-only wallet though

I don't think you can spend from Sparrow Server. Bitcoin core has CLI wallet but then you have to deal with a Bitcoin core wallet which in case you didn't know doesn't even use BIP39 seedwords.