Been there done that. Several times. Librem 5, Sailfish, PostmarketOS, Maemo, MeeGo, ... But it's hard, as there are certain apps (like banking) that alowly (or fast) pull you back or makes the Linux phone less practical.