It's already been done and being done again.

Android Open Source Project brings us GrapheneOS (which I'm using right now), LineageOS, e/OS/, CalyxOS, DivestOS, HavocOS, iodeOS, and probably some others I haven't discovered yet

Linux phones are almost usable as well, and not only are they bringing us a new breed of phones, Linux phones are also bringing true "convergence" where your phone is also your computer, and can be docked to a keyboard, mouse and monitor. Linux phones run ARM Linux desktop apps out if the box.