What's stopping you from moving a significant number of people to separate lists (follow set (event kind 30000), starter pack (39089), etc.) and keeping in your follow list only the ones you actually interact with?

That's a question I ask myself now and then. Managing this stuff is not my current priority because the relatively limited number of npubs I'm currently subscribed to don't cause me major inconvenience so far.