Fetching all users followed by you would be a mess, just like, currently, fetching the list of people that follow you is a mess (and clients may get it wrong, by listing people that used to follow you...
I think just publishing the individual follow / un-follow (delete follow) is consistent with how most things work in Nostr. If I want to see all the replies to a post, I have to query relays to assemble that list, same to see followers (as you mentioned) or reactions to a post. It is a mess in some sense but I think it's a good model.
1
