No. Matrix is message passing architecture and it works fine for groups. You may need someone for moderation though. But you don't need a centralized server, participants servers can pass messages aro...
As groups grow, then yes you benefit from having hubs, but you can use the moderators servers as hubs. Also at large scales encryption is not that useful... At a 1000 users your data is guaranteed to leak by someone.
❤️1