I think the goal was to make a model that could keep learning and improving. It's not panning out that way, so with energy constraints we are learning small models make more sense now.
I do think the...
I'm imagining something like modules where model domain can be expanded or reduced as needed. So you basically have a very basic component, call it module 0 (some basic knowledge or ability), that can be "taught" different domains.