True, but those empires came no mass population movements except under the French, and that was reversed. And before that under the Abbasids (lots of dark-brown Arabs from Arabia).

I suspect the Berbers must have been pretty fair-skinned already when the Phoenicians founded Carthage and started written records there.