@nprofile1q... Limited access to hearing aids may play some small part in this, such that an amplified speaker is an accommodation.
Now, add to this the potential that those most with unmet needs of hearing aids may also lack the technical savvy to turn off speaker mode setting activated in a non-public setting, such that it remains active in public settings.
Developing this point, if you see someone holding their phone to their ear despite same phone being on speaker mode, it is possible that they lack sufficient hearing discrimination to even tell the difference between speaker mode and non-speaker mode.
This something myself can testify to as someone who lives with an elder who has hearing loss. Though privileged enough to have hearing aids, even those advanced enough to have bluetooth, when the bluetooth connection fails they are oblivious to that fact, continuing to use their phone as if they are the only person who can hear it.