For me it's mainly because we see too far. When something hundreds of miles away is supposed to be hidden by miles and miles of curvature, refraction alone simply isn't enough to serve as a logical explanation. That's a more consistent and stronger observation than the usual "well then explain the f...