I do think their activities were bhind most of it. But I also think some Nephilim survived the flood (or the fallen angels created more Nephilim, although in more secretive ways), which is where we get giants like Goliath later.