I don't think you should ignore them at all. In fact, quite the opposite, you should be taking a stance and actively defending your position.
When you are confident in your rights, and have educated yourself on how to defend them, your mental resolve is what saves you when they come knocking.