The fact that it's part of the EU and also bureaucracy is quite a pain. Other than that, no down sides imo. For us it was quite easy, as Switzerland is part of Schengen/ Dublin.