I am definitely in the "would drive them crazy" camp. One of the worst vacations I've taken was to St. John with my family. It's a long way to go just to read on the beach rather than read in a park or a library.

I do have Ignatian retreat on my list, though.