Even in a world where people would be universally willing to do this, it may fail in the case where a unified plot of land is required for a building or project of whatever kind (as in the recent Kelo vs. New London eminent domain case).

Also, "routing around" a particular house may be difficult; it's made impossible if a group of people in the way band together and collectively hold out to take the builder's surplus.