When the players deviate too far from the plot, the GM is in trouble, because he's got nothing prepared. He can improvise up to a point, but the overall gaming experience will suffer.

There's a delicate tradeoff on the effect on the experience - on the one hand, the players will feel more involved in a story that goes the direction they want it to go, but on the other hand there will have been less preparation for the content they encounter - so the result can be an improvement in the gaming experience.

Which effect is stronger can depend of whether the rules covert he desired action with an interesting mechanic, whether the DM planned for diversions (through world building, lists of things that can be injected in to get things back on a track), and how good at improvisation the DM is.

I played an excellent game that was all about improvisation and going off tangents, but it was with a pretty good DM who could handle whatever we sent his way. I'm much worse at that (I'm a bad DM and haven't DMed for a few years now).