Compare to driving vs. being a passenger in a car driving on a twisty road. I often find the former fun, and the latter decidedly uncomfortable, because the first is a tightly coupled feedback loop and the second is highly varying inputs without much in the way of predictability or control.

"Head-eye" coordination is a thing; the neck muscles and the eye muscles communicate closely, and one would expect that the visual cortex might have access to some of that information as well. Breaking that link will violate expectations on a perceptual level.