people have the power fantasies that if they just learned a few magic phrases from the shaman / a guru, they will be capable of doing magic without expending any significant effort.

That's exactly what "expecting short inferential distances" means in this context. Well, there are two options: either this, or assuming that the job also needs some "mysterious essence" you just have to be born with.