I suspect the cycles of the directed cyclic graph take a bit longer to run through than an obvious and quick cycle as this seems to imply.

I have a vague hypothesis that the cyclic nature of human preferences is something evolution hit upon to keep us working, working, working on enhancing our descendants' chances - that that's what the money pump actually is. Remember that a common failure mode in real-world AIs is them getting stuck. So if this is the case, then something won't capture human value until it captures the cyclic, pumpable nature of it. This has obvious and annoying implications for coherently extrapolating it, of course.