I agree that the money pump argument is confusing. I think the real problem with intransitive preferences is that they're self-contradictory. If I have an intransitive preference and you simply ask me whether I want A, B, or C, I am unable to answer. As soon as I open my mouth to say something I already want to change my mind, and I'm stuck. However, there's some conceptual slippage to merely wanting to move in a cycle between states over time, such as wanting to travel endlessly from Chicago to New York to San Francisco and back to Chicago again. This may be considered silly and wasteful but there's nothing inherently illogical about it.

But why would the people who are currently in charge of AI labs want to do that, when they could stay in charge and become god-kings instead?

Okay, but you're not comparing like with like. Terminator 2 is an action movie, and I agree that action movies have gotten better since the 1960s. But in terms of sci-fi concepts introduced per second, I would suspect 2001 has more. Some movies from the 1990s that are more straight sci-fi would be Gattaca or Contact, but I don't think many people would consider these categorically better than 2001.

This seems like more a problem of phone addiction than a problem with the movie. Newer movies aren't improved by being cut off from using a palette that includes calm, slow, contemplative, vibe-setting scenes.

Buying something more valuable with something less valuable should never feel like a terrible deal. If it does, something is wrong.

It's completely normal to feel terrible about being forced to choose only one of two things you value very highly. Human emotions don't map onto utility comparisons in the way you're suggesting.

Any agent that makes decisions has an implicit decision theory, it just might not be a very good one. I don't think anyone ever said advanced decision theory was required for AGI, only for robust alignment.

The second reason that I don’t trust the neighbor method is that people just… aren’t good at knowing who a majority of their neighbors are voting for.

This seems like a point in favor of the neighbor method, not against it. You would want people to find "who are my neighbors voting for?" too difficult to readily answer and so mentally replace it with the simpler question "who am I voting for?" thus giving them a plausibly deniable way to admit to voting for Trump.

Can anyone lay out a semi-plausible scenario where humanity survives but isn't dominated by an AI or posthuman god-king? I can't really picture it. I always thought that's what we were going for since it's better than being dead.

I would guess most of them just want their screen readers to work, but a badly written law assigns the responsibility for fixing it to the wrong party, probably due to excessive faith in Coase's theorem.

I would guess it's because the Americans with Disabilities Act provides a private right of action against businesses whose websites are not accessible to people with disabilities, but doesn't say anything about screen reader software bugs.