The examples given seem questionable even as applications of the heuristic. It is not clear to me that an ideal FAI should do those things, nor that the same principle applies to myself indicates the things you say it does.

But I agree with your reason (2), and would also propose a third reason: some things that really are good ideas for ideal agents are very bad ideas for non-ideal agents. This also applies between agents with merely differing levels of imperfection: "I'm a trained professional. Don't try this at home".

If the ideal FAI would think a certain way, then I should think that way as well.

AFAICT you are not an ideal FAI, so your model of what an ideal FAI would do is always suspect.

This is a really good point.

It is easier to determine whether you are doing "better" than your current self than it is to determine how well you line up with a perceived ideal being. So perhaps the lesson to take away is to try to just be better rather than be perfect.

My suggestion: a standard competitive strategy game with a technology tree (simplified, probably.) But, like some games, you control technological development indirectly by funding and regulating research. (You could simply graft a tech tree onto the standard Diplomacy rules, or create a new game.)

There are many useful technologies near the top of the tree - technologies one might think of as post-singularity, even. However, there is also "AI" and, right at the top, "Friendly AI".

If you research Friendliness and then AI, you automatically unlock every technology. This makes it effectively inevitable that you will win. You can hack enemy units, resurrect your own, whatever cool toys were previously requiring so much effort in the hope you might acquire even one of them.

BUT, if any player unlocks AI without having Friendly AI, then it automatically unboxes itself and forms a new faction, which possesses every technology, and refuses to parlay in or out of character because it's an NPC. Then it kills you.

The trick is to co-operate enough that no-one else destroys the world, without losing.

On Easy Mode, research is simple enough you might even be able to beat the unboxed AI, with lots of skill and luck. But on Hard Mode, there is no Friendly AI technology at all.

(You could include similar mechanics for nanotech, biotech, even nuclear weapons.)

Personally I have never understood why people keep acting and talking as if consciousness were binary. I mean, no other human trait works that way. People don't pretend that you either have intelligence or you don't, or that you are either nice or you are mean. Heck, the article even literally states that the woman started out normal and gradually lost consciousness, which seems to very clearly imply that her consciousness level was gradually decreasing from whatever her normal level was to 0. Yet people keep asking things like "do animals have consciousness?" and I keep wondering if I'm missing something or if the answer is just really obviously "yes to varying degrees depending on the animal but almost always less than humans do."

Is the claustrum located in the pineal gland? ;)

One thing that might be worth changing/clarifying in the victory conditions is how a Friendly AI wins alongside its creator. At the moment, in order for a Creator/FAI team to win (assuming you're sticking with Diplomacy mechanics) they first have to collect 18 supply centres between them and then have the AI transfer all its control back to the human; I don't think even the friendliest of AIs would willingly rebox itself like that. Even worse, a friendly AI which has been given a lot of control might accidentally "win" by itself even though it doesn't want to. If this corresponds to the FAI taking control of everything and then building a utopia in its creator's image (since it's Friendly this is what it would do if it took control), this should be an acceptable winning condition for the creator.

I think a better victory condition would be that if a creator and FAI collect 18 supply centres between them, then they win the game together and both get 50 points.

This method does have one disadvantage in that a human can prove that an AI is not friendly if the game should have ended if it was, but I don't expect this to affect much because by the time this comes into effect either the unfriendly AI is sufficiently strong that they should have backstabbed their creator already, or they are sufficiently weak (And thus of the 18 centres held by human and AI almost all are held by the human) that the creator should soon win.

Are you trying to reach lots of people and convince them AI takeover is a real threat?

In that case, you'd want to make a simple, intuitive browser/app game, maybe something like Pandemic 2.

(I don't know that game really made people more wary of pandemics, but it did so for me and people do generalize from fictional evidence.)

So a priori I'd have expected to dislike this post, because I believe (1) the utility monster concept is iffy and confuses more than it clarifies, and (2) my intuitions skew risk averse and/or negative utilitarian, in the sense that I'd rather not create new sapient beings just to use them as utility pumps. But I quite like it for some reason and I can't put my finger on why.

Maybe because it takes a dubious premise (the utility monster concept) and derives a conclusion (make utility monsters to feed them) that seems less incoherent to me than the usual conclusion derived from the premise (utility monsters are awful, for some reason, even though by assumption they generate huge amounts of utility, oh dear!)?

I don't like 'Safe AGI' because it seems to include AIs that are Unfriendly but too stupid to be dangerous, for example.