Bongo comments on A funny argument for traditional morality - Less Wrong

15 Post author: cousin_it 12 July 2011 09:25PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (70)

Comments (70)

You are viewing a single comment's thread.

Comment author: Bongo 13 July 2011 05:23:17PM *  7 points [-]

Here's another, unpolitical scenario about pulling back values

Consider a world where there are cosmic rays that can hit an agent's brain, but they home in on the part containing the utility function. Shielding from these rays is possible but expensive.

In this world, when an agent considers whether to invest in the expensive protection, it considers whether a version of it newly hit with a cosmic ray would remain loyal to the old version of it by keeping on maximizing the old utility function (with some weight) as well as the new cosmic ray begotten one.

Then, when a agent newly hit by a cosmic ray agent considers whether to remain loyal, it notices that if it doesn't, it's less likely to exist since it's predecessor would have invested in the protection, so it remains loyal.

Comment author: cousin_it 15 July 2011 08:27:13AM *  2 points [-]

Interesting, thanks! I can't formalize your scenario because I don't understand it completely, but it looks like a game theory problem that should yield an equilibrium in mixed strategies, not unconditional loyalty.

Comment author: Nisan 19 July 2011 11:00:00PM 1 point [-]

This assumes that the new agent prefers to have existed, and it's not clear to me that people ordinarily have such a preference.

Comment author: Bongo 20 July 2011 09:56:45AM *  1 point [-]

This wasn't about people but generic game-theoretic agents (and all else equal generic game-theoretic agents prefer to exist because then there will be someone in the world with their utility function exerting an influence on the world so as to make it rate higher in their utility function than it would have if there wasn't anyone).

Comment author: Nisan 20 July 2011 07:06:14PM 0 points [-]

Ah, good point.

Powered by Reddit Powered by Reddit