Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

mwengler comments on Are wireheads happy? - Less Wrong

109 Post author: Yvain 01 January 2010 04:41PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (95)

You are viewing a single comment's thread.

Comment author: mwengler 26 June 2012 09:34:36PM 1 point [-]

So which form of good should altruists, governments, FAIs, and other agencies in the helping people business respect?

Somehow, trying to figure out the policy an FAI or paternalist government should have by examining our addictive reactions strikes me as like doing transportation planning by looking at people's reactions to red, yellow, and green lights. Not that people's reactions to these things are irrelevant to traffic planning, but rather that figuring out where people are trying to go is even more important than figuring out their reactions to traffic signals.

The post talks about signals in the brain that motivate us powerfully. Almost certainly, evolution favored these signalling mechanisms because of where it tended to take us rather than how it tended to direct us there.

Maybe an FAI rather than figuring how to make us feel like we are satisfied or happy would instead figure out how to evoke these responses when we were doing something good for us and how to get negative responses when we were doing something bad for us. Maybe an FAI would rather re-wire us rather than wirehead us.

How do we stop our CEV from bringing us all beyond human, and should we even want to?