Annoyance comments on Raising the Sanity Waterline - Less Wrong

112 Post author: Eliezer_Yudkowsky 12 March 2009 04:28AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (207)

You are viewing a single comment's thread.

Comment author: Annoyance 12 March 2009 07:34:05PM 3 points [-]

If you don't care about consistency, there's really no way to be argued into caring about it. Not rationally, anyway, and not even with unconscious logic. It can only be done by "pushing a motivational button", and without a very detailed model of how your mind works, someone else can only do that by flailing about at random.

Self-consistency is the most basic aspect of effective thought.

Comment author: thomblake 12 March 2009 08:19:22PM 2 points [-]

The contrapositive of this:

If you don't want people to be able to easily manipulate you, be irrational and inconsistent.

Is this right?

Comment author: ciphergoth 13 March 2009 09:23:30AM 5 points [-]

If you were to try this, you would instead be irrational in a consistent way described by well-known cognitive biases, and therefore unusually easy to manipulate.

Comment author: David_Gerard 21 February 2011 11:41:51AM 0 points [-]

How manipulable do you find the determinedly irrational in practice?

Mostly I just find them deeply painful and try to avoid dealing with them except by occasionally poking them with sticks.

Comment author: wedrifid 21 February 2011 05:24:04PM 1 point [-]

How manipulable do you find the determinedly irrational in practice?

Extremely. But when people are already acting against their own interests it is sometimes more convenient to exploit their current practice without trying to influence them significantly.

Comment author: TheOtherDave 21 February 2011 03:59:28PM 2 points [-]

Depends on what I'm trying to get them to do, and how well I understand the framework of their irrationality. In many cases, I find them highly manipulable; in other cases, not at all.

Comment author: Annoyance 12 March 2009 08:39:26PM 5 points [-]

Not exactly. If you want to avoid being manipulated, keeping your motivational structure secret is a good idea. Inconsistency does make it harder for others to anticipate your motivations and positions, but it carries some very high costs.

To the degree that rationality converges on common solutions, it is predictable, and that can be a weakness. But the converge is valuable enough that it's usually worth just putting in some protections.

Comment author: David_Gerard 21 February 2011 11:44:08AM *  1 point [-]

If you want to avoid being manipulated, keeping your motivational structure secret is a good idea.

FWIW, the Church of the SubGenius advocates the "Nameless Mission. To name it is to doom it." I have found this a useful concept when there's something driving me and I've yet to find its true name, but like its results so far. I also gave the concept to a friend who's sick of their field (they've basically won that game) and about to change their entire career, isn't quite sure what to pursue next but is accumulating new hobbies of great interest.