GuySrinivasan comments on SotW: Avoid Motivated Cognition - Less Wrong

20 Post author: Eliezer_Yudkowsky 28 May 2012 03:57PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (81)

You are viewing a single comment's thread. Show more comments above.

Comment author: GuySrinivasan 30 May 2012 07:49:51PM *  1 point [-]

I don't always have a problem with motivated cognition, but when I do, my brain usually makes it some or all of the way through the following steps:

  • Notice that I'm becoming more comfortable with (feeling safer about) a decision or action I'm about to make or just made.
  • Notice that the physical cause of my comfort is that I recently had a thought consisting of a reason the decision could have a good outcome.
  • Some brain process that I haven't pinned down yet that feels like and possibly is a mix of noticing optimization by proxy, feeling disdain for non-generalizable reasoning algorithms, and wondering about the true component strength of the reason.
  • Apply my bullshit detector to my comforting thought.
  • If appropriate, begin to legitimately think about the decision or action.

If this is a procedure that will work more generally, then these exercises may help:

Comment author: GuySrinivasan 30 May 2012 07:51:14PM 0 points [-]

Avoid Optimizing Proxy Measures I have a gut-level averse reaction to being comforted by a reason my decision or action may work out when I only saw that reason in order to feel safer. Here is a possible exercise to show students that optimizing proxy measures is a terrible general algorithm: hand out a worksheet of two questions to each student, asking them to answer individually. There are two possible worksheets. The first says "1. Describe in a short paragraph Vatican City. 2. Write any short paragraph you like containing the following concepts: bright, cyclic, white, round, and dappled." The second says "1. Describe in a short paragraph the appearance of the moon. 2. Write any short paragraph you like containing the following concepts: tiny, historic, beautiful, religious, gardens." Then score every paragraph by giving each a score from 0 to 1 based on how well they mentioned each of the 5 concepts, and a final score from 0 to 5 by summing the scores, and note how the proxy measure is reasonable at picking out good descriptions (is it? playtest) if you don't optimize for the proxy measure but terrible if you do.