Johnicholas comments on Test Your Rationality - Less Wrong

39 Post author: RobinHanson 01 March 2009 01:21PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (76)

You are viewing a single comment's thread.

Comment author: Johnicholas 01 March 2009 03:52:05PM *  5 points [-]

ISO quality certification doesn't look primarily at the results, but primarily at the process. If the process has a good argument or justification that it consistently produces high quality, then it is deemed to be compliant. For example "we measure performance in [this] way, the records are kept in [this] way, quality problems are addressed like [this], compliance is addressed like [such-and-so]".

I can imagine a simple checklist for rationality, analogous to the software carpentry checklist.

  1. Do you have a procedure for making decisions?
  2. Is the procedure available at the times and locations that you make decisions?
  3. How do you prevent yourself from making decisions without following this procedure?
  4. If your procedure depends on calibration data, how do you guarantee the quality of your calibration data?
  5. How does your procedure address (common rationality failure #1)?
  6. et cetera

Sorry, it's just a sketch of a checklist, not a real proposal, but I think you get the idea. Test the process, not the results. Of course, the process should describe how it tests the results.

Comment author: Eliezer_Yudkowsky 01 March 2009 04:43:08PM 4 points [-]

How do you know whether the checklist actually works or if it's just pointless drudgery?

Comment author: Johnicholas 01 March 2009 05:05:46PM 2 points [-]

Sorry, I said "Test the process, not the results", which is a strictly wrong misstatement. It is over-strong in the manner of a slogan.

A more accurate statement would be "Focus primarily on testing process, and secondarily on testing results."

Comment author: Eliezer_Yudkowsky 01 March 2009 05:11:21PM 1 point [-]

Okay but how do you test the results?

Comment author: Johnicholas 01 March 2009 05:47:42PM 0 points [-]

Consult someone else who commented on this article. I didn't have an idea for how to solve Dr. Hanson's original question. I was trying to pull the question sideways.