JulianMorrison comments on Hardened Problems Make Brittle Models - Less Wrong

51 Post author: cousin_it 06 May 2009 06:31PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (40)

You are viewing a single comment's thread.

Comment author: JulianMorrison 06 May 2009 07:45:53PM 5 points [-]

I forget where, but I read a blog post that described these sorts of things as controlled experiments - you want to test one part of your decision apparatus, not have it confounded by all the others.

Also, you are anthropomorphizing the paperclipper AI. It would accept your bargain, but demand not just a handful but as many paperclips as you could be arm-twisted into making - where your pain at the expense and effort is just below where you'd prefer the billion deaths. And then it would exploit your exhausted state to stab you in the back anyway. It's not psychopathic, it's just incrementing a number by expedient means. You can't negotiate with something that won't stay bought.

Comment author: cousin_it 07 May 2009 12:14:36AM *  3 points [-]

What's the goal of that controlled experiment? If my decision apparatus fails on Newcomb's problem or the "true PD", does it tell you anything about my real world behavior?

Comment author: SoullessAutomaton 07 May 2009 10:11:00AM 3 points [-]

It tells us that your real world behavior has the potential to be inconsistent.

Many people carry a decision apparatus that consists of a mess of unrelated heuristics and ad hoc special-cases. Examining extreme cases is a tool for uncovering places where the ad hoc system falls down, so that a more general system can be derived from basic principles, preferably before encountering a real world situation where the flaws in the ad hoc system become apparent.

To my mind, a better analogy than "controlled experiment" would be describing these as decision system unit tests.