Valentine comments on Compartmentalization in epistemic and instrumental rationality - Less Wrong

77 Post author: AnnaSalamon 17 September 2010 07:02AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (121)
Tags: wireheading singlethink productivity compartmentalization rationality

Comments (121)

You are viewing a single comment's thread. Show more comments above.

Comment author: pjeby 17 September 2010 06:05:21PM *  6 points [-]

The key is simple: the downsides from de-compartmentalization stem from allowing a putative fact to overwrite other knowledge (e.g., letting one’s religious beliefs overwrite knowledge about how to successfully reason in biology, or letting a simplified ev. psych overwrite one's experiences of what dating behaviors work). So, the solution is to be really damn careful not to let new claims overwrite old data.

This is leaving out the danger that realistic assessments of your ability can be hazardous to your ability to actually perform. People who over-estimate their ability accomplish more than people who realistically estimate it, and Richard Wiseman's luck research shows that believing you're lucky will actually make it so.

I think instrumental rationalists should perhaps follow a modified Tarski litany, "If I live in a universe where believing X gets me Y, and I wish Y, then I wish to believe X". ;-)

Actually, more precisely: "If I live in a universe where anticipating X gets me Y, and I wish Y, then I wish to anticipate X, even if X will not really occur". I can far/symbolically "believe" that life is meaningless and I could be killed at any moment, but if I want to function in life, I'd darn well better not be emotionally anticipating that my life is meaningless now or that I'm actually about to be killed by random chance.

(Edit to add a practical example: a golfer envisions and attempts to anticipate every shot as if it were going to be a hole-in-one, even though most of them will not be... but in the process, achieves a better result than if s/he anticipated performing an average shot. Here, X is the perfect shot, and Y is the improved shot resulting from the visualization. The compartmentalization that must occur for this to work is that the "far" mind must not be allowed to break the golfer's concentration by pointing out that the envisioned shot is a lie, and that one should therefore not be feeling the associated feelings.)

Comment author: Valentine 23 October 2012 06:28:42PM 0 points [-]

I've been wondering about this lately. I don't have a crisp answer as yet, though for practical reasons I'm definitely working on it.

That said, I don't think your golfer example speaks to me about the nature of the potential danger. This looks to me like it's highlighting the value of concretely visualizing goals in some situations.

Here are a few potential examples of the kind of phenomenon that nags at me:

  • I'm under the impression that I'm as physically strong as I am because I learned early on how to use the try harder for physical tasks. I noticed when I was a really young kid that if I couldn't make something physically budge and then I doubled my effort, I still had room to ramp up my effort but the object often gave way. (I would regularly test this around age 7 by trying to push buildings over.) Today this has cashed out as simple muscular strength, but when I hit resistance I can barely manage to move (such as moving a "portable" dance floor) my first instinct is still to use the try harder rather than to find an easier way of moving the thing.
  • This same instinct does not apply to endurance training, though. I do Tabata intervals and find my mind generating adamant reasons why three cycles is plenty. I attribute this to practicing thinking that I'm "bad at endurance stuff" from a young age.
  • Possibly relatedly, I don't encounter injuries from doing weight-lifting at a gym, but every time I start a jogging regimen I get a new injury (illiotibal band syndrome, overstretching a tendon running inside my ankles, etc.). This could be coincidence, but it's a weird one, and oddly consistent.
  • My impression is that I am emotionally capable of handling whatever I think I'm emotionally capable of handling, and conversely that I can't handle what I think I can't handle. For instance, when I'm in danger of being rejected in a social setting, I seem to have a good sense of whether that's going to throw me emotionally off-kilter (being upset, feeling really hurt, having a harder time thinking clearly, etc.) and if so by roughly how much. That counts as evidence that I'm just good at knowing the range of emotional impacts I can handle - but the thing is, I seem to be able to game this. If I change how I think about the situation, I'm able to increase or decrease the emotional impact it has on me. Not without bound, but pretty significantly.
  • Whether I enjoy an outing with some friends seems to depend at least in part on my anticipation of how much fun we're going to have. If I get excited enough, it takes some pretty major setbacks to keep me from enjoying myself.

I also faintly remember having heard of some research showing that people who think that a puzzle has been solved are better-able to solve it than those who are told it's unsolved. But I could be misremembering this by quite a bit. I do know that some people speculate that the Manhattan Project might owe a lot of its success to rumors that the Nazis already had the bomb and that the Americans were playing catch-up before the Nazis could build one.

Powered by Reddit Powered by Reddit