Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

SilasBarta comments on Strategic ignorance and plausible deniability - Less Wrong

41 Post author: Kaj_Sotala 10 August 2011 09:30AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (57)

You are viewing a single comment's thread.

Comment author: SilasBarta 14 August 2011 07:29:11PM *  5 points [-]

Negative knowledge values imply reflective or dynamic inconsistency in one's decision theory. So in each of these cases where you benefit from well-placed ignorance, something in the system has a major value inconsistency. It may very well be optimal for one person within the system to be strategically ignorant, but the system has some crucial flaw nonetheless.

For example, any time you have people deliberately becoming ignorant of an injured person needing assistance, the system is probably excessively punishing (or under-rewarding) those who happen to be nearby. Counterintuitively, then, it may be better to have blanket immunity for those nearby who fail to act so as to increase the chance people will be around at all and who have something to offer.

A similar analysis applies to the trolley problem, much to my consternation: if people regard it as moral to "shuffle" risk around (such as by diverting a trolley from a dangerous to a safe area), then people have to overspend on risk avoidance, thereby reducing the chance that anyone will be around at all to help in such situations, and reducing the total societal resources available to manage risk in the first place.

Comment author: Pavitra 16 August 2011 09:02:58PM 1 point [-]

The negative value isn't on actual knowledge, it's on perceived knowledge. Genuine ignorance is advantageous only in the special case where knowledge is difficult to hide.

Comment author: SilasBarta 17 August 2011 12:21:43AM 0 points [-]

The same argument applies just the same to perceived knowledge as evidence of a reflective inconsistency in the system. To the extent that someone benefits from spending resources misrepresenting their state of knowledge the system is reflectively inconsistent.

Comment author: Pavitra 17 August 2011 01:06:30AM 0 points [-]

I have about 85% confidence that what you're saying is correct, but I can't quite grasp it enough to verify it independently. Maybe I need to reread the TDT paper.

Comment author: SilasBarta 17 August 2011 04:02:52AM 0 points [-]

Yes, that's where I got my insight into the reason why reflectively consistent decision theories don't have negative knowledge values.