Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Tom_McCabe2 comments on Living By Your Own Strength - Less Wrong

23 Post author: Eliezer_Yudkowsky 22 December 2008 12:37AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (31)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Tom_McCabe2 22 December 2008 01:40:30AM 0 points [-]

"But with a sufficient surplus of power, you could start doing things the eudaimonic way. Start rethinking the life experience as a road to internalizing new strengths, instead of just trying to keep people alive efficiently."

It should be noted that this doesn't make the phenomenon of borrowed strength go away, it just outsources it to the FAI. If anything, given the kind of perfect recall and easy access to information that an FAI would have, the ratio of cached historical information to newly created information should be much *higher* than that of a human. Of course, an FAI wouldn't suffer the problem of losing the information's deep structure like a human would, but it seems to be a fairly consistent principle that the amount of cached data grows faster than the rate of data generation.

The problem here- the thing that actually decreases utility- is humans taking actions without sufficient understanding of the potential consequences, in cases where "Humans seem to do very well at recognizing the need to check for global consequences by perceiving local features of an action." (CFAI 3.2.2) fails. I wonder, out of a sense of morbid curiosity, what the record is for the highest amount of damage caused by a single human without said human ever realizing that they did anything bad.