Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Vladimir_Nesov comments on Eutopia is Scary - Less Wrong

34 Post author: Eliezer_Yudkowsky 12 January 2009 05:28AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (121)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Vladimir_Nesov 12 January 2009 09:07:10PM 1 point [-]

About the arguably good future that frightens.

Naively extrapolating from my thoughts about semantics of intelligence and economy of cognition, I suspect that it might be a good thing for people of the future to specialize and increasingly differ in their morality, while composing a community that as a whole robustly continues on the path set by the present human psychology. It doesn't look like a good thing even for some people to start liking "paperclipping", and maybe it isn't, but maybe it is.

Several steps further, if "person" remains a morally significant abstraction, people, while distinct, may become increasingly abstract and interleaved, so that the same process can be interpreted as a part of one person's thought and another person's different thought at once, with substrate interpreted as implementing individual subjective timelines distributed in space and time. One person as a high-level regularity in many other people's thoughts. It's more an implementation detail than morally significant picture though, like running a program in obfuscated binary code, with originally intended algorithm surprisingly managing to carry through.