= 27d567bc2d48d9f40e9ccc20ea370b15)
Vladimir_Nesov comments on Evaluating the feasibility of SI's plan - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (186)
It is a bad thing, in the sense that "bad" is whatever I (normatively) value less than the other available alternatives, and value-drifted WBEs won't be optimizing the world in a way that I value. The property of valuing the world in a different way, and correspondingly of optimizing the world in a different direction which I don't value as much, is the "value drift" I'm talking about. In other words, if it's not bad, there isn't much value drift; and if there is enough value drift, it is bad.
You're right in a sense that we'd like to avoid it, but if it occurs gradually, it feels much more like "we just changed our minds" (like we definitely don't value "honor" as much as the ancient greeks, etc), as compared to "we and our values were wiped out".
The problem is not with "losing our values", it's about the future being optimized to something other than our values. The details of the process that leads to the incorrectly optimized future are immaterial, it's the outcome that matters. When I say "our values", I'm referring to a fixed idea, which doesn't depend on what happens in the future, in particular it doesn't depend on whether there are people with these or different values in the future.
I think one reason why people (including me, in the past) have difficulty accepting the way you present this argument is that you're speaking in too abstract terms, while many of the values that we'd actually like to preserve are ones that we appreciate the most if we consider them in "near" mode. It might work better if you gave concrete examples of ways by which there could be a catastrophic value drift, like naming Bostrom's all-work-and-no-fun scenario where
or some similar example.