= 27d567bc2d48d9f40e9ccc20ea370b15)
TheOtherDave comments on The Design Space of Minds-In-General - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (82)
Reference class confusion. "Paperclipper" refers to any "universe tiler", not just one that tile with paperclips. Specifying paperclips in particular is hard. If you don't care about exactly what gets tiled, it's much easier.
For well-understood goals, that's easy. Just hardcode the goal. It's making goals that can change in a useful way that's hard. Part of the hardness of FAI is we don't understand friendly, we don't know what humans want, and we don't know what's good for humans, and any simplistic fixed goal will cut off our evolution.
I don't see how you could possibly believe that, except out of wishful thinking. Human values are contingent on our entire evolutionary history. My parochial values are contingent upon cultural history and my own personal history. Our values are not universal. Different types of creature will develop radically different values with only small points of contact and agreement.
Well, a number of voices here do seem to believe that there is something instantiable which is the thing that humans actually want and will continue to want no matter how much they improve/modify.
Presumably those voices would not call something that tiles the universe with that thing a paperclipper, despite agreeing that it's a universe tiler, at least technically.
I believe that most voices here think that there are conditions that humans actually want and that one of them is variety. This in no way implies a material tiling. Permanency of these conditions is more questionable, of course.
The only plausible attractive tiling with something material that I could see even a large minority agreeing to would be a computational substrate, "computronium".
If we're talking about conditions that don't correlate well with particular classes of material things, sure.
That's rarely true of the conditions I (or anyone else I know) value in real life, but I freely grant that real life isn't a good reference class for these sorts of considerations, so that's not evidence of much.
Still, failing a specific argument to the contrary, I would expect whatever conditions humans maximally value (assuming there is a stable referent for that concept, which I tend to doubt, but a lot of people seem to believe strongly) to implicitly define a class of objects that optimally satisfies those conditions. Even variety, assuming that it's not the trivial condition of variety wherein anything is good as long as it's new.
Computronium actually raises even more questions. It seems that unless one values the accurate knowledge (even when epistemically indistinguishable from the false belief) that one is not a simulation, the ideal result would be to devote all available mass-energy to computronium running simulated humans in a simulated optimal environment.
I think that needs a bit of refinement. Having lots of food correlates quite well with food. Yet no one here wants to tile the universe with white rice. Other people are a necessity for a social circle, but again few want to tile the universe with humans (okay, there are some here that could be caricatured that way).
I think we all want to eventually have the entire universe able to be physically controlled by humanity (and/or its FAI guardians), but tiling implies a uniformity that we won't want. Certainly not on a local scale, and probably not on a macroscale either.
Right, which is why I borught it up as one of the few reasonable counterexamples. Still, it's the programming makes a difference between heaven and hell.