= 27d567bc2d48d9f40e9ccc20ea370b15)
Yvain comments on The Blue-Minimizing Robot - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (159)
Ah, excellent. This post comes at a great time. A few weeks ago, I talked with someone who remarked that although decision theory speaks in terms of preferences and information being separate, trying to apply that into humans is fitting the data to the theory. He was of the opinion that humans don't really have preferences in the decision theoretic sense of the word. Pondering that claim, I came to the conclusion that he's right, and have started to increasingly suspect that CEV-like plans to figure out the "ultimate" preferences of people are somewhat misguided. Our preferences are probably hopelessly path-, situation- and information-dependent. Which is not to say that CEV would be entirely pointless - even if the vast majority of our "preferences" would never converge, there might be some that did. And of course, CEV would still be worth trying, just to make sure I'm not horribly mistaken on this.
The ease at which I accepted the claim "humans don't have preferences" makes me suspect that I've myself had a subconscious intuition to that effect for a long time, which was probably partially responsible for an unresolved disagreement between me and Vladimir Nesov earlier.
I'll be curious to hear what you have to say.
At the end of this, I'm going to try to argue that something like CEV is still justified. Before I started thinking it through I was hoping that taking an eliminativist view of preferences to its conclusion would help tie up the loopholes in CEV, and so far it hasn't done that for me, but it hasn't made it any harder either.
CEV has worse problems that worries about convergence. The big one is that it's such a difficult thing to implement that any AI capable of doing so has already crossed the threshold of extremely dangerous transhuman capability, and there's no real solution to how to regulate its behavior while it's in the process of working on the extrapolation. It could very well turn the planet into computronium before it gets a satisfactory implementation, by which point it doesn't much matter what result it arrives at.
Presumably it matters if it then turns the planet back?
Even if you're the type who thinks a Star Trek transporter is a transportation device rather than a murder+clone system, there's no reason to think the AI would have detailed enough records to re-create everyone. Collecting that level of information would be even harder than getting enough to extrapolate CEV.
So I suppose it might matter to the humanity it re-creates, assuming it bothers. But we'd all still be dead, which is a decidedly suboptimal result.
Well, a neverending utopia fit to the exact specifications of humanity's CEV is still pretty darn good, all things considered.