= 27d567bc2d48d9f40e9ccc20ea370b15)
Eliezer_Yudkowsky comments on The Sword of Good - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (292)
Not Hirou, Vhazhar. For some reason, even as a very young child facing religious indoctrination, I couldn't quite accept that Abraham had made the right choice in trying to sacrifice Isaac upon God's command. That was one of my first moral breaks with Judaism. The Lord of Dark is - almost necessarily - actually visualizing situations and reacting to them as if seen, rather than processing words however the people around him expect to process them; there's no other way he could reject the values of his society to that extent, and even then, the amount of convergence he exhibits with our own civilization is implausible barring extremely optimistic assumptions about (a) the amount of absolute coherence (b) our own society's intelligence and (c) the Lord of Dark's intelligence; but of course the story wouldn't have worked otherwise.
Vhazhar's been working on it for some unknown number of years, having successfully realized that sucking the life from worms may be icky but doesn't actually hurt any sentient beings. (Though I wasn't assuming Vhazhar was ancient, he very well could be, and that would make a number of things more plausible, really.) Hirou has a whole civilization behind him and just needed to wake up and actually think.
Okay, Hirou has evidence that Vhazhar is a moral savant. But the reader, and Hirou, sees little evidence that Vhazhar has worked out a formal, rigorous theory of Friendliness. I thought that anything less than that, on your view, virtually guaranteed the obliteration of almost everything valuable.
But I draw a weaker inference from Vhazhar's ability to overcome indoctrination. Yes, it implies that he probably had a high native aptitude for correct moral reasoning. But the very fact that he was subjected to the indoctrination means that he's probably damaged anyways. If someone survives a disease that's usually deadly, you should expect that she went into the disease with an uncommonly strong constitution. But, given that she's had the disease, you should expect that she's now less healthy than average.
Only by AIs. Human uploads would be a whole different story. Not necessarily a good story, but a different story, and one in which - whatever the objective frequency of winning - I'd have to say that, relative to my subjective knowledge, there's a pretty sizable chunk of chance.
If Vhazhar was literally casting a spell to run the world directly, and he wasn't able to take advantage of moral magic like that embodied in the Sword of Good itself (which, conceivably, could be a lot less sophisticated than its name implies) then it's a full-fledged Friendly AI problem.
What are the justifiable expectations one could have about the Sword of Good? In particular, why suppose that it's a Sword of Good in anything other than name only? Why suppose that it's any protection against evil?
I also didn't consider the possibility that Vhazhar was planning to run the world himself directly. A human just doesn't have the computational capacity to run the world. If a human tried to run the world, there would still be both fortune and misfortune.
For that reason, I assumed that his plan was for some extrapolated version of his volition to run the world. But if he's created something that will implement his CEV accurately, hasn't he solved FAI?
There could be less misfortune. A cautious human god who wasn't corrupted by power certainly could plausibly accomplish a lot of good with a few minimal actions. Of course the shaky part is that "cautious" and "not corrupted" part.
Where does the ability to specify complex wishes become distinct from the ability to implement them though? What are the capabilities of a god with human mind? If there is a lot of automation for implementing the wishes, how much of the person's preference does this automation anticipate? In what sense does the limitation on a god's mind to be merely human affect god's capacity to control the world? There doesn't seem to be a natural concept that captures this.
Okay. I had taken the Prophecy of Doom to be saying that there would no longer be both "luck and misfortune". I can see that it could be read otherwise, though.
Well, there are at least several obvious fixes that we humans would want to make to the world we live in, but are unable to. For example, we would like to wipe out the malaria parasite that infects humans. The dragon is bad, the world is full of really, really horrible things, and I'd rather just make it stop rather than worry too much about being corrupted by power.