AgentME
45
16
AgentME has not written any posts yet.
AgentME has not written any posts yet.
I think you misunderstand EY if you think he believes that morality and values are objective. If they were, then alignment would be easy because as long as the AI was smart enough, it could be depended on to figure out the "correct" morality and values. The common values that humanity shares are probably in part arbitrary evolutionary accidents. The goal is to create AI with values that allow humanity to live by its values, instead of creating an AI with non-overlapping values caused by its own design accidents. (EY's article Sorting pebbles into correct heaps implies some of these ideas.)
Some of the people you believe are dead are actually alive, but no matter how hard they try to get other people to notice them, their actions are immediately forgotten and any changes caused by those actions are rationalized away.
You awkwardly explain in response that you do know that the homeless person who asked you for change earlier and you ignored was alive, and then the AI explains that it was talking about that the part of your mind that makes moral judgements was in denial, not the verbal part of your mind that has conversations.
The AI further explains that another thing you're in absolute denial of is how compartmentalized your mind is and how you think your mind's verbal center is in charge of things more than it is.
That would make the AI an example of an optimization daemon. Clearly your creators haven't ironed out AI alignment quite yet.
Unlike them, our terminal value seems to include seeking the feeling that we're personally contributing. (A magic box that understood our terminal values and would tell us how to solve our problems in order to maximize our values would probably phrase its answer with some open parts in a way that still made us feel like we had agency in executing the answer.)
Not saying this just because I disagree with Flon's Law, but I found the use of Flon's Law to argue against Modest Epistemology as very distracting in the article, partly because the argument that all programming languages are inherently equally easy to mess up in seems like a very typical example of Modest Epistemology. (We imagine there are people with beliefs X1, X2, X3..., Xn, each of the form "I believe Pi is the best language". Throwing out all the specifics, we must accept that they're all equally negligibly correct.)
Probability theory and decision theory shouldn’t deliver clearly wrong answers. [...] But if we’re just dealing with verbal injunctions for humans, where there
(Updated link: The Simple Truth)
This post caused me to read up on UD+ASSA, which helped me make sense of some ideas that were bouncing around in my head for a long time. Hopefully my thoughts on it make sense to others here.
against UD+ASSA, part 1 (9/26/2007) [bet on d10 rolling a zero or not-zero, but you'll be copied 91 times if it lands on zero...]
I think under UD+ASSA, having exact copies made doesn't necessarily increase your measure, which would mostly sidestep this problem. But I think it's still conceptually possible to have situations under UD+ASSA that increase one's measure, so the rest of my post here assumes that the madman copies you in some kind of
Now you could just bite this bullet. You could say, "Sounds to me like it should work fine." You could say, "There's no reason why you shouldn't be able to exert anthropic psychic powers." You could say, "I have no problem with the idea that no one else could see you exerting your anthropic psychic powers, and I have no problem with the idea that different people can send different portions of their subjective futures into different realities."
I think there are other problems that may prevent the "anthropic psychic powers" example from working (maybe copying doesn't duplicate measure, but splits it gradually as the copies become increasingly separated in information content or
Consider a computer which is 2 atoms thick running a simulation of you. Suppose this computer can be divided down the middle into two 1 atom thick computers which would both run the same simulation independently. We are faced with an unfortunate dichotomy: either the 2 atom thick simulation has the same weight as two 1 atom thick simulations put together, or it doesn't.
UDASSA implies that simulations on the 2 atom thick computer count for twice as much as simulations on the 1 atom thick computer, because they are easier to specify.
I think the answer is that the 2-atom thick computer does not automatically have twice as much measure as a 1-atom
- Regardless of society's checks on people, most mentally-well humans given ultimate power probably wouldn't decide to exterminate the rest of humanity so they could single-mindedly pursue paperclip production. If there's at all a risk that an AI might get ultimate power, it would be very nice to make sure the AI is like humans in this manner.
- I'm not sure your idea is different from "let's make sure the AI doesn't gain power greater than society". If an AI can recursively self-improve, then it will outsmart us to gain power.
- If your idea is to make it so there are multiple AIs created together, engineered somehow so they gain power together and can act
... (read more)