= 27d567bc2d48d9f40e9ccc20ea370b15)
benelliott comments on Death Note, Anonymity, and Information Theory - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (47)
That's the thing about mathematical proofs, you need to conclusively rule out every possibility. When dealing with something like a super-intelligence there will be unforeseen circumstances, and nothing short of full mathematical rigour will save you.
I don't know of one off-hand, but I think AIXI can easily be made Timeless. Just modify the bit which says roughly "calculate a probability distribution over all possible outcomes for each possible action" and replace it with "calculate a probability distribution over all possible outcomes for each possible decision".
This may be worth looking into further, I havn't looked very deeply into the literature around AIXI.
This looks like you might be stumbling towards Updateless Decision Theory, which is IMHO even stronger than TDT and may solve an even wider range of problems.
I could come up with an argument for this falling into either category.
I'm claiming that the concept of self-modification is useless since it's a special case of engineering. We have to get engineering right, and if we do that, we'll get self-modification right. I'm struggling to interpret your statement so it bears on my claim. Perhaps you agree with me? Perhaps you're ignoring my claim? You don't seem to be arguing against it.
The scenario I proposed (creating a new UFAI from scratch) doesn't fit well into the second category (self-modification) because I didn't say the original AI goes away. After the misbegotten creation of the UFAI, you have two, the original failed FAI and the new UFAI.
Actually, the second category (bad self-modification) seems to fit well into the first category (destroying the planet in one go), so these two categories don't support the idea that self-modification is a useful concept.
Okay, I think I see what you mean about engineering and self-modification, but I don't think its particularly important, it appears you're thinking in terms of two concepts:
Self-modification: Anything the AI does to itself, for a fairly strict definition of 'itself', as in, the same physical object' or something like that.
Engineering: Building any kind of machine.
However, I think that when most FAI researchers talk about 'self-modification' they mean something broader than your definition, which would include building another AI of roughly equal or greater power but would not include building a toaster.
Any mathematical conclusions drawn about self-modification should apply just as well to any possible method of doing so, and one such method is to construct another AI. Therefore constructing a UFAI falls into the category of 'self modification error' in the sense that it is the sort of thing TDT is designed to help prevent.
Sorry, I don't believe you. I've been paying attention to FAI people for some time and never heard "self-modification" used to include situations where the machine performing the "self-modification" does not modify itself. If someone actually took the initiative to define "self-modification" the way you say, I'd perceive them as being deliberately deceptive.
You're being overly literal.
I have seen SIAI affiliated people on Less Wrong arguing that self modification is impossible to prevent by pointing out that even if you include an injunction against rewriting its own source-code would not prevent it from building something else.
Self modification as you describe it is a useless mathematical concept for Friendliness, as is engineering. Worse, it is not even well-defined, if an AI copies itself onto another computer, and alters the copy, is that self-modification? If it modifies itself, but keeps a copy of its old code around, is that self-modification? Where do you draw the line between the two?
You are violating the principle of charity by assuming that interpretation that makes them look worse.
Mostly when SIAI people talk about self-modification they imagine a machine that just goes in and edits its own source code because that is presumably the most efficient way to self modify and the one that most AI's would use. This does not mean the 'builds another AI' is not included, but it seems like a very stupid and inefficient way to go about things, so you are wasting your time by worrying too much about it.
I'll bet you £100 that whatever conclusions the SIAI eventually draws about self modification will apply just as well to all kinds, I really cannot see how a silly distinction like the one you are making would find its way into a mathematical proof.
We're certainly agreed on that. I'm willing to go further -- I believe any mathematical conclusions that apply to self-modification (your definition) will apply to all possible actions. I don't think your definition carves out a part of the world that has any usefully special properties.
Agreed.
I don't think your definition is well-defined either. Where's the important line between self-modification and making a toaster?
We appear to have no useful definition for the word. Time to stop using it, IMO.
I disagree. "An ideal CDT agent that anticipates facing only action-determined problems will always choose not to self modify" is true "An ideal CDT agent that anticipates facing only action-determined problems will always choose not to do anything" is false.
I'm not a hundred percent clear on this, and I'll be the first to admit that this is a problem and needs to fixed before the problem can be solved. From a very brief period of thought it seems to me a good line to draw is the point at which the new agent becomes more powerful, in the sense of optimization power, than the old one.
I think the word points to something, and I have a feeling that something is the heart of the problem. Interestingly, in terms of mathematical decision theory self-modification seems quite well defined.
After some heat, we're starting to get light. This is good.
I'm not sure that's true. Imagine I'm an ideal CDT. I am in North America. If I wish to react to something that happens in China, there will be some lag. If I could deal with the situation better when there is no lag, I would benefit from cloning myself and sending a copy to China. Would that be self-modification?
(This presupposes that I have access to materials sufficient to copy myself. That might not be true, depending on whether an ideal CDT is physically realizable.)
I should probably have specified that building another agent doesn't really count as self modification if the other agent is identical to the original (or maybe it does count as self modification, but in a very vacuous sense, the same way 'do nothing' is technically an algorithm). So if the other agent is CDT this is not a counter-example.
If the other agent is a more primitive approximation to a CDT then I would view constructing it not as self-modification, but simply as making a choice in an action-determined problem.
If the other agent is TDT or UDT or something then this may count as self-modification, but there is no need to make it this way.
Suppose we use the rigorous definition where an action-determined problem is just a list of choices, each of which leads to a probability distribution across possible outcomes, each of which has a utility assigned to it. In this case I think it is clear that "An ideal CDT agent that anticipates facing only action-determined problems will always choose not to self modify" is true while "An ideal CDT agent that anticipates facing only action-determined problems will always choose not to do anything" is false.
That's plausible, but my counterexample still holds, apparently. I'm sure the desired theorem is true under the right hypotheses, but I can't quite guess what they are right now.
In the cloning scenario, Tim-in-China would have to be a modified version of Tim-in-US. Tim-in-US is optimizing for a utility function U of the environment which perhaps can only be evaluated based on information available to Tim-in-US. Tim-in-China would be constructed to optimize for the best estimate of U it can make, given that it's in China. This best estimate will be different from U. If everything important happens in China and needs quick responses, and Tim-in-US can't move, it might even be worthwhile for Tim-in-US to sacrifice himself to create Tim-in-China.
Tim-in-China is clearly a self-modification, since the utility function is different, right?
In general, we can contrive the circumstances so the agent is paid to self-modify. If the agent is rational and it's paid enough to self-modify, it will.