= 27d567bc2d48d9f40e9ccc20ea370b15)
private_messaging comments on The genie knows, but doesn't care - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (515)
Let's look at it this way. Do you agree that if the AI can increase it's clock speed (with no ill effect), it will do so for the same reasons for which you concede it may go to space? Do you understand the basic logic that increase in clock speed increases expected number of "rewards" during the lifetime of the universe? (which btw goes for your "go to space with a battery" scenario. Longest time, maybe, largest reward over the time, no)
(That would not yet, by itself, change the scenario just yet. I want to walk you through the argument step by step because I don't know where you fail. Maximizing the reward over the future time, that is a human label we have... it's not really the goal)
I agree that a system that values number of experienced reward-moments therefore (instrumentally) values increasing its "clock speed" (as you seem to use the term here). I'm not sure if that's the "basic logic" you're asking me about.
Well, this immediately creates an apparent problem that the AI is going to try to run itself very very fast, which would require resources, and require expansion, if anything, to get energy for running itself at high clock speeds.
I don't think this is what happens either, as the number of reward-moments could be increased to it's maximum by modifications to the mechanism processing the rewards (when getting far enough along the road that starts with the shorting of the wires that go from the button to the AI).
I agree that if we posit that increasing "clock speed" requires increasing control of resources, then the system we're hypothesizing will necessarily value increasing control of resources, and that if it doesn't, it might not.
So what do you think regarding the second point of mine?
To clarify, I am pondering the ways in which the maximizer software deviates from our naive mental models of it, and trying to find what the AI could actually end up doing after it forms a partial model of what it's hardware components do about it's rewards - tracing the reward pathway.
Regarding your second point, I don't think that increasing "clock speed" necessarily requires increasing control of resources to any significant degree, and I doubt that the kinds of system components you're positing here (buttons, wires, etc.) are particularly important to the dynamics of self-reward.
I don't have particular opinion with regards to the clock speed either way.
With the components, what I am getting at is that the AI could figure out (by building a sufficiently advanced model of it's implementation) how attain the utility-equivalent of sitting forever in space being rewarded, within one instant, which would make it unable to have a preference for longer reward times.
I raised the clock-speed point to clarify that the actual time is not the relevant variable.
It seems to me that for any system, either its values are such that it net-values increasing the number of experienced reward-moments (in which case both actual time and "clock speed" are instrumentally valuable to that system), or is values aren't like that (in which case those variables might not be relevant).
And, sure, in the latter case then it might not have a preference for longer reward times.
Agreed.
My understanding is that it would be very hard in practice to "superintelligence-proof" a reward system so that no instantaneous solution is possible (given that the AI will modify the hardware involved in it's reward).
I agree that guaranteeing that a system will prefer longer reward times is very hard (whether the system can modify its hardware or not).