= 27d567bc2d48d9f40e9ccc20ea370b15)
mwengler comments on Natural selection defeats the orthogonality thesis - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (71)
I think you're anthropomorphizing. A paperclipper AI doesn't need any values except maximizing paperclips. (To be well defined, that needs something like a time discount function, so let's assume it has one.) If maximizing paperclips requires the AI to survive, then it will try to survive. See Omohundro's "basic AI drives".
Value drift is not necessary for maximizing paperclips. If a paperclip maximizer can see that action X leads to more expected paperclips than action Y, then it will prefer X to Y anyway, without the need for value drift. That argument is quite general, e.g. X can be something like "try to survive" or "behave like mwengler's proposed agent with value drift".
Do you believe that a paper clip maximizer can survive in a world where another self-modifying AI exists whos value is to morph itself into the most powerful and prevalent AI in the world? I don't see how something like a paper clip maximizer, which must split its exponential growth between becoming more powerful and creating paper clips, can ever be expected to outgrow an AI which must only become more powerful.
I realize that my statement is equivalent to saying I don't see how FAI can ever defeat UAI. (Because FAI has more constraint on its values evolution, which must cost it something in growth rate.) So I guess I realize that the conventional wisdom here is that I am wrong, but I don't know the reasoning that leads to my being wrong.
Yeah, if the paperclipper values a paperclip today more than a paperclip tomorrow, then I suppose it will lose out to other AIs that have a lower time discounting rate and can delay gratification for longer. Unless these other AIs also use time discounting, e.g. the power-hungry AI could value a 25% chance of ultimate power today the same as a 50% chance tomorrow.
But then again, such contests can happen only if the two AIs arise almost simultaneously. If one of them has a head start, it will try to eliminate potential competition quickly, because that's the utility-maximizing thing to do.
I suppose that's the main reason to be pessimistic about FAI. It's not just that FAI is more constrained in its actions, it also takes longer to build, and a few days' head start is enough for UAI to win.