= 27d567bc2d48d9f40e9ccc20ea370b15)
Will_Newsome comments on David Chalmers' "The Singularity: A Philosophical Analysis" - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (202)
(Upvoted.)
I suppose I mostly meant 'irrational', not stupid. I just expected people to expect me to understand basic SIAI arguments like "value is fragile" and "there's no moral equivalent of a ghost in the machine" et cetera. If I didn't understand these arguments after having spent so much time looking at them... I may not be stupid, but there'd definitely be some kind of gross cognitive impairment going on in software if not in hardware.
There were a few cues where I acknowledged that I agreed with the standard argument (AGI won't automatically converge to Eliezer's "good"), but was interested in a different argument about philosophically-sound AIs that didn't necessarily even look at humanity as a source of value but still managed to converge to Eliezer's good, because extrapolated volitions for all evolved agents cohere. (I realize that your intuition is interestingly perhaps somewhat opposite mine here, in that you fear more than I do that there won't be much coherence even among human values. I think that we might just be looking at different stages of extrapolation... if human near mode provincial hyperbolic discounting algorithms make deals with human far mode universal exponential discounting algorithms, the universal (pro-coherence) algorithms will win out in the end (by taking advantage of near mode's hyperbolic discounting). If this idea is too vague or you're interested I could expand on this elsewhere.)
Your parable makes sense, it's just that I don't think I was proposing a perpetual motion device, just something that could sound like a perpetual motion device if I'm not clear enough in my exposition, which it looks like I wasn't. I was just afraid of italicizing and bolding the disclaimers because I thought it'd appear obnoxious, but it's probably less obnoxious than failing to emphasize really important parts of what I'm saying.
What does time discounting have to do with coherence? Of course exponential discounting is "universal" in the sense that if you're going to time-discount at all (and I don't think we should), you need to use an exponential in order to avoid preference reversals. But this doesn't tell us anything about what exponential discounters are optimizing for.
I think your comments would be better received if you just directly talked about your ideas and reasoning, rather than first mentioning your shocking conclusions ("theism might be correct," "volitions of evolved agents cohere") while disclaiming that it's not how it looks. If you make a good argument that just so happens to result in a shocking conclusion, then great, but make sure the focus is on the reasons rather than the conclusion.
vs.
It really really seems like these two statements contradict each other; I think this is the source of the confusion. Can you go into more detail about the second statement?
In particular, why would two agents which both evolved but under two different fitness functions be expected to have the same volition?