TheOtherDave comments on Terminal Bias - Less Wrong

18 [deleted] 30 January 2012 09:03PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (125)

You are viewing a single comment's thread. Show more comments above.

Comment author: wedrifid 31 January 2012 01:18:15PM *  5 points [-]

(I agree that the the thinking is incomplete but disagree on the detail regarding in which sense the thinking is incomplete.)

I currently do not wish to be wireheaded, but I'm uncertain whether that aversion will turn out to be coherent. A technique for preventing AIs from wireheading has not yet been discovered, and there is no proof that one exists. So in that sense LW's thoughts on the subject are necessarily incomplete.

While proving things formally about something as complicated as AI is hard it would be misleading to act as if this makes the question "is it possible to have AIs that don't wirehead" at all an open question. The probability that such an AI is impossible is sufficiently small as to be safely ignored. Objective functions which wirehead are fairly obviously just a subset of objective functions which optimize towards any arbitrary state of the universe. That paperclip maximisers could theoretically exist is not something that is an open question on LW and so therefore it would be rather incoherent for LW to have much doubt whether something in a far broader class that includes paperclippers could exist.

Where the thoughts are actually more complicated and in doubt is in regards to what humans in general or ourselves in particular actually want. Introspection is far too uncertain in that regard!

Comment author: TheOtherDave 31 January 2012 02:37:22PM 2 points [-]

While I agree completely with this comment, I'm upvoting almost entirely because of this sentence:

(I agree that the the thinking is incomplete but disagree on the detail regarding in which sense the thinking is incomplete.)

It sort of encapsulates what attracted me to this site in the first place.

Comment author: wedrifid 31 January 2012 03:06:56PM 1 point [-]

While I agree completely with this comment, I'm upvoting almost entirely because of this sentence:

Thanks, it felt like just leaving the refutation there was more argumentative when I intended more discussion/elaboration so I tried to preemptively de-escalate and emphasize that we mostly agreed.