Dolores1984 comments on General purpose intelligence: arguing the Orthogonality thesis - Less Wrong

20 Post author: Stuart_Armstrong 15 May 2012 10:23AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (156)

You are viewing a single comment's thread. Show more comments above.

Comment author: Wei_Dai 16 May 2012 08:39:57PM 5 points [-]

The primary point of my comment was to argue that an agent that has a goal in the strong sense would not abandon its goal as a result of philosophical consideration.

Such an agent would also not change its decision theory as a result of philosophical consideration, which potentially limits its power.

Philosophy probably has to play some role in clarifying our "pseudo-goals" as actual goals, but so does looking at our "pseudo-goals", however arbitrary they may be.

I wouldn't argue against this as written, but Stuart was claiming that convergence is "very unlikely" which I think is too strong.

Comment author: Dolores1984 16 May 2012 09:37:06PM 1 point [-]

Not strictly. If strongly goal'd agent determines that a different decision theory (or any change to itself) better maximizes its goal, it would adopt that new decision theory or change.