Gunnar_Zarncke comments on Debunking Fallacies in the Theory of AI Motivation - LessWrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (343)
I think most of the misunderstanding boils down to this section:
I think Loosemores reasoning is plausible. But it is based on a misunderstanding of what efficiency and understanding means here. Not efficiency in and of itself but relating to its function. And 'its function' is neither some source code nor the reasoner nor the checking logic but the aggregate effect of these on whatever is optimized within the optimizer. And some optimizer you need otherwise you have no smart choice - and no AGI. And because the AGI presumably is super-smart to optimize this aggregate - however tricky you tune it - the results will likely be problematic.
Consider humans which after all also have a brain with a component which presumably tries to optimize (don't you try to choose what is in a complex way best?). Our preferrences are complex. Extreme cases like sketched for AI which presumaby cause havoc and (self-)destruction have been heavy weeded out by evolution. But still sometimes some peoples brain the optimizer locks into an area of the optimizing space which still fails. Think suicide bomber. Despite lots and lots of 'checking code' (here e.g. self-preservation instinct).