soreff comments on Open Thread, August 2010 - Less Wrong

4 Post author: NancyLebovitz 01 August 2010 01:27PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (676)

You are viewing a single comment's thread. Show more comments above.

Comment author: soreff 08 August 2010 04:23:09AM 3 points [-]

That isn't necessarily feasible. My department writes electronic design automation software, and we have a hard time putting in enough diagnostics in the right places to show to us when the code is taking a wrong turn without burying us in an unreadably huge volume of output. If an AI's deciding to lie is only visible as it's having a subgoal of putting an observer's mental model into a certain state, and the only way to notice that this is a lie is to notice that the intended mental state mismatches with the real world in a certain way, and this is sitting in a database of 10,000 other subgoals the AI has at the time - don't count on the scan finding it...

Comment author: timtyler 08 August 2010 07:35:17AM 0 points [-]

Extraspection seems likely to be a design goal. Without it it is harder to debug a system - because it is difficult to know what is going on inside it. But sure - this is an engineering problem with difficulties and constraints.