You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

Gram_Stone comments on Learning to get things right first time - Less Wrong Discussion

8 Post author: owencb 29 May 2015 10:06PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (35)
Tags: strategy safety ai

Comments (35)

You are viewing a single comment's thread.

Comment author: Gram_Stone 29 May 2015 11:38:40PM 2 points [-]

Maybe I'm missing the point, but it seems that good old-fashioned formal verification would largely mitigate the issue of bugs in the classical sense, which seems to be what MIRI already intends to use. The greatest problem lies outside of the issue of verifying that the program functions in the obvious sense; it lies in ensuring that the specification is referring to that which we want it to refer: the full complement of human values. This is a philosophical and scientific problem that is likely beyond the purview of formal verification or best programming practices.

Comment author: gjm 30 May 2015 12:47:13AM 5 points [-]

I think you may indeed be missing the point, which is not

  • that formal bug-minimizing software development techniques will suffice to produce safe AI

but

  • that producing safe AI requires (what is currently) extraordinary success at "getting things right first time", so that
    • it would be beneficial to get better at doing that, and to foster a culture of trying to do it,
    • and practice at making bug-free software might be an effective way to do so
    • not least because value-specification and programming are sufficiently parallel that some of the same techniques or patterns of thought might be useful in both.

However, what I take to be your main point -- that value-specification and software development are in fact not terribly similar, so that practice at one may not help much with the other -- is as applicable either way.

Comment author: owencb 30 May 2015 08:49:30AM 2 points [-]

Yes, gjm's summary is right.

I agree that there are some important disanalogies between the two problems. I thought software development was an unusually good domain to start trying to learn the general skill, mostly because it offers easy-to-generate complex challenges where it's simple to assess success.

Comment author: jacob_cannell 30 May 2015 06:17:23PM 0 points [-]

that producing safe AI requires (what is currently) extraordinary success at "getting things right first time", so that

See my reply above - this line of thought is fundamentally mistaken. Simulation testing is far more of an effective general solution than formal verification.

Comment author: gjm 30 May 2015 08:36:34PM 1 point [-]

Just to clarify: I was stating the thesis of the OP, not asserting it. Neither am I now denying it. (I don't find myself altogether convinced either by the OP or by your arguments for why the OP is "probably completely mistaken".)

Powered by Reddit Powered by Reddit