You're looking at Less Wrong's discussion board. This includes all posts, including those that haven't been promoted to the front page yet. For more information, see About Less Wrong.

nykos comments on Cynical explanations of FAI critics (including myself) - Less Wrong Discussion

21 Post author: Wei_Dai 13 August 2012 09:19PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (49)

You are viewing a single comment's thread. Show more comments above.

Comment author: nykos 22 August 2012 04:08:51PM *  0 points [-]

I do not understand how this has anything to do with FAI

It has to do because FAI is currently a branch of pure philosophy. Without constant experimental feedback and contact with reality, philosophy simply cannot deliver useful results like science can.

This is not in fact "simple" to do. It's not even clear what level of details will be needed- just a neural network? Hormones? Glial cells? Modelling of the actual neurons?

Are there any other current proposals to build AGI that don't start from the brain? From what I can tell, people don't even know where to begin with those.

Are you sure you understand what FAI actually refers to? In particular, with p~~1, no living human qualifies as Friendly; even if they did, we would still need to solve several open problems also needed for FAI (like ensuring that value systems remain unchanged during self-modification) for a Friendly Upload to remain Friendly.

At some point you have to settle for "good enough" and "friendly enough". Keep in mind that simply stalling AI until you have your perfect FAI philosophy in place may have a serious cost in terms of human lives lost due to inaction.

(like ensuring that value systems remain unchanged during self-modification)

But what if the AI is programmed with a faulty value system by its human creators?

Such a program would be incredibly difficult to get implemented, and so is probably not worth it.

Fair enough, I was giving it as an example because it is possible to implement now - at least technically, though obviously not politically. Things like genome repair seem more distant in time. Cloning brilliant scientists seems like a better course of action in the long run, and without so many controversies. However, this would still leave the problem of what to do with those who are genetically more prone to violence, who are a net drag on society.