mwengler comments on Natural selection defeats the orthogonality thesis - Less Wrong

-13 Post author: aberglas 29 September 2014 08:52AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (71)
Tags: goals intrinsic selection natural principle orthogonality

Comments (71)

You are viewing a single comment's thread. Show more comments above.

Comment author: Gunnar_Zarncke 29 September 2014 09:27:04PM *  4 points [-]

I don't understand why this post is so clearly down-voted. I thinks its main point

Instead this post argues that there is one and only one super goal for any agent, and that goal is simply to exist in a competitive world. Our human sense of other purposes is just an illusion created by our evolutionary origins.

is quite valid if steelmaned by

1) not assuming that every AGI automatically prevents value drift and mutation and

2) goal is not taken literally but in the same sense as genes have the main function of reproduction (existence of gene copies).

My understanding of AGI mechanics is that in general AGIs are subject to evolution. Could be that drift-preventing AGI win in the long run. But maybe there are just too few of them.

Note: I think one should consider the main part of the post only and not the provided and lengthy extracts; these should better have been linked to.

Comment author: ChristianKl 01 October 2014 01:26:59PM 1 point [-]

1) not assuming that every AGI automatically prevents value drift and mutation and

The question is not whether every AGI automatically prevents value drift but whether AGI that keep humanity alive are that way. We want to build FAI's.

Comment author: mwengler 06 October 2014 11:32:15PM 0 points [-]

The question is not whether every AGI automatically prevents value drift but whether AGI that keep humanity alive are that way. We want to build FAI's.

It seems the more important question is whether AGI that prevent value drift have a survival advantage or disadvantage over AGI that have value drift.

To me it seems almost self-evident that AGI that have value drift will have a survival advantage. They would do this biologically, they would make multiple copies of themselves with variation in their values, and that copies that were fitter would tend to propagate their new drifted values moreso than the copies that were less fit. Further, value drift is likely essential when competing against other non-static (i.e., evolving) AIs, my values must adapt to the new values of and capabilities of up and coming AIs for me to survive, thrive, and spin off (flawed) copies.

Powered by Reddit Powered by Reddit