Caledonian2 comments on Interpersonal Morality - Less Wrong

14 Post author: Eliezer_Yudkowsky 29 July 2008 06:01PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (28)

Sort By: Old

You are viewing a single comment's thread.

Comment author: Caledonian2 29 July 2008 09:38:23PM 6 points [-]

As I understand it, it is not possible for a human to design a machine that is "smarter-than-human", by definition.

Then we couldn't design a machine that could design a machine that would be smarter than we were, either. Machine-2's design couldn't be smarter than it, and machine-2 couldn't be smarter than machine-1 which designed it, which in turn couldn't be smarter than the designers of machine-1: us.

We can't hold in our individual minds a design that is more complex than one of those minds, or in our collective minds what is more complex than the collective. But the design doesn't have to be as complex as the thing that is designed, and the representation of the design is simpler still.

It's trivially easy for a human being to design a data-encoding-and-storage system that can hold more data than is contained in that human's brain. The brain just can't represent that system.