Comment on The genie knows, but doesn't care

Thu, 05 Sep 2013 09:07:20 GMT

I do not reject that step 10 does not follow if you reject that the AI will not "care" to learn what it is meant to do. But I believe there to be good reasons for an AI created by humans to care.

If you assume that this future software does not care, can you pinpoint when software stops caring?

1. Present-day software is better than previous software generations at understanding and doing what humans mean.

2. There will be future generations of software which will be better than the current generation at understanding and doing what humans mean.

3. If there is better software, there will be even better software afterwards.

4. ...

5. Software will be superhuman good at understanding what humans mean but catastrophically worse than all previous generations at doing what humans mean.

What happens between step 3 and 5, and how do you justify it?

My guess is that you will write that there will not be a step 4, but instead a sudden transition from narrow AIs to something you call a seed AI, which is capable of making itself superhuman powerful in a very short time. And as I wrote in the comment you replied to, if I was to accept that assumption, then we would be in full agreement about AI risks. But I reject that assumption. I do not believe such a seed AI to be possible and believe that even if it was possible it would not work the way you think it would work. It would have to aquire information about what it is supposed to do, for pratical reasons.

Discuss