Although technically any AI has a utility function, the usual arguments about the failings of utility functions don't apply to unusual utility functions like the type that may be more easily described using other paradigms.

For instance, Google Maps can be thought of as having a utility function: it gains higher utility the shorter the distance is on the map. However, arguments such as "you can't exactly specify what you want it to do, so it might blackmail the president into building a road in order to reduce the map distance" aren't going to work, because you can program Google Maps in such a way that it never does that sort of thing.

i agree with the sentiment behind what you say here.

The difficult part is to shake ourselves free of any unexamined, implicit assumptions that we might be bringing to the table, when we talk about the problem.

For example, when you say:

And this is the reason why we should be worried about an AI with a poorly made utility function

... you are talking in terms of an AI that actually HAS such a thing as a "utility function". And it gets worse: the idea of a "utility function" has enormous implications for how the entire control mechanism (the motivations and goals system) is designed.

A good deal of this debate about my paper is centered in a clash of paradigms: on the one side a group of people who cannot even imagine the existence of any control mechanism except a utility-function-based goal stack, and on the other side me and a pretty large community of real AI builders who consider a utility-function-based goal stack to be so unworkable that it will never be used in any real AI.

Other AI builders that I have talked to (including all of the ones who turned up for the AAAI symposium where this paper was delivered, a year ago) are unequivocal: they say that a utility-function-and-goal-stack approach is something they wouldn't dream of using in a real AI system. To them, that idea is just a piece of hypothetical silliness put into AI papers by academics who do not build actual AI systems.

And for my part, I am an AI builder with 25 years experience, who was already rejecting that approach in the mid-1980s, and right now I am working on mechanisms that only have vague echoes of that design in them.

Meanwhile, there are very few people in the world who also work on real AGI system design (they are a tiny subset of the "AI builders" I referred to earlier), and of the four others that I know (Ben Goertzel, Peter Voss, Monica Anderson and Phil Goetz) I can say for sure that the first three all completely accept the logic in this paper. (Phil's work I know less about: he stays off the social radar most of the time, but he's a member of LW so someone could ask his opinion).

Applause!

I like the minimalist UI - compare with far more cluttered sites like memrise and fluentu .

No, today's good theistic scientists, to the extent that they still exist, are precisely those who have stopped to take religion seriously as a scientific hypothesis.

That is extremely obvious and something of the first thing I said in this article is that you mustn´t make a religious belief into a premise for science. Of course you can´t mix up scientific work with religion.

He was agnostic most part of his life. But you are right that at one point in his life he openly declared himself an atheist. I remembered wrong. Heisenberg at the other hand was openly a theist. If you can only think of Francis Collins, maybe you shouldn´t base all your beliefs on just one person?

Wikipedia on Schrödinger:

Despite being raised in a religious household, he called himself an atheist. However, he had strong interests in Eastern religions, pantheism and used religious symbolism in his works. He also believed his scientific work was an approach to the godhead, albeit in a metaphorical sense.

Why did you mention him then? Why not mention Erwin Schrödinger or Heisenberg for example?

I don´t fully agree though. There many scientists today whom are also theists. What do you base that last sentence on? As for Newton trying to extract information from the bible, as far as I remember this was in his senior years when he started with numerology. He kept this obscure hobby quite secret and separated from his scientific work as far as I know.

You are right :) I was amused when I first heard about his theories regarding the soul. We have to remember though, that he was philosopher and not a real doctor or a scientist in the modern meaning. Besides, not all old theistic scientists based their science on religious premises.

I see you mention Rene Descartes. He believed in the existence of souls, and, taking that hypothesis seriously, he concluded there has to be a way for the soul to send signals to the body. He went looking for an organ that might fulfil this purpose, and concluded that it is the pineal gland.

This conclusion is false, the true function of the pineal gland is known today, but it illustrates a point : the old theistic scientists tended to take religion seriously, they viewed it as a valid scientific hypothesis whose implications in the real world could be studied. In that sense, they were wholly unlike modern-day theists who all too often seek refuge in unfalsifyability.

No, he's certainly not completely wrong, but he's bringing up irrelevant complications and missing the main point. Quantum vs classical has nothing to do with it, for example.