This post argues that there is one and only one super goal for any agent, and that goal is simply to exist in a competitive world. Our human sense of other purposes is just an illusion created by our evolutionary origins. It is not the goal of an apple tree to make apples. Rather it is the goal of the apple tree's genes to exist. The apple tree has developed a clever strategy to achieve that, namely it causes people to look after it by producing juicy apples.

Humans are definitely a result of natural selection, but it does not seem to be difficult at all to find goals of ours that do not serve the goal of survival or reproduction at all. Evolution seems to produce these other preferences accidentally. One thing how that happens may be examplified by the following: Our ability to contemplate our thinking from an almost external perspective (sometimes referred to as self-consiousness), is definitely helpful for learning / improving our thinking and could therefore prevail in evolution. However, it may also be the cause of altruism, because it makes every single one of us realize, that they are not very special. (This is by no means an attempt to explain altruism scientifically or something...) More generally, it would be a really strange coincidence, if all cognitive features of an organism in our physical world that serve the goal to survive and reproduce do not serve any other goal. In conclusion, even evolution can (probably) produce (by coincidence) organisms with goals that are not subgoals of the goal to survive and reproduce.

Likewise the paper clip making AI only makes paper clips because if it did not make paper clips then the people that created it would turn it off and it would cease to exist. (That may not be a conscious choice of the AI anymore than than making juicy apples was a conscious choice of the apple tree, but the effect is the same.)

Now, imagine the paper clip maximizer to be more than a robot arm, imagine it to be a well-programmed Seed AI (or the like). As pointed out in ViliamBur's and cousinit's comment, its goal will probably not be easily changed (by coincidence or evolution of several such AIs), for example it could save its source code on several hard drives that are synchronized by a hard-wired mechanism or something... Now this paper clip maximizer would start turning all matter into paper clips. To achieve its goal, it would certainly remain in existence (and thereby give you the illusion of having the supergoal to exist in the first place) and protect its values (which is not extremely difficult). Assuming, it is successful (and we can expect this from a seed AI/superintelligence), the only matter (in reach) left, would at some point be the hardware of the paper clip maximizer itself. What would the paper clip maximizer do then? In conclusion, self-preservation and maybe propagation of value may be important subgoals, but it is certainly not the supergoal.

My experience is that philosophers often carelessly use words to avoid conveying a clear statement, that could be refutable.

If they do it with the purposes of not making a statement that's open to certain refutations I don't see how that's careless.

Eliezer has publications in the field of artificial intelligence? Where?

Philosophy surely is not useless, but some of their arguments just do not make sense to me.

Physicists tend to express bafflement that philosophers care so much about the words. Philosophers, for their part, tend to express exasperation that physicists can use words all the time without knowing what they actually mean.

My experience is that philosophers often carelessly use words to avoid conveying a clear statement, that could be refutable.

This leads directly to the other common misunderstanding among physicists: that philosophers waste their time on grandiose-sounding “Why?” questions that may have no real answers. Perhaps “misunderstanding” isn’t the right word – some such questions are a waste of time, and philosophers do sometimes get caught up in them. (Just as physicists sometimes spend their time on questions that are kind of boring.)

To me, there seems to be a huge difference between "boring" scientific questions and "grandiose-sounding Why?-questions that ..] have no real answers" what Yudkowsky calls [wrong questions, e.g. "Why is there anything instead of nothing?" where it remains very unclear how an answer to that problem would look like.

The quest for absolute clarity of description and rigorous understanding is a crucially important feature of the philosophical method.

As Jacob Bronowski and Bruce Mazlish state in The Western Intellectual Tradition, "our confidence in any science is roughly proportional to the amount of mathematics it employs - that is, to its ability to formulate its concepts with enough precision to allow them to be handled mathematically." In my experience, some philsophers sometimes confuse precision with difficult to read sentences, use of latin words etc. If they knew mathematics (or other formalisms) better, they'd probably produce less material that is of no use (in other scientific disciplines) due to lack of precision.

Science often gives us models of the world that are more than good enough [...]. But that’s not really what drives us to do science in the first place. We shouldn’t be happy to do “well enough,” or merely fit the data – we should be striving to understand how the world really works.

How do they expect an answer to the question of how the world really works to look like? More specifically, what would stop one from responding to any answer with: Yeah, but ... how does the world really, actually work?

There are helpful models and there are unhelpful models. I can model the universe as a wave function in a gigantic Hilbert space, and this is an incredibly general model as it applies to any quantum-mechanical system, but it's not necessarily a helpful model for making predictions at the level I care about most of the time. My claim is that, even if you believe that utility functions can model human preferences (which I also dispute), then it's still true that utility functions are in practice an unhelpful model in this sense.

I can't help but suspect, though, that LW people are drawn to utilitarianism because of what they see as the inevitability of using utility functions to model preferences. Maybe this impression is mistaken.

For practical purposes I agree that it does not help a lot to talk about utility functions. As the We Don't Have a Utility Function article points out, we simply do not know our utility functions but only vague terminal values. However, as you pointed out yourself that does not mean that we do not "have" a utility function at all.

The soft (and hard) failure seems to be a tempting but unnecessary case of pseudo-rationalization. Still, the concept of an agent "having" (maybe in the sense of "acting in a complex way towards optimizing") a utility funktion seems to be very important for defining utilitarian (hence the name, I guess...) ethical systems. In contrast, the notion of terminal values seems to be a lot more vague and not sufficient for defining utilitarianism. Similar things (practical uselessness but theoretical importance) apply to the evaluation of the intelligence of an agent. Therefore, I think that the term 'utility function' is essential for theoretical debate, even though I agree that it is sometimes used in the wrong place.

our body has some impact on our brain and physical activity has some (positive) impact on our body, it seems rational to me to engage in some physical activity rather than "waisting your intelligence" on playing chess (which I also did a lot).

Huh? In the absence of further evidence I'd think the best way to get better at abstract reasoning is to practice abstract reasoning, which chess is going to be better at than sports. Sure, sport has effects on your brain, but so does chess.

e-sports would provide no mental benefit? Here is an article with some studies that disagree: http://www.psychologytoday.com/blog/the-athletes-way/201310/video-gaming-can-increase-brain-size-and-connectivity