OK, since this is a rationalist scientist community, I should have warned you about the eccentric scientific opinions in Garcia's book. The most valuable thing about Garcia is that he spent 30 years communicating with whoever seemed sincere about the ethical system that currently has my loyalty, so he has dozens of little tricks and insights into how actual humans tend to go wrong when thinking in this region of normative belief space.

Whether an agent's goal is to maximize the number of novel experiences experienced by agents in the regions of space-time under its control or whether the agent's goal is to maximize the number of gold atom in the regions under its control, the agent's initial moves are going to be the same. Namely, your priorities are going to look some like the following. (Which item you concentrate on first is going to depend on your exact circumstances.

(1) ensure for yourself an adequate supply of things like electricity that you need to keep on functioning;

(2) get control over your own "intelligence" which probably means that if you do not yet know how reliably to re-write your own source code, you acquire that ability;

(3a) make a survey of any other optimizing processes in your vicinity;

(3b) try to determine their goals and the extent to which those goals clash with your own;

(3c) assess their ability to compete with you;

(3d) when possible, negotiate with them to avoid negative-sum mutual outcomes;

(4a) make sure that the model of reality that you started out with is accurate;

(4b) refine your model of reality to encompass more and more "distant" aspects of reality, e.g., what are the laws of physics in extreme gravity? are the laws of physics and the fundamental constants the same 10 billion light years away as they are here? -- and so on.

Because those things I just listed are necessary regardless of whether in the end you want there to be lots of gold atoms or lots of happy humans, those things have been called "universal instrumental values" or "common instrumental values".

The goal that currently has my loyalty is very simple: everyone should pursue those common instrumental values as an end in themselves. Specifically, everyone should do their best to maximize the ability of the space, time, matter and energy under their control (1) to assure itself ("it" being the space, time, matter, etc) a reliable supply of electricity and the other things it needs; (2) to get control over its own "intelligence"; and so on.

I might have mixed my statement or definition of that goal (which I call goal system zero) with arguments as to why that goal deserves the reader's loyalty, which might have confused you.

I know it is not completely impossible for someone to understand because Michael Vassar successfully stated goal system zero in his own words. (Vassar probably disagrees with the goal, but that is firm evidence that he understands it.)

--and making deletions transparent to anyone interested in seeing them is not hard. For example, if a registered user of the open-source software behind Hacker News sets the SHOWDEAD bit in his or her profile, then from then on he or she will see unpublished submissions and comments in the place where they would have appeared if they had not been unpublished.

Robin, my most complete description of this system of valuing things consists of this followed by this. Someone else wrote 4 books about it, the best one of which is this.

Eliezer seems to do most of the moderation

It does not seem that way from where I am standing: although I comment more on posts by Eliezer than on posts by Robin and although I am one of the most persistent critics of Eliezer's plans and moral positions, none of my comments on Eliezer's posts were unpublished, but 3 of my comments on Robin's posts were.

Note that I do not think Robin did anything wrong. Contrary to what many commentators believe, unpublishing comments is necessary IMHO to keep the quality of the comments high enough that busy thoughtful people continue to read them. (In fact, if I thought there was a chance he might agree to do it, I would ask Robin to edit or moderate my own posts on my own blog.)

Richard, that's a good point . . . - but then what should I believe about markets and investments, conditioned on scientific and technological progress having been slower than expected?

Well, if you have been misled into believing that scientific progress having been slower than expected entails economic production falling or stagnating, then you will tend to have assigned too high a value to investment strategies or hedging strategies that bet on the performance of the economy as a whole (e.g., shorting index funds). So, perhaps look for more specific bets. E.g., sell short shares in companies that produce "emergence engines" or "consciousness capacitors" or some other output demand for which will fall if progress stagnates in the area of science under discussion.

Mirror neurons and the human empathy-sympathy system play a central role in my definition of consciousness, sentience and personhood or rather my dissolving the question of what is consciousness, sentience and personhood.

Eliezer, there was rapid scientific progress in late-1600s Western Europe even though wealth per capita was vastly lower than current levels. Ditto scientific and technological progress in Victorian England. Could it be that the reason you believe that an economic slump would stall R & D is that the global public-opinion apparatus has fooled you and those you have trusted on this issue? "But it is necessary for scientific progress," might have been a convenient false argument to convince certain sectors of public opinion whose are sceptical about other arguments about the need for pro-growth policies.

Buffy lives in Sunnydale, not Sunnyvale.

I think you've heard this one before: IMHO it has to do with the state in which reality "ends up" and has nothing to do with the subjective experiences of the intelligent agents in the reality. In my view, the greatest evil is the squandering of potential, and devoting the billion galaxies to fun is squandering the galaxies just as much as devoting them to experiments in pain and abasement is. In my view there is no important difference between the two. There would be -- or rather there might be -- an important difference if the fun produced by the billion galaxies is more useful than the pain and abasement -- more useful, that is, for something other than having subjective experiences. But that possibility is very unlikely.

In the present day, a human having fun is probably more useful toward the kinds of ends I expect to be important than a human in pain. Actually the causal relationship between subject human experience and human effectiveness or human usefulness is poorly understood (by me) and probably quite complicated.

After the engineered explosion of engineeered intelligence, the humans are obsolete, and what replaces them is sufficiently different from the humans that my previous paragraph is irrelevant. In my view, there is no need to care whether or what subjective experiences the engineered intelligences will have.

What subjective experiences the humans will have is relevant only because the information helps us predict and control the effectiveness and the usefulness of the humans. We will have proofs of the correctness of the source code for the engineered intelligent agents, so there is no need to inquire about their subjective experiences.

For if all goes well, the question "What is fun?" shall determine the shape and pattern of a billion galaxies.

I object to most of the things Eliezer wants for the far future, but of all the sentences he has written lately, that is probably the one I object to most unequivocally. A billion galaxies devoted to fun does not leave Earth-originating intelligence at lot to devote to things that might be actually important.

That is my dyspeptic two cents.

Not wanting to be in a rotten mood keeps me from closely reading this series on fun and the earlier series on sentience or personhood, but I have detected no indication of how Eliezer would resolve a conflict between the terminal values he is describing. If for example, he learned that the will of the people, oops, I mean, the collective volition, oops, I mean, the coherent extrapolated volition does not want fun, would he reject the coherent extrapolated volition or would he resign himself to a future of severely submaximal quantities of fun?