Poke, if anyone could give credence to the idea that "science is just another religion", it would be you.

"Very Bad Sign #243: Many people all agree that a statement is definitely true but they can't agree on why and/or what the statement means."

Do you mean very bad for the person rejecting the statement? Such a consensus would seem to suggest that there are many reasons for affirming its truth, and many different ways it can be interpreted which are still true.

Michael Vassar: instead of arguing about the meaning of "honest" or "dishonest", do you think it is possible for a person to know by introspection whether or not he has "really been trying" to get at the truth about something or not?

If it is, then people still shouldn't disagree: the one who knows that he hasn't been trying to get at the truth should just admit it, and accept the position of the other guy as more reasonable.

If it isn't, then your account does not supply an argument against Robin Hanson (which I take it you thought that it does.)

Peter Turney: yes, I define Occam's Razor in such a way that all orderings of the hypotheses are Occamian.

The razor still cuts, because in real life, a person must choose some particular ordering of the hypotheses. And once he has done this, the true hypothesis must fall relatively early in the series, namely after a finite number of other hypotheses, and before an infinite number of other hypotheses. The razor cuts away this infinite number of hypotheses and leaves a finite number.

In fact, an anti-Occam prior is impossible. As I've mentioned before, as long as you're talking about anything that has any remote resemblance to something we might call simplicity, things can decrease in simplicity indefinitely, but there is a limit to increase. In other words, you can only get so simple, but you can always get more complicated. So if you assign a one-to-one correspondence between the natural numbers and potential claims, it follows of necessity that as the natural numbers go to infinity, the complexity of the corresponding claims goes to infinity as well. And if you assign a probability to each claim, while making your probabilities sum to 1, then the probability of the more and more complex claims will go to 0 in the limit.

In other words, Occam's Razor is a logical necessity.

I don't see why we shouldn't admit that the more times the sun rises, the more likely it is not to rise the next time... In general life insurance agencies assume that the more days you have lived, the more likely you are to die the next day.

First interpretation is true; third interpretation is partially true. Second interpretation false, fourth interpretation mainly false (because people can execute a process which will change their desires in some unforeseeable manner.)

I've thought about Space Cannibals and the like before (i.e. creatures that kill one of the sexes during sexual reproduction). My suspicion is that even if such creatures evolved and survived, by the time they had a civilization, many would be saying to one another, "There really should be a better way..."

Evidence for this is the fact that even now, there are many human beings claiming it is wrong to kill other animals, despite the fact that humans evolved to kill and eat other animals. Likewise, in the ancestral environment, various tribes usually did kill each other rather than cooperate. But this didn't stop them from beginning to cooperate. So I suspect that Space Cannibals would do something similar. And in any case, I would fully admit that murder couldn't in fact be wrong for the Space Cannibals in the same way it is for us, even if there is an external moral truth.

In answer to Robin's question, assuming that morality exists, it probably has a number of purposes. And if one of the purposes is to preserve things in existence (i.e. moral truths correspond roughly with what is necessary to preserve things), then of course there will be a selection pressure to perceive moral truth. The disclaimer should not be needed, but this is not in any way a claim that it is moral to maximize inclusive genetic fitness.

Eliezer: as you are aware yourself, we don't know how to compute it, nor how to run a computation that computes how to compute it. If we leave it up to the superintelligence to decide how to interpret "helping" and "hurting," it will be in a position no worse than our own, and possibly better, seeing that we are not superintelligent.

"Fuzzle" = "Morally right."

Only in terms of how this actually gets into a human mind, there is a dynamic first: before anyone has any idea of fuzzleness, things are already being sent to the action system. Then we say, "Oh, these are things are fuzzle!", i.e. these are the type of things that get sent to the action system. Then someone else tells us that something else is fuzzle, and right away it gets sent to the action system too.