What makes "science vs. bayes" a dichotomy? The scientific method is just a special case of Bayesian reasoning. I mean, I understand the point of the article, but it seems like it's way less of a dilemma in practice.

I know this is an old post, I just wanted to write down my answers to the "morality as preference" questions.

Why do people seem to mean different things by "I want the pie" and "It is right that I should get the pie"? Why are the two propositions argued in different ways?

Do the statements, "I liked that movie" and "That movie was good" sound different? The latter is phrased as a statement of fact, while the former is obviously a statement of preference. Unless the latter is said by a movie critic or film professor, no one thinks it's a real statement of fact. It's just a quirk of the English language that we don't always indicate why we believe the words we say. In English, it's always optional to state whether it's a self-evident fact, the words of a trusted expert or merely a statement of opinion.

When and why do people change their terminal values? Do the concepts of "moral error" and "moral progress" have referents? Why would anyone want to change what they want?

"Moral progress" doesn't really refer to individuals. The entities we refer to making "moral progress" tend to be community level, like societies, so I don't really get the first and last questions. As for the concept of moral progress, it refers to the amount of people who have their moral preferences met. The reason democracy is a "more ethical" society than totalitarianism is because more people have a chance to express their preferences and have them met. If I think a particular war is immoral, I can vote for the candidate or law that will end that war. If I think a law is immoral I can vote to change it. I think this theory lines up pretty well with the concept of moral progress.

Why and how does anyone ever "do something they know they shouldn't", or "want something they know is wrong"? Does the notion of morality-as-preference really add up to moral normality?

Usually people who do something they "know is wrong" are just doing something that most other people don't like. The only reason it feels like it's wrong to steal is because society has developed, culturally and evolutionary, in such a way that most people think stealing is wrong. That's really all it is. There's nothing in physics that encodes what belongs to who. Most people just want stuff to belong to them because of various psychological factors.

This is an awesome article. But I've always been bothered by people's expectations when it comes to arriving on time for things. In my experience, people are less annoyed at the person who leaves early than the person who arrives late, even if they miss the same amount of the meeting. The usual reasons people give for avoiding being late (missing content, disrupting the meeting) apply just as much to leaving early. Why the double standard? Also, people are generally more understanding if you have to miss something than if you are an hour late, for some reason.

This is all completely anecdotal, obviously.

It occurs to me, reading your post, that I have almost no idea what people mean by "conscious system". I'm quite certain I am one, and I regularly experience other people apparently claiming to belong to that set too. I suspect that if we can nail down what it means to belong to the set of "conscious systems", we'll be much more readily able to determine if not being a member of that set disqualifies a thing from being an "observer".

It occurred to me that I have no idea what people mean by the word "observer". Rather, I don't know if a solid reductionist definition for observation exists. The best I can come up with is "an optimization process that models its environment". This is vague enough to include everything we associate with the word, but it would also include non-conscious systems. Is that okay? I don't really know.

The problem arises because we don't have a habit of distinguishing "I believe X because I personally investigated the topic", "I believe X because leading experts in the field, who investigated the topic closely, believe X", and "I believe X because other smart-seeming people believe X".

There are languages, real and artificial, in which every sentence grammatically must contain evidential markers, distinguishing such things as "I experienced this myself", "This is a conclusion I inferred from what I experienced", "I heard this from someone I trust", etc.

Always optional in English, unfortunately. It might be useful to cultivate the habit of at least considering the evidential status of whatever one says, and deciding whether it warrants explicit mention. Linking to sources, like I did above, is one sort of evidential.

I'm more interested in how to explain our state of the art understanding of giving a reductionist account of actual physics, which seems to terminate at questions about the ontological status of quantum amplitude.

The next step below quantum amplitude, if there is one, is Tegmarkian multiverses, which are a reduction fixpoint (they reduce to themselves). (There might be one intermediate in between - I have a strong suspicion that quantum amplitudes are a continuous approximation of something discrete). However, there is pretty good reason to believe that we cannot gather evidence about them, even in principle.

A meta-anthropic explanation for why people today think about the Doomsday Argument: observer moments in our time period have not solved the doomsday argument yet, so only observer moments in our time period are thinking about it seriously. Far-future observer moments have already solved it, so a random sample of observer moments that think about the doomsday argument and still are confused are guaranteed to be on this end of solving it.

(I don't put any stock in this. [Edit: this may be because I didn't put any stock in the Doomsday argument either.])

Not sure how much this post has to do with the economic fact of scarcity. Seems like it would be very easy to mistake actual rationality based on economic knowledge for this bias.