Are you sure Eliezer does equate reflective consistency with alignment with what-he-calls-"right"? Because my recollection is that he doesn't claim either (1) that a reflectively consistent alien mind need have values at all like what he calls right, or (2) that any individual human being, if made reflectively consistent, would necessarily end up with values much like what he calls right.

(Unless I'm awfully confused, denial of (1) is an important element in his thinking.)

I think he is defining "right" to mean something along the lines of "in line with the CEV of present-day humanity". Maybe that's a sensible way to use the word, maybe not (for what it's worth, I incline towards "not") but it isn't the same thing as identifying "right" with "reflectively consistent", and it doesn't lead to a risk of confusion if the two turn out to have different referents (because they can't).

Eliezer equates reflective consistency (a la CEV) with alignment with the big blob of computation he calls "right"

He most certainly does not.

One simple example: Eliezer equates reflective consistency (a la CEV) with alignment with the big blob of computation he calls "right"; I still don't know what argument, technical or non-technical, could justify such an intuition, and I don't know how Eliezer would make tradeoffs if the two did in fact have different referents.

If I understand you correctly then this particular example I don't think I have a problem with, at least not when I assume the kind of disclaimers and limitations of scope that I would include if I were to attempt to formally specify such a thing.

This strikes me as a significant problem in itself, and there are many more problems like it.

I suspect I agree with some of your objections to various degrees.

Possibly he means this.

I'm not clear on the relevance of caring to beliefs. I would prefer that those I care about not be tortured, but once they're out of my future light cone whatever happens to them is a sunk cost- I don't see what I (or they) get from my preferring or believing things about them.

You seem to be modeling the AGW disputant's decision policy as if he is internally representing, in a way that would be introspectively clear to him, his belief about AGW and his public stance about AGW as explicitly distinguished nodes;—as opposed to having "actual belief about AGW" as a latent node that isn't introspectively accessible.

I'm describing it that way but I don't think the introspection is necessary- it's just easier to talk about as if he had full access to his mind. (Private beliefs don't have to be beliefs that the mind's narrator has access to, and oftentimes are kept out of its reach for security purposes!)

But if you want to model actual people's policies in complex situations then the naive Bayesian approach (e.g. with influence diagrams) doesn't work or is way too cumbersome. Does your experience differ from mine?

I don't think I've seen any Bayesian modeling of that sort of thing, but I haven't gone looking for it.

Bayes nets in general are difficult for people, rather than computers, to manipulate, and so it's hard to decide what makes them too cumbersome. (Bayes nets in industrial use, like for fault diagnostics, tend to have hundreds if not thousands of nodes, but you wouldn't have a person traverse them unaided.)

If you wanted to code a narrow AI that determined someone's mood by, say, webcam footage of them, I think putting your perception data into a Bayes net would be a common approach.

Political positions / psychology seem tough. I could see someone do belief-mapping and correlation in a useful way, but I don't see analysis on the level of Steve_Rayhawk's post coming out of a computer-run Bayes net anytime soon, and I don't think drawing out a Bayes net would help significantly with that sort of analysis. Possible but unlikely- we've got pretty sophisticated dedicated hardware for very similar things.

Another more theoretical reason I encourage caution about the "belief as anticipation" idea is that I don't think it correctly characterizes the nature of belief in light of recent ideas in decision theory. To me, beliefs seem to be about coordination

Hmm. I'm going to need to sleep on this, but this sort of coordination still smells to me like anticipation.

(A general comment: this conversation has moved me towards thinking that it's useful for the LW norm to be tabooing "belief" and using "anticipation" instead when appropriate, rather than trying to equate the two terms. I don't know if you're advocating for tabooing "belief", though.)

Actually for the last few days I've been thinking about emailing you, because I've been planning on writing a long exegesis explaining my ideas about decision theory and theology, but you've stated that you don't think it's generally useful to proselytize about your intuitions before you have solid formally justified results that resulted from your intuitions. Although I've independently noticed various ideas about decision theory (probably due to Steve's influence), I haven't at all contributed any new insights, and the only thing I would accomplish with my apologetics is to convince other people that I'm not obviously crazy. You, Nesov, and Steve have made comments that indicate that you recognize that various of my intuitions might be correct, but of course that in itself isn't anything noteworthy: it doesn't help us build FAI. (Speaking of which, do you have any ideas about a better name than "FAI"? 'Friendliness' implies "friendly to humans", which itself is a value judgment. Justified Artificial Intelligence, maybe? Not Regrettable Artificial Intelligence? I was using "computational axiology" for awhile a few years ago, but if there's not a fundamental distinction between epistemology and axiology then that too is sort of misleading.)

Now, I personally think that certain results about decision theory should actually affect what we think of as morally justified, and thus I think my intuitions are actually important for not being damned (whatever that means). But I could easily be wrong about that.

The reason I've made references to theology is actually a matter of epistemology, not decision theory: I think LessWrong and others' contempt for theology and other kinds of academic philosophy is unjustified and epistemically poisonous. (Needless to say, I am extremely skeptical of arguments along the lines of "we only have so much time, we can't check out every crackpot thesis that comes our way": in my experience such arguments are always, without exception the result of motivated cognition.) I would hold this position about normative epistemology even if my intuitions about decision theory didn't happen to support various theological hypotheses.

Anyway, my default position is to write up the aforementioned exegesis in Latin; that way only people that already give my opinions a substantial amount of weight will bother to read it, and I won't be seen as unfairly proselytizing about my own justifiably-ignorable ideas.

(I'm pretty drunk right now, apologies for errors. I might respond to your comment again when I'm sober.)

I may well be being obtuse, but it seems to me that there's something very odd about the phrase "theism / objective morality", with its suggestion that basically the two are the same thing.

This reflects a confusion I have about how popular philosophical opinion is in favor of moral realism, yet against theism. It seems that getting the correct answer to all possible moral problems would require prodigious intelligence, and so I don't really understand the conjunct of moral realism and atheism. This likely reflects my ignorance of the existent philosophical literature, though to be honest like most LessWrongers I'm a little skeptical of the worth of the average philosopher's opinion, especially about subjects outside of his specialty. Also if I averaged philosophical opinion over, say, the last 500 years, then I think theism would beat atheism. Also, there's the algorithm from music appreciation, which is like "look at what good musicians like", which I think would strongly favor theism. Still, I admit I'm confused.

Have you actually argued forcefully that your god is also Aquinas's and Leibniz's? I ask because first you say you could, which kinda suggests you haven't actually done it so far (at least not in public), but then you start talking about "counterarguments", which kinda suggests that you have and people have responded.

I've kinda argued it on the meta-level, i.e. I've argued about when it is or isn't appropriate to assume that you're actually referring to the same concept versus just engaging in syncretism. But IIRC I haven't yet forcefully argued that my god is Leibniz's God. So, yeah, it's a mixture.

I replied to Wei Dai's comment here.

BTW, realistically, I won't be able to reply to your comment re CEV/rightness, though as a result of your comment I do plan on re-reading the meta-ethics sequence to see if "right" is anywhere (implicitly or explicitly) defined as CEV.

(Inebriated, apologies for errors or omissions.)

I don't think your solution actually resolves things since you still need to figure out what weights to assign to each of your biases/values.