Yet more "stupid" questions

NancyLebovitz

This is a thread where people can ask questions that they would ordinarily feel embarrassed for not knowing the answer to. The previous thread is at close to 500 comments.

I find that claim to be very implausible: to name just one objection to it, it seems to assume that morality is essentially "logical" and based on rational thought, whereas in practice moral beliefs seem to be much more strongly derived from what the people around us believe in. And in general, the hypothesis that all moral beliefs will eventually converge seems to be picking out a very narrow region in the space of possible outcomes, whereas "beliefs will diverge" contains a much broader space. Do you personally believe in that claim?

That argument seems like it would apply equally well to non-moral beliefs.

1niceguyanon13y

I'm not sure what I was expecting, but I was a little surprised after seeing you say you object to objective morality. I probably don't understand CEV well enough and I am pretty sure this is not the case, but it seems like there is so much similarity between CEV and some form of objective morality as described above. In other words, if you don't think moral beliefs will eventually converge, given enough intelligence, reflection, and gathering data, etc, then how do you convince someone that FAI will make the "correct" decisions based on the extrapolated volition?

0RolfAndreassen13y

It seems there are two claims: One, that each human will be reflectively self-consistent given enough time; two, that the self-consistent solution will be the same for all humans. I'm highly confident of the first; for the second, let me qualify slightly: * Not all human-like things are actually humans, eg psychopaths. Some of these may be fixable. * Some finite tolerance is implied when I say "the same" solution will be arrived at. With those qualifications, yes, I believe the second claim with, say, 85% confidence.