I think I'd probably agree with Kaj Sotala's remarks if I had read the passages she^H^H^H^H xe had, and judging by your response in the linked comment, I think I would still come to the same conclusion as she^H^H^H^H xe. I don't think your argument actually cuts with the grain of reality, and I am sure it's not sufficient to eliminate concern about UFAI.

Edit: I hasten to add that I would agree with assumption A in a sufficiently slow-takeoff scenario (such as, say, the evolution of human beings, or even wolves). I don't find that sufficiently reassuring when it comes to actually making AI, though.

Edit 2: Correcting gender of pronouns.

Where is the logical fallacy in the presented arguments

The claim "[Compassion is a universal value] = true. (as we have every reason to believe)" was rejected, both implicitly and explicitly by various commenters. This isn't a logical fallacy but it is cause to dismiss the argument if the readers do not, in fact, have every reason to have said belief.

To be fair, I must admit that the quoted portion probably does not do your position justice. I will read through the paper you mention. I (very strongly) doubt it will lead me to accept B but it may be worth reading.

Ah - that's interesting. I hadn't read the comments. That changes the picture, but by making the result somewhat less relevant.

(Incidentally, when I said, "it may be perfectly obvious", I meant that "some people, observing the statement, may evaluate it as true without performing any complex analysis".)

This quotation accurately summarizes the post as I understand it. (It's a short post.)

I think I speak for many people when I say that assumption A requires some evidence. It may be perfectly obvious, but a lot of perfectly obvious things aren't true, and it is only reasonable to ask for some justification.

I was going to be nice and not say anything, but, yeah.

Re: "Assumption A: Human (meta)morals are not universal/rational. Assumption B: Human (meta)morals are universal/rational.

Under assumption A one would have no chance of implementing any moral framework into an AI since it would be undecidable which ones they were." (source: http://rationalmorality.info/?p=112)

I think we've been over that already. For example, Joe Bloggs might choose to program Joe's preferences into an intelligent machine - to help him reach his goals.

I had a look some of the other material. IMO, Stefan acts in an authoritative manner, but comes across as a not-terribly articulate newbie on this topic - and he has adopted what seems to me to be a bizarre and indefensible position.

For example, consider this:

"A rational agent will always continue to co-exist with other agents by respecting all agents utility functions irrespective of their rationality by striking the most rational compromise and thus minimizing opposition from all agents." http://rationalmorality.info/?p=8

Do you disagree with Eliezer substantively? If so, can you summarize how much of his arguments you've analyzed, and where you reach different conclusions?

Who's considered my main arguments besides you?

Fun investment fact: the two trades that over 40 years turned 1'000 USD into >1'000'000 USD

1'000 USD in Gold on Jan 1970 for 34.94 USD / oz (USD 1'000.00)

1st Trade Sell Gold in Jan 1980 at 675.30 USD / oz (USD 19'327.41) Buy Dow on April 18 1980 at 763.40 (USD 19'327.41)

2nd Trade Sell Dow on Jan 14 2000 at 11'722.98 (USD 296'797.14) Buy Gold on Nov 11 2000 at 264.10 USD / oz (USD 296'797.14)

Portfolio value today: ~1'187'188.57 USD

:-)