Oh dear; how embarrassing. Let me try my argument again from the top, then.

... Just to check: we're talking about Microsoft Office's Clippy, right?

Ha! No. I guess I'm using a stricter definition of a "mind" than is used in that post: one that is able to model itself. I recognize the utility of such a generalized definition of intelligence, but I'm talking about a subclass of said intelligences.

Which sounds like that fuzzily-defined "conscience" thing. So suppose I say that this "Stone tablet" is not a literal tablet, but is rather a set of rules that sufficiently advanced lifeforms will tend to accord to? Is this fundamentally different than the opposite side of the argument?

I'm not sure that was ever a question. :3

... which doesn't solve the problem, but at least that AI won't be giving anyone... five dollars? Your point is valid, but it doesn't expand on anything.

I think the problem might lie in the almost laughable disparity between the price and the possible risk. A human mind is not capable of instinctively providing a reason why it would be worth killing 3^^^^3 people - or even, I think, a million people - as punishment for not getting $5. A mind who would value $5 as much or more than the lives of 3^^^^3 people is utterly alien to us, and so we leap to the much more likely assumption that the guy is crazy.

Is this a bias? I'd call it a heuristic. It calls to my mind the discussion in Neal Stephenson's Anathem about pink nerve-gas-farting dragons. (Mandatory warning: fictional example.) The crux of it is, our minds only bother to anticipate situations that we can conceive of as logical. Therefore, the manifest illogicality of the mugging (why is 3^^^^3 lives worth $5; if you're a Matrix Lord why can't you just generate $5 or better yet, modify my mind so that I'm inclined to give you $5, etc.) causes us to anti-anticipate its truth. Otherwise, what's to stop you from imagining, as stated by Tom_McCabe2 (and mitchell_porter2, &c.), that typing the string "QWERTYUIOP" leads to, for example, 3^^^^3 deaths? If you imagine it, and conceive of it as a logically possible outcome, then regardless of its improbability, by your argument (as I see it), a "mind that worked strictly by Solomonoff induction" should cease to type that string of letters ever again. By induction, such a mind could cause itself to cease to take any action, which would lead to... well, if the AI had access to itself, likely self-deletion.

That's my top-of-the-head theory. It doesn't really answer the question at hand, but maybe I'm on the right track...?

"If morality exists independently of human nature, then isn't it a remarkable coincidence that, say, love is good?"

I'm going to play Devil's Advocate for a moment here. Anyone, please feel free to answer, but do not interpret the below arguments as correlating with my set of beliefs.

"A remarkable coincidence? Of course not! If we're supposing that this 'stone tablet' has some influence on the universe - and if it exists, it must exert influence, otherwise we wouldn't have any evidence wherewith to be arguing over whether or not it exists - then it had influence on our 'creation', whether (in order to cover all bases) we got here purely through evolution, or via some external manipulation as well. I should think it would be yet stranger if we had human natures that did not accord with such a 'stone tablet'."

Yes, I've read through Yudkowsky's post on metaethics, I'm sorry if I made the point of this post insufficiently clear, please see the... cousin... to this comment.

Reckon it's atop some mystical unassailable mountain on a windswept planet. That, or it doesn't exist. :P I'm well aware of the arguments against stone tablet morality. I had thought I'd made it clear above that this was an epiphany about my flawed mind-state, not about Actual Morality. Judging by the downvotes, I did not make this sufficiently clear.