Psy-Kosh: Hrm. I'd think "avoid destroying the world" itself to be an ethical injunction too.

The problem is that this is phrased as an injunction over positive consequences. Deontology does better when it's closer to the action level and negative rather than positive.

Imagine trying to give this injunction to an AI. Then it would have to do anything that it thought would prevent the destruction of the world, without other considerations. Doesn't sound like a good idea.

No more so, I think, than "don't murder", "don't steal", "don't lie", "don't let children drown" etc.

Of course, having this ethical injunction - one which compels you to positive action to defend the world - would, if publicly known, rather interfere with the Confessor's job.

"In that extremity, I went into the Department of Mysteries and I invoked a password which had never been spoken in the history of the Line of Merlin Unbroken, did a thing forbidden and yet not utterly forbidden."

So, this is the single change that makes this story an AU?

But didn't he note in the confrontation in the Defense Against the Dark Arts class that Harry had chosen Quirrell as his Wise Old Wizard?

"“Harry… you must realize that if you choose this man as your teacher and your friend, your first mentor, then one way or another you will lose him, and the manner in which you lose him may or may not allow you to ever get him back.”"

Dumbledore's comment in his note just don't seem congruent with this comment earlier on, and it's this comment and not the note which seems congruent with reality.

The obvious next question would be to ask if you're OK with your family being tortured under the various circumstances this would suggest you would be.

I've lost the context to understand this question.

How would you react to the idea of people being tortured over the cosmological horizon, outside your past or future light-cone? Or transferred to another, undetectable universe and tortured?

I mean, it's unverifiable, but strikes me as important and not at all meaningless. (But apparently I had misinterpreted you in any case.)

I don't like the idea of it happening. But if it does, I can certainly disclaim responsibility since it is by definition impossible that I can affect that situation if it exists.

The usual version of this I hear is from people who've read Minsky and/or Moravec, and feel we should treat any entity that can pass some reasonable Turing test as legally and morally human. I disagree because I believe a self-aware entity can be simulated -- maybe not perfectly, but to an arbitrarily high difficulty of disproving it -- by a program that is not self-aware. And if such a standard were enacted, interest groups would use it to manufacture a large supply of these fakes and have them vote and/or fight for their side of political questions.

Oh. That's an important distinction, yeah, but standard Singularity arguments suggest that by the time that would come up humans would no longer be making that decision anyway.

Um, if something is smart enough to solve every problem a human can, [how] relevant is the distinction? I mean, sure, it might (say) be lying about it's preferences, but ... surely it'll have exactly the same impact on society, regardless?

That appears to me to be an insoluble problem. Once intelligence (not a particular person but the quality itself) can be impersonated in quantity, how can any person or group know he/they are behaving fairly? They can't. This is another reason I'd prefer that the capability continue not to exist.

On the other hand, this sounds like a tribal battle-cry rather than a rational, non-mindkilled discussion.

It is. At some point I have trouble justifying the one without invoking the other. Some things are just so obvious to me, and so senselessly not-believed by many, that I see no peaceful way out other than dismissing those people. How do you argue with someone who isn't open to reason?

ahem ... I'm ... actually from the other tribe. Pretty heavily in favor of a Nanny Welfare State, and although I'm not sure I'd go quite so far as to say it's "obvious" and anyone who disagrees must be "senseless ... not open to reason".

Care to trade chains of logic? A welfare state, in particular, seems kind of really important from here.

I could argue about the likely consequences, but the logic chain behind my arguments is quite short and begins with postulates about individual rights that you probably don't accept.

When it comes down to it, ethics are entirely a matter of taste (though I would assert that they're a unique exception to the old saw "there's no accounting for taste" because a person's code of ethics determines whether he's trustworthy and in what ways).

I think the trouble with these sort of battle-cries is that they lead to, well, assuming the other side must be evil strawmen. It's a problem. (That's why political discussion is unofficially banned here, unless you make an effort to be super neutral and rational about it.)

One can't really have a moral code (or, I believe, self-awareness!) without using it to judge everyone and everything one sees or thinks of. This more or less demands one take the position that those who disagree are at least misguided, if not evil.

For the same reason, I never expect judges, journalists, or historians to be "unbiased" because I don't believe true "unbiasedness" is possible even in principle.

I hate to spoil the mood for nerd grieving and geek hermeneutics, but Star Trek made a certain kind of sense in the late 1960's (nearly 50 years ago!) when the U.S. and the Soviet Union had real space programs which tried to do new things, one after another. But because astronautics has regressed since then, despite all accelerationist propaganda you hear from transhumanists, this genre of mythological framework for thinking about "the future" makes less and less sense. Given the failure of the "space age," would people 50 years from now, in a permanently Earth-bound reality, bother to watch these ancient shows and obsess over the characters?

You have a credible reason for thinking it will take longer?

So if you were trying to maximise total points, wouldn't it be best to never let it out because you lose a lot more if it destroys the world than you gain from getting solutions?

What values for points make it rational to let the AI out, and is it also rational in the real-world analogue?

many now believe that strong AI may be achieved sometime in the 2020s

Yikes, but that's early. That's a lot sooner than I would have said, even as a reasonable lower bound.

This may be somewhat off-topic, but I've been noticing for some time that your comments frequently receive seemingly "random" downvotes--that is to say, downvotes that appear to be there for no good reason. As an example, the comment that I am replying to right now has a karma score of -1 at the time of posting, despite not being clearly mistaken or in violation of some LW social norm. Checking your profile reveals that your total karma score is in the negatives, despite the fact that you seem like a fairly well-read person, as well as a long-time user of LW. Does anyone have any idea why this is the case?