Let's assume that Hermione had actually been sentenced to Azkaban. How many advantages would Quirrelmort have gained?

• Discredited Dumbledore somewhat with a student almost being killed
• Directly eliminated a Light-side witch showing skill at military command and Battle Magic
• Made Harry more vulnerable by knocking out an ally/friend/moral compass
• Driven a wedge between Harry and House Malfoy, eliminating Draco as an ally/friend and ensuring no Malfoy-Potter alliance could form against a resurgent Voldemort
• Broken the Dumbledore-Harry alliance forever if Dumbledore actually let Hermione go to Azkaban; otherwise force Dumbledore to go into open rebellion against the law.
• Made Harry take the majority of the Wizengamot as enemies who needed to be punished, both encouraging him to become darker and the members to have reason to be hostile to Harry in turn.
• Provoked Harry into a (possibly) suicidal effort to destroy Azkaban, which (possibly) could enable a mass breakout of Voldemort supporters from same.
• Isolated Magical Britain from the rest of the wizarding world for sentencing a child to Azkaban.
• Delegitimized the Wizengamot in the eyes of everyone in Magical Britain horrified at the sentence.

There may be more that aren't coming to mind, but, well, the potential payoffs for Quirrelmort were pretty high.

I'm not sure it's appropriate to consider the money the average human will accept for a micromort as a value that's actually useful for making rational decisions, because that's a value that's badly skewed by irrational biases. Actions are mentally categorized into those the thinker does and doesn't believe (on a subconscious level) to possibly lead to death. I doubt the average person even considers a "risk" factor at all when driving their car or walking several blocks to the car (just a time factor and a gasoline factor), unless their trip takes them through a "bad" neighborhood, in which case they'll inflate their perceived risk severalfold without actually looking up that neighborhood's crime rates (moreso if they know someone who was hurt in a manner similar to that). They're probably quite likely to consider a "getting a ticket" risk factor, however. It's sadly true that most people believe themselves invincible and completely ignore many categories of existential risk, thinking only of the "flashier" risks and likely inflating their likelihood. And if you told someone that you would give them $100 and then use a fair RNG and shoot them either on a 1 in 10,000 or 1 in 100,000 chance, I doubt you'd get very different responses.

And I'm going to be so bold as to declare that it's impossible for ANY individual to accurately judge the relative likelihood of two things to kill you without looking it up; "which is more likely" is doable but "is it twice as likely or three times" is not.

edit: The end result of everything I just said is that the "value" being assigned to a micromort is probably more a reflection of how the EPA ran their test than what people really value; they'd get a different result evaluating people's aversion to micromorts via car crash and people's aversion to micromorts via being mugged, and either would be skewed if they first spent a half hour talking about ways to mitigate such a risk (thus reminding you it's there).

I'm planning on doing a statistical study with a sample size of 21 companies. This is a financial study, and the companies chosen are the only ones that will be reporting their 2011 financial results on a certain basis necessary for the testing. (Hence the sample size.)

I'm going to do this regardless of which hypothesis is supported (the null hypothesis, my alternative hypothesis, or neither). So, I'm promising an absence of publication bias. (The null hypothesis will be supported by a finding of little or no correlation; my alternative hypothesis by a negative correlation.)

In this case, the small sample size is the result of available data, and not the result of data-mining. If the results are statistically significant and have a sizable effect, I'm of the opinion that the conclusions will be valid.

I like the first two, and the chess one's pretty interesting though I can't imagine I'd have an easy time getting someone to stand still long enough to hear the whole thing as an argument. But I don't really like the last one. You've been tricked into accepting his premise, that death lets you create more meaningful art, and trying to regain ground from there. It's that premise itself that you should be arguing against--point out all the great literature and art that isn't about death, and that you could still have all of that once death was gone. Also point out that to someone with cancer today the availability of art is probably less valuable than the availability of a cure would be, and there's no reason to assume that'll change if you double his age, even if you double it several times.

Or am I missing some key factor here? Did I misinterpret the lesson?

The key factor is that the 60,20 box is not in isolation - it is the top box, and so not only do you expect it to have more "signal" (gold) than average, you also expect it to have more noise than average.

You can think of the numbers on the boxes as drawn from a probability distribution. If there was 0 noise, this probability distribution would just be how the gold in the boxes was distributed. But if you add noise, it's like adding two probability distributions together. If you're not familiar with what happens, go look it up on wikipedia, but the upshot is that the combined distribution is more spread out than the original. This combined distribution isn't just noise or just signal, it's the probability of having some number be written on the outside of the box.

And so if something is the top, very highest box, where should it be located on the combined distribution?

Now, if you have something that's high on the combined distribution, how much of that is due to signal, and how much of it is due to noise? This is a tougher question, but the essential insight is that the noise shouldn't be more improbable than the signal, or vice versa - that is, they should both be about the same number of standard deviations from their means.

This means that if the standard deviation of the noise is bigger, then the probable contribution of the noise is greater.

Me saying the same thing a different way can be found here.

In any group there's going to be random noise, and if you choose an extreme value, chances are that value was inflated by noise. In Bayesian, given that something has the highest value, it probably had positive noise, not just positive signal. So the correction is to correct out the expected positive noise you get from explicitly choosing the highest value. Naturally, this correction is greater for when the noise is bigger.

So imagine choosing between black boxes. Each black box has some number of gold coins in it, and also two numbers written on it. The first number, A, on the box is like the estimated expected value, and the second number, B, is like the variance. What happened is that someone rolled two distinct dice with B sides, subtracted die 1 from die 2, and added that to the number of gold coins in the box.

So if you see a box with 40, 3 written on it, you know that it has an expected value of 40 gold coins, but might have as few as 37 or as many as 43.

Now comes the problem: I put 10 boxes in front of you, and tell you to choose the one with the most gold coins. The first box is 50, 1 - a very low-variance box. But the last 9 boxes are all high-uncertainty, all with B=20. The expected values printed on them are as follows [I generated the boxes honestly] : 53, 52, 37, 60, 44, 36, 56, 45, 54. Ooh, one of those boxes has a 60 on it! Pick that one!

Okay, don't pick that one. Think about it - there are 9 boxes with high variance, and the one you picked probably has unusually large noise. To be special among 9 proposals with high variance, it probably has noise at the 80th+ percentile. What's the 80th percentile of noise for 1d20 - 1d20? I bet it's larger than 10. You're better off just going with the 50, 1 box.

And it's a good thing you applied that correction, because I generated the boxes by typing "RandomInteger[20,9] - RandomInteger[20,9] + 45" into Wolfram alpha - they each 45 coins each.

So this illustrates that what beating the optimizer's curse really is is a sort of "correction for multiple comparisons." If you have a lot of noisy boxes, some of them will look large even when they're not, even larger than non-noisy boxes.

Loophole: Harry doesn't want to use the stone, he wants to reverse engineer it, and mass produce more. So he can easily commit to not using the stone.

That brings up another point. In the Philosopher's Stone, Dumbledore enchants Erised so that only those who want to find the stone, but not use it, would be able to have it. If Dumbledore did in fact hide the stone in Hogwarts, I can't see either Harry or Quirrell not wanting to use the stone.

Is it even possible for Dumbledore to hide anything in such a way that Harry can get at it, but Quirrell cannot? Harry's major ideal difference -- his war against death -- isn't even understood by Dumbledore. Not to mention that such a hiding place would have been constructed before Dumbledore even met Harry.

Despite having seen you say it in the past, it wasn't until reading this article that in sunk in for me just how little danger we were actually in of Eliezer1997 (or even Eliezer2000) actually making his AI. He had such a poor understanding of the problem, I don't see how he could've gotten there from here without having to answer the question of "Ok, now what do I tell the AI to do?" The danger was in us almost never getting Eliezer2008, or in Eliezer2000 wasting a whole bunch of future-minded peoples' money getting to the point where he realized he was stuck.

Except I suppose he did waste a lot of other people's money and delay present-you by several years. So I guess that danger wasn't entirely dodged after all. And maybe you did have something you planned to tell the AI to do anyways, something simple and useful sounding in and of itself with a tangible result. Probably something it could do "before" solving the question of what morality is, as a warmup. That's what the later articles in this series suggest, at least.

I also peeked at the Creating Friendly AI article just to see it. That, unlike this, looks like the work of somebody who is very, very ready to turn the universe into paperclips. There was an entire chapter about why the AI probably won't ever learn to "retaliate", as if that was one of the most likely ways for it to go wrong. I couldn't even stand to read more than half a chapter and I'm not you.

"To the extent that they were coherent ideas at all" you've said of half-baked AI ideas in other articles. It's nice to finally understand what that means.

I'm kind of surprised at how complicated everyone is making this, because to me the Bayesian answer jumped out as soon as I finished reading your definition of the problem, even before the first "argument" between one and two boxers. And it's about five sentences long:

Don't choose an amount of money. Choose an expected amount of money--the dollar value multiplied by its probability. One-box gets you >(1,000,000*.99). Two-box gets you <(1,000*1+1,000,000*.01). One-box has superior expected returns. Probability theory doesn't usually encounter situations in which your decision can affect the prior probabilities, but it's no mystery what to do when that situation arises--the same thing as always, maximize that utility function.

Of course, while I can be proud of myself for spotting that right away, I can't be too proud because I know I was helped a lot by the fact that my mind was in a "thinking about Eliezer Yudkowsky" mode already, a mode it's not necessarily in by default and might not be when I am presented with a dilemma (unless I make a conscious effort to put it there, which I guess now I stand a better chance of doing). I was expecting for a Bayesian solution to the problem and spotted it even though it wasn't even the point of the example. I've seen this problem before, after all, without the context of being brought up by you, and I certainly didn't come up with that solution at the time.