HPMOR is an excellent choice.

What's your audience like? A book club (presumed interest in books, but not significantly higher maturity or interest in rationality than baseline), a group of potential LW readers, some average teenagers?

The Martian (Andy Weir) would be a good choice for a book-club-level group- very entertaining to read and promotes useful values. Definitely not of the "awareness raising" genre, though.

If you think a greater than average amount of them would be interested in rationality, I'd consider spending some time on Ted Chiang's work- only short stories at the moment, but very well received, great to read, and brings up some very good points that I'd bet most of your audience hasn't considered.

Edit: Oh, also think about Speaker for the Dead.

Giving What We Can recommends over 10% of income. I currently donate what I can spare when I don't need the money, and have precommitted to 50% of my post-tax income in the event that I acquire a job that pays over $30,000 a year (read: once I graduate college). The problem with that is that you already have a steady income and have arranged your life around it- it's much easier to not raise expenses in response to income than it is to lower expenses from a set income.

Like EStokes said, however, the important thing isn't to get caught up in how much you should be donating in order to meet some moral requirement. It's to actually give in a way that you, yourself, can give. We all do what we can :)

Your problem setup contains a contradiction. You said that X and X*_i are identical copies, and then you said that they have different utility functions. This happened because you defined the utility function over the wrong domain; you specified it as (world-history, identity)=>R when it should be (world-history)=>R.

So on the face of it it seems that the only accessible outcomes are:

• original-X chooses "sim" and gets +1; all simulated copies also choose "sim" and get +0.2 (and then get destroyed?)
• original-X chooses "not sim" and gets +0.9; no simulated copies are made

and it seems like in fact everyone does better to choose "sim" and will do so. This is also fairly clearly the best outcome on most plausible attitudes to simulated copies' utility, though the scenario asks us to suppose that X doesn't care about those.

I'm not sure what the point of this is, though. I'm not seeing anything paradoxical or confusing (except in so far as the very notion of simulated copies of oneself is confusing). It might be more interesting if the simulated copies get more utility when they choose "not sim" rather than less as in the description of the scenario, so that your best action depends on whether you think you're in a simulation or not (and then if you expect to choose "sim", you expect that most copies of you are simulations, in which case maybe you shouldn't choose "sim"; and if you expect to choose "not sim", you expect that you are the only copy, in which case maybe you should choose "sim").

I'm wondering whether perhaps something like that was what pallas intended, and the current version just has "sim" and "not sim" switched at one point...

Most of what I know about CEV comes from the 2004 Yudkowsky paper. Considering how many of his views have changed in similar timeframes, and how the paper states multiple times that CEV is a changing work in progress, this seems like a bad thing for my knowledge of the subject. Has there been any significant public changes since then, or are we still debating based on that paper?

Why do you think the scale of the bias is unlikely to be more than a few decades?

Because the differences between estimates made by people who should be highly selected for optimism (e.g. AGI researchers) and people who should be much less so (other AI researchers, and more importantly but more noisily, other people) are only a few decades.

Suppose we created an AGI the greatest mind ever conceived and we created it to solve humanities greatest problems. An ideal methodology for the AGI to do this would to ask for factories to produce physical components to copy itself over and over. The AGI then networks its copies all over the world creating a global mind and then generates a hoard of "mobile platforms" from which to observe, study and experiment with the world for its designed purpose.

The "robbery" is not intentional, its not intending to make mankind meaningless. The machine is merely meeting its objective of doing its utmost to find the solutions to problems for humanity. The horror is that as the machine mind expands networking its copies together and as it sends its mobile platforms out into the world eventually human discovery and invention would be dwarfed by this being. Outside of social and political forces destroying or dismantling the machine(quite likely) human beings would ultimately be forced with a problem: with the machine thinking of everything for us, and its creations doing all the hard work we really have nothing to do. In order to have anything to do we must improve ourselves to at the very lest have a mind that can compete.

Basically this is all a look at what the world would be like if our current AGI researchers did succeed in building their ideal machine and what it would mean for humanity.

The answer is, as always, "it depends." Seriously , though- I time discount to an extent, and I don't want to stop totally. I prefer more happiness to less, and I don't want to stop. (I don't care about ending date, and I'm not sure why I would want to). If a trade off exists between starting date, quality, and duration of a good situation, I'll prefer one situation over the other based on my utility function. A better course of action would be to try and get more information about my utility function, rather than debating which value is more sacred than the rest.

The answer is, as always, "it depends." Seriously , though- I time discount to an extent, and I don't want to stop totally. I prefer more happiness to less, and I don't want to stop. (I don't care about ending date, and I'm not sure why I would want to). If a trade off exists between starting date, quality, and duration of a good situation, I'll prefer one situation over the other based on my utility function. A better course of action would be to try and get more information about my utility function, rather than debating which value is more sacred than the rest.