The problem I see here is that if you literally care only about the "worst life at any given moment", then situations "seven billion extremely happy people, one mildly unhappy person" and "seven billion mildly hapy people, one mildly unhappy person" are equivalent, because the worst one is in the same situation. Which means, if you had a magical button that could convert the latter situation to the former, you wouldn't bother pressing it, because you wouldn't see a point in doing so. Is that what you really believe?
I care about wellbeing, but only second to pain. I'd definitely press a button maximizing happiness if it didn't cause individual unhappiness worse than it cured. Doesn't that make sense?
On second thought, two equally happy people > one and likewise with unhappiness. Maybe it doesn't make sense after all. Or it's a mix of a moral guideline (NU) and personal preference?
I am trying to decide how to allocate my charitable donations between GiveWell's top charities and MIRI, and I need a probability estimate to make an informed decision. Could you help me?
Background on my moral system: I place a greater value on reducing high doses of suffering of conscious entities than merely preventing death. An unexpected, instant, painless death is unfortunate, but I would prefer it to a painful and chronic condition.
Given my beliefs, it follows logically that I would pay a relatively large amount to save a conscious entity from prolonged torture.
The possibility of an AI torturing many conscious entities has been mentioned1 on this site, and I assume that funding MIRI will help reduce its probability. But what is its current probability?
Obviously a difficult question, but it seems to me that I need an estimate and there is no way around it. I don't even know where to start...suggestions?
1 http://lesswrong.com/lw/1pz/the_ai_in_a_box_boxes_you/