Consider the choice: cause the world of today to exist, or the world of tomorrow.
What actual choice is informed by considering this decision problem, what motivates posing it? (I don't know, which is why I originally ruled that out as a possible interpretation, and suggested a less straightforward interpretation that involved an actual choice.)
We don't choose between the world of today or the world of tomorrow, but we also don't choose between performing better or worse on a test.
(I'm not appealing to the impossibility-in-practice of choosing today vs. tomorrow, so the distinction between solvable and unsolvable decision problems doesn't seem relevant. The difficulty is that I'm not seeing the motivation for the problem.)
Edit: New hypothesis (which seems to fit with your reply to Qiaochu): You are talking about "progress" in the post, so the implicit choice in question might be between keeping the world closer to the past and pushing it faster closer to the future, so you are asking which is better and how much better. This makes sense.
What actual choice is informed by considering this decision problem, what motivates posing it? (I don't know, which is why I originally ruled that out as a possible interpretation, and suggested a less straightforward interpretation that involved an actual choice.)
Consider economic development, population growth, technological development, basic science progress. My intuition is that all of these things are good, but that seems at odds with this analysis. If the new analysis is right, that would cause me to reconsider those valuations, which are relevan...
Over the last few months I've started blogging about effective altruism more broadly, rather than focusing on AI risk. I'm still focusing on abstract considerations and methodological issues, but I hope it is of interest to others here. Going forward I intend to cross-post more often to LW, but I thought I would post the backlog here anyway. With luck, I'll also have the opportunity to post more than bi-weekly.
I welcome thoughts, criticisms, etc.