But the real problem we face is how to build or become a superintelligence that shares our values, and given that this seems very difficult, any progress that doesn't contribute to the solution but brings forward the date by which we must solve it (or be stuck with something very suboptimal even if it doesn't kill us), is bad.

Assume you manage to communicate this idea to the typical AI researcher. What do you expect him to do next? It's absurd to think that the typical researcher will quit his field and work on strategies for mitigating intelligence explosion or on foundations of value. You might be able to convince him to work on some topic within AI instead of another. However, while some topics seem more likely to advance AI capabilities than others, this is difficult to tell in advance. More perniciously, what the field rewards are demonstrations of impressive capabilities. Researchers who avoid directions that lead to such demos will end up with less prestigious jobs, i.e., jobs where they are less able to influence the top students of the next generation of researchers. This isn't what the typical AI researcher wants either. So, what's he to do?

Now that SI has been rebranded into MIRI, I've had "figure out new framing for AI risk talk, and concise answers to common questions" on my to-do list for several months, but haven't gotten to it yet. I would certainly appreciate your help with that, if you're willing.

Partly, I'm using "Effective Altruism and the End of the World" as a tool for testing out different framings of some of the key issues. I'll be giving the talk many times, and I'm iterating the talk between each presentation, and taking notes on which questions people ask most frequently, and which framings and explanations seem to get the best response.

Christiano has been testing different framings of things, too, mostly with the upper crust of cognitive ability. Maybe we should have a side-meeting about framing issues when you're in town for MIRI's September workshop?

This is similar in spirit to my complaint about the focus on intelligence explosion. Your framing though requires acceptance of consequentialist optimization view on decision making (so that "good enough" is not considered good enough if it's possible to do better), and of there being a significant difference between the better outcomes and the "default" outcomes.

For this, high risk of indifferent UFAI is a strong argument (when accepted), which in turn depends on orthogonality of values and optimization power. So while it's true that this particular argument doesn't have to hold for the problem to remain serious, it looks like one of the best available arguments for the seriousness of the problem. (It also has the virtue of describing relatively concrete features of the possible future.)

That said, I agree that there should exist a more abstract argument for ensuring the control of human value over the future, that doesn't depend on any particular scenario. It's harder to make this argument convincing, as it seems to depend on acceptance of some decision-theoretic/epistemic/metaethical background. Since it's not necessary to accept this argument (if you accept other arguments, such as intelligence explosion), for the same reasons as it's not necessary to accept plausibility of intelligence explosion, it seems to me that the current focus should be on making a good case for the strongest arguments, but once the low-hanging fruit on the other arguments is collected, it might be a good idea to develop this more general case as well.

But the real problem we face is how to build or become a superintelligence that shares our values, and given that this seems very difficult, any progress that doesn't contribute to the solution but brings forward the date by which we must solve it (or be stuck with something very suboptimal even if it doesn't kill us), is bad.

I still don't see how you can solve something that you virtually know nothing about. I think it will take real progress towards dangerous AGI before one can make it safe.

I just don't see how someone could have designed a provably secure online banking software before the advent of the Internet.

If someone wants to question the importance of facing this problem, they really instead need to argue that a superintelligence isn't possible (not even a modest one), or that the future will turn out to be close to the best possible just by everyone pushing forward their own research without any concern for the big picture, or perhaps that we really don't care very much about the far future and distant strangers and should pursue AI progress just for the immediate benefits.

False dilemma. For example, someone may think that superintelligences cannot arise quickly. Or they may think that human improvement to our own intelligent will make us as effective superintelligences well before we solve the AI problem (because it is just that tricky).

perhaps we really don't care very much about the far future and distant strangers and should pursue AI progress just for the immediate benefits.

You could ask Robin Hanson whether people really care about far future or it's just signalling.

but brings forward the date by which we must solve it

Does it really? I already explained that if someone makes an automated engineering tool, all users of that tool are at least as powerful as some (U)FAI based upon this engineering tool. Addition of independent will onto tank doesn't make it suddenly win the war against much larger force of tanks with no independent will.

You are rationalizing the position here. If you actually reason forwards, it is clear that creation of such tools may, instead, be the life-saver when someone who thought he solved morality unleashes some horror upon the world. (Or sometime, hardware gets so good that very simple evolution simulator like systems could self improve to point of super-intelligence by evolving, albeit that is very far off into the future)

Suppose I were to convince you of butterfly effect, and explain that you sneezing could kill people, months later. And suppose you couldn't think that non sneezing has same probability. You'd be trying real hard not to sneeze, for nothing, avoid the sudden bright lights (if you have sneeze reflex on bright lights), and so on.

The engineering super-intelligences don't share our values to such profound extent, as to not even share the desire to 'do something' in the real world. Even the engineering intelligence inside my own skull, as far as I can feel. I build designs in real life, because I have rent to pay, or because I am not sure enough it will work and don't trust the internal simulator that I use for design (i.e. imagining) [and that's because my hardware is very flawed]. This is also the case with all my friends whom are good engineers.

The issue here is that you conflate things into 'human level AI'. There's at least three distinct aspects to AI:

1: Engineering, and other problem solving. This is a creation of designs in abstract design space.

2: Will to do something in real world in real time.

3: Morality.

People here see first two as inseparable, while seeing third as unrelated.