This is an attempt (against my preference) to defend SIAI's reasoning.

Let's characterize the predictions of the future into two broad groups: 1. business as usual, or steady-state. 2. aware of various alarmingly exponential trends broadly summarized as "Moore's law". Let's subdivide the second category into two broad groups: 1. attempting to take advantage of the trends in roughly a (kin-) selfish manner 2. attempting to behave extremely unselfishly.

If you study how the world works, the lack of steady-state-ness is everywhere. We cannot use fossil fuels or arable land indefinitely at the current rates. "Business as usual" depends crucially on progress in order to continue! We're investing heavily in medical research, and there's no reason to expect that to stop, Self-replicating molecular-scale entities already exist, and there is every reason to expect that we will understand how to build them better than we currently do.

Supposing that the above paragraph has convinced the reader that the lack of steady--state-ness is fairly obvious, and given the reader's knowledge of human nature, how many people do you expect would be trying to behave in extremely unselfish manner?

Your post seems to expect that if various luminaries believed that incomprehensibly-sophisticated computing engines and dangerous self-replicating atomic-scale entities were likely in our future, then they would behave extremely unselfishly - is that reasonable? Supposing that almost everyone was aware of this probable future, how would they take advantage of it in a (kin-)selfish manner as best they can? I think the hypothesized world would look very much like this one.