Worst of all you build stuff from the dubious concepts like that monolithic "intelligence".
The reason for that is that the AIs that are worrying are those of human-like levels of ability. And humans have shown skill in becoming intelligent in many different domains and the ability to build machine intelligence in the domains we have little skill in. So whatever its design, a AGI (artificial general intelligence) will have a broad pallet of abilities, and probably the ability to acquire others - hence the details of its design are less important than meta considerations. This is not the case for non-AGI AIs.
Or the orthogonality thesis. You think - are the goals of that monolithic intelligence arbitrary?
I think "how to convince philosophers that high intelligence will not automatically imply certain goals" - ie that they are being incorrectly meta.
Meanwhile if you try to build bottom up or at least from the concepts with known bottom up definitions, well...
Moor's law is a better way of predicting the future than knowing the exact details of the current research into microprocessors. Since we don't have any idea how the first AGI will be built (assuming it can be built), why bother focusing down on the current details when we're pretty certain they won't be relevant?
I think "how to convince philosophers that high intelligence will not automatically imply certain goals"
Contradicts:
Since we don't have any idea how the first AGI will be built
If you don't have any idea how AGI will be built, how can you be so confident about the distribution of its goals?
It's just occurred to me that, giving all the cheerful risk stuff I work with, one of the most optimistic things people could say to me would be:
"You've wasted your life. Nothing of what you've done is relevant or useful."
That would make me very happy. Of course, that only works if it's credible.