The simplest reason to care about Friendly AI is that we are going to be coexisting with AI, and so we should want it to be something we can live with. I don't see that anything important would be lost by strongly foregrounding the second set of assumptions, and treating the first set of possibilities just as possibilities, rather than as the working hypothesis about reality.
That sounds terribly open-minded and inclusive. However, when it comes to the first three bullets:
- A single AI will take over the world vs There will always be multiple centers of power
The probability assigned to these scenarios completely and fundamentally changes all expectations of experience with the GAIs in question and all the ethical and practical considerations that are relevant are entirely different. For most intents and purposes you are advocating just talking about an entirely different thing than the one labelled FAI. Conversation which is conditional on one is completely incompatible with conversation conditional on the other (or, if made to be compatible it is stilted).
- A future galactic civilization depends on 21st-century Earth vs What's at stake is, at most, the future centuries of a solar-system civilization
Conversation is at least possible when such premises are substituted. The main thing that is being changed is the priority that certain outcomes are assigned when calculating expected utilities. ie. Success vs failure on FAI production becomes an order of magnitude or two less important so probability of success to make a given investment worthwhile has to be larger.
- 10n-year lifespans are at stake, n greater than or equal to 3 vs No assumption that individual humans can survive even for hundreds of years, or that they would want to
When it comes to this one the line of thought that advocates the latter rather than the former seems inane and ridiculous but assuming the kind of thinking that lead to it doesn't creep in to the rest of the discussion it wouldn't be a huge problem.
- We might be living in a simulation
- Acausal deal-making
- Multiverse theory
These ones don't seem to interfere with Friendly AI discussion much at all now. They can be---and are---fairly isolated to discussions that are about their respective subjects. Nothing need be changed in order to have discussions about FAI that don't talk about those issues.
I don't see that anything important would be lost by strongly foregrounding the second set of assumptions
Integrity, and the entire premise of the website. "Strongly foregrounding assumptions" that are ridiculous just isn't what we do here. In particular I refer to trying to aggressively privilege the hypothesis that:
Followed closely by the "We're going to co-exist with AIs vaguely as equals" that is being implied between the lines an in particular by rejecting discussion premised on "A single AI will take over the world".
I strongly reject the suggestion that we should pretend things are "given far too much credence" just because they aren't "down to earth". The "important thing that would be lost" is basically any point to having the conversation here at all. If you want to have "Sober" conversations at the expense of sane ones then have them elsewhere.
On these three issues - I'll call them post-singularity concentration of power, size and duration of post-singularity civilization, and length of post-singularity lifespan - my preference is to make the low estimate the default, but to regard the high estimates as defensible.
In each case, there are arguments (increasing returns on self-enhancement, no sign of alien civilizations, fun theory / basic physics) in favor of the high estimate (one AI rules the world, the winner on Earth gets this Hubble volume as prize, no psychological or physical limits on li...
The project of Friendly AI would benefit from being approached in a much more down-to-earth way. Discourse about the subject seems to be dominated by a set of possibilities which are given far too much credence:
Add up all of that, and you have a great recipe for enjoyable irrelevance. Negate every single one of those ideas, and you have an alternative set of working assumptions that are still consistent with the idea that Friendly AI matters, and which are much more suited to practical success:
The simplest reason to care about Friendly AI is that we are going to be coexisting with AI, and so we should want it to be something we can live with. I don't see that anything important would be lost by strongly foregrounding the second set of assumptions, and treating the first set of possibilities just as possibilities, rather than as the working hypothesis about reality.
[Earlier posts on related themes: practical FAI, FAI without "outsourcing".]