Do you disagree that it's worth avoiding conflating losing and extinction?
I agree that it's worth avoiding the conflation. There are losing scenarios that don't involve extinction - most highly concentrated in the realm of not-quite-friendly-AI. (And, naturally, excluding from the class under consideration all "uFAIs" that are insufficiently capable or insufficiently motivated to do anything much at all.)
For the purpose of this utterance it happens that I would make both the claims:
I expect I have different predictions regarding likely uFAI results and in particular different expectations regarding how acausal costs will be calculated.
I expect I have different predictions regarding likely uFAI results and in particular different expectations regarding how acausal costs will be calculated.
FWIW acausal considerations don't figure as much into my calculations as straightforward "wow I hope there isn't an AI already chilling 'round these parts who will get pissed at me if I try to kill all the humans" considerations do.
It really seems to me that you'd have to be very, very confident that there were no gods around to punish you for you to think it was worth it to turn the humans...
Are there any essays anywhere that go in depth about scenarios where AIs become somewhat recursive/general in that they can write functioning code to solve diverse problems, but the AI reflection problem remains unsolved and thus limits the depth of recursion attainable by the AIs? Let's provisionally call such general but reflection-limited AIs semi-general AIs, or SGAIs. SGAIs might be of roughly smart-animal-level intelligence, e.g. have rudimentary communication/negotiation abilities and some level of ability to formulate narrowish plans of the sort that don't leave them susceptible to Pascalian self-destruction or wireheading or the like.
At first blush, this scenario strikes me as Bad; AIs could take over all computers connected to the internet, totally messing stuff up as their goals/subgoals mutate and adapt to circumvent wireheading selection pressures, without being able to reach general intelligence. AIs might or might not cooperate with humans in such a scenario. I imagine any detailed existing literature on this subject would focus on computer security and intelligent computer "viruses"; does such literature exist, anywhere?
I have various questions about this scenario, including: