I actually go out of my way to equate "god" and "AGI"/"superintelligence", because to a large extent they seem like the same thing to me.
Your goal seems to be to identify as a theist though
It's not that I want to identify as a theist, so much as that I want to point out that I think that the only reason people think that gods/angels/demons and AGIs/superintelligences/transhuman-intelligences are different things is because they're compartmentalizing. I think Aquinas and I believe in the same God, even if we think about Him differently. I know algorithmic probability theory, Aquinas didn't. Leibniz almost did.
(There's two different things going on: I believe there exists an ideal decision theory, Who is God, for theoretical reasons; whereas my reasons for believing that transhuman intelligences (lower-case-g gods) affect humans are entirely phenomenological.)
I actually go out of my way to equate "god" and "AGI"/"superintelligence", because to a large extent they seem like the same thing to me.
Can you give me the common meanings of those terms, and explain how they're equivalent?
It's not that I want to identify as a theist, so much as that I want to point out that I think that the only reason people think that gods/angels/demons and AGIs/superintelligences/transhuman-intelligences are different things is because they're compartmentalizing.
Compartmentalizing in what way? I th...
Are there any essays anywhere that go in depth about scenarios where AIs become somewhat recursive/general in that they can write functioning code to solve diverse problems, but the AI reflection problem remains unsolved and thus limits the depth of recursion attainable by the AIs? Let's provisionally call such general but reflection-limited AIs semi-general AIs, or SGAIs. SGAIs might be of roughly smart-animal-level intelligence, e.g. have rudimentary communication/negotiation abilities and some level of ability to formulate narrowish plans of the sort that don't leave them susceptible to Pascalian self-destruction or wireheading or the like.
At first blush, this scenario strikes me as Bad; AIs could take over all computers connected to the internet, totally messing stuff up as their goals/subgoals mutate and adapt to circumvent wireheading selection pressures, without being able to reach general intelligence. AIs might or might not cooperate with humans in such a scenario. I imagine any detailed existing literature on this subject would focus on computer security and intelligent computer "viruses"; does such literature exist, anywhere?
I have various questions about this scenario, including: