I think you're way too confident that the people you disagree with are obviously wrong, to the extent that I don't think we can usefully communicate. I'm tapping out of this discussion.
Have you observed my discussions elsewhere on this website, and came to the conclusion that I'm way too confident in that way in general, or are you referring only to this particular exchange?
This discussion seems like sort of a unique case. I wouldn't say I'm generally so confident in that respect, but I'm certainly extremely confident in this discussion, even to the point that I don't yet have a sufficiently detailed model of you to account for how you could possibly spend so much time on this website and still engage in the sort of communication that we...
Are there any essays anywhere that go in depth about scenarios where AIs become somewhat recursive/general in that they can write functioning code to solve diverse problems, but the AI reflection problem remains unsolved and thus limits the depth of recursion attainable by the AIs? Let's provisionally call such general but reflection-limited AIs semi-general AIs, or SGAIs. SGAIs might be of roughly smart-animal-level intelligence, e.g. have rudimentary communication/negotiation abilities and some level of ability to formulate narrowish plans of the sort that don't leave them susceptible to Pascalian self-destruction or wireheading or the like.
At first blush, this scenario strikes me as Bad; AIs could take over all computers connected to the internet, totally messing stuff up as their goals/subgoals mutate and adapt to circumvent wireheading selection pressures, without being able to reach general intelligence. AIs might or might not cooperate with humans in such a scenario. I imagine any detailed existing literature on this subject would focus on computer security and intelligent computer "viruses"; does such literature exist, anywhere?
I have various questions about this scenario, including: