Scenario analysis: semi-general AIs

Will_Newsome

Are there any essays anywhere that go in depth about scenarios where AIs become somewhat recursive/general in that they can write functioning code to solve diverse problems, but the AI reflection problem remains unsolved and thus limits the depth of recursion attainable by the AIs? Let's provisionally call such general but reflection-limited AIs semi-general AIs, or SGAIs. SGAIs might be of roughly smart-animal-level intelligence, e.g. have rudimentary communication/negotiation abilities and some level of ability to formulate narrowish plans of the sort that don't leave them susceptible to Pascalian self-destruction or wireheading or the like.

At first blush, this scenario strikes me as Bad; AIs could take over all computers connected to the internet, totally messing stuff up as their goals/subgoals mutate and adapt to circumvent wireheading selection pressures, without being able to reach general intelligence. AIs might or might not cooperate with humans in such a scenario. I imagine any detailed existing literature on this subject would focus on computer security and intelligent computer "viruses"; does such literature exist, anywhere?

I have various questions about this scenario, including:

How quickly should one expect temetic selective sweeps to reach ~99% fixation?
To what extent should SGAIs be expected to cooperate with humans in such a scenario? Would SGAIs be able to make plans that involve exchange of currency, even if they don't understand what currency is or how exactly it works? What do humans have to offer SGAIs?
How confident can we be that SGAIs will or won't have enough oomph to FOOM once they saturate and optimize/corrupt all existing computing hardware?
Assuming such a scenario doesn't immediately lead to a FOOM scenario, how bad is it? To what extent is its badness contingent on the capability/willingness of SGAIs to play nice with humans?

Those are the questions that immediately spring to mind, but I'd like to see who else has thought about this and what they've already considered before I cover too much ground.

My intuition says that thinking about SGAIs in terms of population genetics and microeconomics will somewhat counteract automatic tendencies to imagine cool stories rather than engage in dispassionate analysis. I'd like other suggestions for how to achieve that goal.

I'm confused that I don't see people talking about this scenario very much; why is that? Why isn't it the default expected scenario among futurologists? Or have I just not paid close enough attention? Is there already a name for this class of AIs? Is the name better than "semi-general AIs"?

Thanks for any suggestions/thoughts, and my apologies if this has already been discussed at length on LessWrong.

I have various questions about this scenario, including:

How quickly should one expect temetic selective sweeps to reach ~99% fixation?
To what extent should SGAIs be expected to cooperate with humans in such a scenario? Would SGAIs be able to make plans that involve exchange of currency, even if they don't understand what currency is or how exactly it works? What do humans have to offer SGAIs?
How confident can we be that SGAIs will or won't have enough oomph to FOOM once they saturate and optimize/corrupt all existing computing hardware?
Assuming such a scenario doesn't immediately lead to a FOOM scenario, how bad is it? To what extent is its badness contingent on the capability/willingness of SGAIs to play nice with humans?

Those are the questions that immediately spring to mind, but I'd like to see who else has thought about this and what they've already considered before I cover too much ground.

Thanks for any suggestions/thoughts, and my apologies if this has already been discussed at length on LessWrong.

I've never even seen a shred of evidence suggesting I should believe there's a deity prepared to punish you if you kill people or do something immoral (or anything else for that matter), so it's on the exact same level as worrying about the possibility that snapping your fingers more than 300 times throughout the course of your lifetime may lead to an afterlife of eternal torture.

There's absolutely no reason why one should consider it more likely that there's a deity waiting to judge you after your death than really anything else at all. Maybe playing disc golf even once is bound to lead to an afterlife of hell. Ever played? If not, you may still have a chance!

If you consider it from a sound epistemic point of view, it becomes obvious that in our current state of knowledge, Pascal's Wager is equally apt to everything, and thus utterly useless or even simply meaningless. It's only a particular vulnerability in human brain hardware that makes it seem any different.

A lot of people still believe the whole god hypothesis, and many more have believed it over the course of human history, but that doesn't mean it's any more useful than an equally ridiculous hypothesis that I could invent right now: Learning to type with proper mechanics may help you in the short term (your mortal life), but beware, for God may not approve, and He doesn't mess around with his revenge.

We as humans are designed to think in groups, and it takes a special sort of introspection to pull yourself out of the mess and realize that the system is broken, and that a large percentage of the output is not to be trusted. That much you should know by virtue of having spent more than an hour reading Less Wrong, but perhaps you don't see this particular application.

I understand the pull of Pascal's Wager. I've felt it, and I still do. It feels like there's some special evidence for God's existence, and like we should privilege this hypothesis over another, but that's only because of how many people have believed in religion in the past, or rather how many times you've heard it espoused in an approving way. We're simply wired that way.

AI is thinking more straight than you. It doesn't need to believe in God. It can assign some uncertainty. The only good argument for non-existence of God is Occam's razor, and Occam's razor doesn't say the complex explanation is impossible, merely unlikely (and shouldn't be privileged).

Imagine the AI that knows precisely how much more complex are the rules that lead to universe full of Gods who create sims of our universe, than rules for our universe. This AI will tell you exactly how likely it is that it is sitting in a simulation, instead of making some... (read more)

1Will_Newsome14y

Other peoples' beliefs are evidence. Many people believe in God. No one believes that disc golf causes eternal torture. The two hypotheses should not be assigned equal probability. [...] So you do not believe that others' beliefs are evidence?