I think you're underselling the developmental power of a culture. Bits of your brain literally don't grow properly if you're not raised in a human culture. Ignore a baby at the wrong points in its development and it'll fail to ever be able to learn any language, feel certain emotions or comprehend some social constraints.
Not denying this at all. Just pointing out that the brain makes astonishingly good use of very noisy and arbitrary input when it does get exposed to other language-using humans, compared to what you'd expect any sort of machine learning AI to be capable of. I'm a lot more impressed at a thing made of atoms getting to be complex enough to be able to start the learning process than the further input it needs to actually learn the surrounding culture.
Think about it this way: Which is more impressive, designing and building a robot that can perceive the world and move around it and learn things as well as a human growing from infant to adulthood, or pointing things to the physically finished but still-learning robot and repeating their names, and doing the rest of the regular teaching about stuff thing people already do with children?
(For anyone offended at the implied valuation, since Parenting Human Children Is The Most Important Thing, imagine that the robot looks like a big metal spider and therefore doesn't count as a Parented Child.)
My basic idea here is that the newborn baby crawling about is already a lot more analogous to an AI well in the way of going FOOM than a bunch of scattered clever pattern recognition algorithms and symbol representation models that just need the overall software architecture design to tie them together, since the things that stop humans from going FOOM might be a lot more related to physiological shortcomings than the lack of extremely clever further design. The baby has moved from being formed from the initial hard design information that went in it into discovering the new information it needs to grow from its surroundings. I'd be rather worried about an AI that reaches a similar stage.
My basic idea here is that the newborn baby crawling about is already a lot more analogous to an AI well in the way of going FOOM than a bunch of scattered clever pattern recognition algorithms and symbol representation models that just need the overall software architecture design to tie them together
I'll credit that. A baby is a machine for going FOOM.
(Specifically, I'd guess, because so much has to be left out to produce a size of offspring that can be born without killing the mother too often. Hence the appalling, but really quite typical of evolution, hack of having the human memepool be essential to the organism expressed by the genes growing right.)
For no reason in particular I'm wondering about the size of the smallest program that would constitute a starting point of a recursively self-improving AI.
The analysis of FOOM as a self-amplifying process would seem to indicate that in principle one could get it started from a relatively modest starting point -- perhaps just a few bytes of the right code could begin the process. Or could it? I wonder whether any other considerations give tighter lower-bounds.
One consideration is that FOOM hasn't already happened -- at least not here on Earth. If the smallest FOOM seed were very small (like a few hundred bytes) then we would expect evolution to have already bumped into it at some point. Although evolution is under no specific pressure to produce a FOOM, it has probably produced over the last few billion years all the interesting computations up to some minor level of complexity, and if there were a FOOM seed among those then we would see the results about us.
Then there is the more speculative analysis of what minimal expertise the algorithm constituting the FOOM seed would actually need.
Then there is the fact that any algorithm that naively enumerates some space of algorithms qualifies in some sense as a FOOM seed as it will eventually hit on some recursively self-improving AI. But that could take gigayears so is really not FOOM in the usual sense.
I wonder also whether the fact that mainstream AI hasn't yet produced FOOM could lower-bound the complexity of doing so.
Note that here I'm referring to recursively self-improving AI in general -- I'd be interested if the answers to these questions change substantially for the special case of friendly AIs.
Anyway, just idle thoughts, do add yours.