I think you're underselling the developmental power of a culture. Bits of your brain literally don't grow properly if you're not raised in a human culture. Ignore a baby at the wrong points in its development and it'll fail to ever be able to learn any language, feel certain emotions or comprehend some social constraints. Etc.
That is, the hardware grows to meet the software and data, because (as usual) the data/software/hardware divides in the brain are very fuzzy indeed.
(This suggests Kurzweil was plausibly approximately correct about the genome having the information needed to make the brain of a fresh-out-the-womb newborn, but that the attention-catching claim he was implicitly making of emulating an interesting, adult-quality brain based on the amount of information in a genome is rather more questionable.)
(And, of course, it brings to mind all manner of horribly unethical experiments to work out the minimum quantity of culture needed to stimulate the brain to grow right, or what the achievable dimensions of "right" are. You just can't get the funding for the really mad science these days.)
Of course, the baby's brain goes actively looking for cultural data. I will always treasure the memory of my daughter meowing back at the cat and trying to have a conversation with it and learn its language. Made more fun by the fact that cats only meow like that in the first place as a way of getting humans to do things.
I think you're underselling the developmental power of a culture. Bits of your brain literally don't grow properly if you're not raised in a human culture. Ignore a baby at the wrong points in its development and it'll fail to ever be able to learn any language, feel certain emotions or comprehend some social constraints.
Not denying this at all. Just pointing out that the brain makes astonishingly good use of very noisy and arbitrary input when it does get exposed to other language-using humans, compared to what you'd expect any sort of machine learnin...
For no reason in particular I'm wondering about the size of the smallest program that would constitute a starting point of a recursively self-improving AI.
The analysis of FOOM as a self-amplifying process would seem to indicate that in principle one could get it started from a relatively modest starting point -- perhaps just a few bytes of the right code could begin the process. Or could it? I wonder whether any other considerations give tighter lower-bounds.
One consideration is that FOOM hasn't already happened -- at least not here on Earth. If the smallest FOOM seed were very small (like a few hundred bytes) then we would expect evolution to have already bumped into it at some point. Although evolution is under no specific pressure to produce a FOOM, it has probably produced over the last few billion years all the interesting computations up to some minor level of complexity, and if there were a FOOM seed among those then we would see the results about us.
Then there is the more speculative analysis of what minimal expertise the algorithm constituting the FOOM seed would actually need.
Then there is the fact that any algorithm that naively enumerates some space of algorithms qualifies in some sense as a FOOM seed as it will eventually hit on some recursively self-improving AI. But that could take gigayears so is really not FOOM in the usual sense.
I wonder also whether the fact that mainstream AI hasn't yet produced FOOM could lower-bound the complexity of doing so.
Note that here I'm referring to recursively self-improving AI in general -- I'd be interested if the answers to these questions change substantially for the special case of friendly AIs.
Anyway, just idle thoughts, do add yours.