This post is mostly concerned with a superintelligent AI performing recursive self-improvement, this analysis is done to help make sense of the take off speed of such an operation.
Plausibility and Limits
Before considering upper limits, it may be worth considering whether general superintelligence is possible at all. It has been suggested that the idea of recursive self-improvement is similar to the infamous concept of a "perpetual motion machine". We know that a p.m.m. is impossible because it violates thermodynamics. Is there an analogous proof or argument that shows recursive self-improvement impossible? Some good places to start looking for hard limits on superintelligence are mathematics, computability and physics. It's also useful to think about... (read 1395 more words →)
Great post. Based on these ideas I've created a PR that adds a "kill switch" to Auto-GPT. I've written up some stuff about potential llm based worms might work here: https://gist.github.com/rain-1/cc67caa8873ee38098aefb0f35f46014
I did a strange experiment once, I implemented a markov predictor like this and used it to predict the next bit of a sequence. Then I extended the sequence with the negation of that bit an increased the length of the markov chain. The result was a very random unpredictable looking sequence, but of course it was completely deterministic. I wonder if this sequence has a name?
I would recommend Lawvere's book Sets for mathematics to anybody who is interested in learning some mathematics. It's a beginner book but it covers interesting category theoretical stuff you wont find in any other beginner book.