In Friendly Teams Robin Hanson talks about the guy who tried to get his company to undergo recursive self-improvement and how he was a really smart fellow who saw a lot of things coming. Robin Hanson further argues that key insights are not enough but that it takes many small insights that are the result of a whole society of agents. Robin further asks what it is that makes the singleton AI scenario more reasonable if does not work out for groups of humans, not even remotely. Well, I can see that people would now say that an AI can directly improve its own improvement algorithm. I suppose the actual question that Robin asks is how the AI will reach that point in the first place. How is it going to acquire the capabilities that are necessary to improve its capabilities indefinitely.

In Engelbart As UberTool? Robin Hanson talks about a dude who actually tried to apply recursive self-improvement to his company. He is till trying (wow!). It seems humans, even groups of humans, are not capable of fast recursive self-improvement. That they didn't take over the world might be partly due to strong competition from other companies that are constantly trying the same. What is it that is missing that doesn't allow one of them to prevail? Robin Hanson further asks what would have been a reasonable probability estimate to assign to the possibility of a company taking over the world at that time. I have no idea how I could possible assign a number to that. I would just have said that it is unlikely enough to be ignored. Or that there is not enough data to make a reasonable guess either way. I don't have the resources to take every idea seriously and assign a probability estimate to it. Some things get just discounted by my intuitive judgment.