As I've pointed out here before, a lot of the versions of fooming that are discussed here seem to rest on assuming massive software optimization, not just hardware optimization. This runs into strongly believed theoretical comp sci limits such as the likelyhood that P != NP. These issues also come up in hardware design. It may be my own cognitive biases in trying to make something near my field feel useful, but it does seem like this sort of issue is not getting sufficient attention when discussing the probability of AI going foom.
I don't really see how P vs NP concerns are relevant, frankly.
Moreover, when trying to foom, an AI will probably want to directly improve the algorithms it is using on a regular basis. While not directly a P/NP issue, it is noteworthy that for some common procedures, such as finding gcds, and solving linear programming, we have what are in many ways close to optimal algorithms (the situation is more complicated for linear programing than that but the general point is correct). So the AI probably cannot focus primarily on modifying software because a lot of the basic things that the AI would want to optimize are already close to best possible even for both the average cases and the worst cases.
The core technical problem of machine intelligence is building an agent that correctly performs inductive inference. That is not a problem where we are particularly close to an optimal solution. Rather, solving it looks really, really difficult. Possibly machines will crack the problem and rapidly zoom up to optimal performance. However, that would represent tremendous improvement, not a lack of self-improvement.
So: your concerns about P and NP don't seem very relevant to me.
The overall upshot of this is that while it is possible that an AI with access to a quantum computer could plausibly foom with most of the focus on software improvement, comp sci issues suggest that an AI trying to foom on a classical system would need to engage in both hardware and software improvement.
Not really. We have more hardware than we know how to use. Some call it a "hardware overhang". Software improvement alone could take us far, at today's general tech level. However, of course, faster software improvement leads to faster hardware improvement. Any suggestion that we could conceivably have one without the other seems unwarranted.
The core technical problem of machine intelligence is building an agent that correctly performs inductive inference.
This seems to be highly non-obvious. Even if an AI already had access to a theory of everything, and could engage in near-optimal induction, it isn't at all clear that this helps much for practical purposes. The most obvious example is the example of cryptography as brought up by Roy. And many other things an AI might want to do seem to simply be computationally intensive by our current methods.
Say for example an AI wants to synthesize a ...
Link: johncarlosbaez.wordpress.com/2011/04/24/what-to-do/
His answer, as far as I can tell, seems to be that his Azimuth Project does trump the possibility of working directly on friendly AI or to support it indirectly by making and contributing money.
It seems that he and other people who understand all the arguments in favor of friendly AI and yet decide to ignore it, or disregard it as unfeasible, are rationalizing.
I myself took a different route, I was rather trying to prove to myself that the whole idea of AI going FOOM is somehow flawed rather than trying to come up with justifications for why it would be better to work on something else.
I still have some doubts though. Is it really enough to observe that the arguments in favor of AI going FOOM are logically valid? When should one disregard tiny probabilities of vast utilities and wait for empirical evidence? Yet I think that compared to the alternatives the arguments in favor of friendly AI are water-tight.
The problem why I and other people seem to be reluctant to accept that it is rational to support friendly AI research is that the consequences are unbearable. Robin Hanson recently described the problem:
I believe that people like me feel that to fully accept the importance of friendly AI research would deprive us of the things we value and need.
I feel that I wouldn't be able to justify what I value on the grounds of needing such things. It feels like that I could and should overcome everything that isn't either directly contributing to FAI research or that helps me to earn more money that I could contribute.
Some of us value and need things that consume a lot of time...that's the problem.