The core technical problem of machine intelligence is building an agent that correctly performs inductive inference.
This seems to be highly non-obvious. Even if an AI already had access to a theory of everything, and could engage in near-optimal induction, it isn't at all clear that this helps much for practical purposes. The most obvious example is the example of cryptography as brought up by Roy. And many other things an AI might want to do seem to simply be computationally intensive by our current methods.
Say for example an AI wants to synthesize a virus to modify some species out there or some members of a species (like say some of those pesky humans). Well, that requires at minimum being able to do protein folding in advance.Similarly, if the AI decides it needs to use its memory more efficiently, that leads to difficult computational tasks.
It may be that we're focusing on different issues. It seems that you are focusing on "how difficult is inductive inference from a computational perspective?" which is relevant for what sorts of AI we can practically build. That's not connected to once we have an AI what it will do.
We have more hardware than we know how to use. Some call it a "hardware overhang". Software improvement alone could take us far, at today's general tech level.
This seems irrelevant. Hardware overhang is due to the fact that the vast majority of personal clock cycles aren't being used. The vast majority of that hardware won't be accessible to our AGI unless something has already gone drastically wrong. I agree that an AGI that can get control of a large fraction of the internet accessible computers will likely quickly get very powerful completely separately from computational complexity questions.
It may be that we are imagining different situations. My intent was primarily to address foom narratives that put much more emphasis on software than improvements in hardware. Moreover, to make the point that without increasing software efficiency, one could easily have diminishing marginal returns in attempts to improve hardware.
The vast majority of that hardware won't be accessible to our AGI unless something has already gone drastically wrong. I agree that an AGI that can get control of a large fraction of the internet accessible computers will likely quickly get very powerful completely separately from computational complexity questions.
What's the problem? Google got quite a few people to contribute to Google Compute.
You think that a machine intelligence would be unsuccessful at coming up with better bait for this? Or that attempts to use user cycles are necessarily evil?
Link: johncarlosbaez.wordpress.com/2011/04/24/what-to-do/
His answer, as far as I can tell, seems to be that his Azimuth Project does trump the possibility of working directly on friendly AI or to support it indirectly by making and contributing money.
It seems that he and other people who understand all the arguments in favor of friendly AI and yet decide to ignore it, or disregard it as unfeasible, are rationalizing.
I myself took a different route, I was rather trying to prove to myself that the whole idea of AI going FOOM is somehow flawed rather than trying to come up with justifications for why it would be better to work on something else.
I still have some doubts though. Is it really enough to observe that the arguments in favor of AI going FOOM are logically valid? When should one disregard tiny probabilities of vast utilities and wait for empirical evidence? Yet I think that compared to the alternatives the arguments in favor of friendly AI are water-tight.
The problem why I and other people seem to be reluctant to accept that it is rational to support friendly AI research is that the consequences are unbearable. Robin Hanson recently described the problem:
I believe that people like me feel that to fully accept the importance of friendly AI research would deprive us of the things we value and need.
I feel that I wouldn't be able to justify what I value on the grounds of needing such things. It feels like that I could and should overcome everything that isn't either directly contributing to FAI research or that helps me to earn more money that I could contribute.
Some of us value and need things that consume a lot of time...that's the problem.