In the last five decades humans have created algorithms for solving many problems that had previously been intractable, and given orders of magnitude improvement on others. Many of these have come from math/compsci innovation that was not particularly hardware-limited, i.e. if you had the same (or a larger/smarter-on-average/better-organized) research community but with frozen primitive hardware many of the insights would have been found.
Yes. I agree strongly with this. One major thing we've found in the last few years is just that P turns out to be large and a lot of problems have turned out to be in there that were not obviously so. If one asked people in the early 1970s whether they would expect primality testing to be in P they would probably say no. Moreover, some practical problems have simply had their implementations improved a lot even as the space and time of the algorithms remain in the same complexity classes.
There are also problems where we are clearly far from the reachable frontier (whether that is near-optimal performance, or just the best that can be done given resource constraints).
Can you expand on this? I'm not sure I follow.
So long as enough domains have room to grow, they can translate into strategic advantage even if others are stable
Sure. But that doesn't say much about how fast that growth will occur. The standard hard-take off narratives have the AI becoming functionally in control of its light cone in a matter of hours or at most weeks. I agree that there is likely a lot of room for improvement in cognitive capability but the issue in this context is whether it is likely for that sort of improvement to occur quickly.
linear gains in chess performance translate into an exponential drop-off in the number of potential human challengers, etc.
I agree with your other examples. And it is a valid point. I don't think that a strong form P !=NP makes fooming impossible, just that it makes it much less likely. The chess example however has an issue that needs to be nitpicked. As I understand it, this isn't really about linear gains in chess translating into exponential drop off but rather an artifact of the Elo system which sort of requires that linear increase corresponds to quick improvement.
The standard hard-take off narratives have the AI becoming functionally in control of its light cone in a matter of hours or at most weeks.
The human field of AI is about half a million hours old, computer elements can operate at a million times human speed (given enough parallel elements). To the extent that many of the important discoveries were not limited by chip speeds but by the pace of CS, math, and AI researchers' thinking (with most of the work done by some thousands of people who spent much of that time eating, sleeping, goofing off, getting up...
Link: johncarlosbaez.wordpress.com/2011/04/24/what-to-do/
His answer, as far as I can tell, seems to be that his Azimuth Project does trump the possibility of working directly on friendly AI or to support it indirectly by making and contributing money.
It seems that he and other people who understand all the arguments in favor of friendly AI and yet decide to ignore it, or disregard it as unfeasible, are rationalizing.
I myself took a different route, I was rather trying to prove to myself that the whole idea of AI going FOOM is somehow flawed rather than trying to come up with justifications for why it would be better to work on something else.
I still have some doubts though. Is it really enough to observe that the arguments in favor of AI going FOOM are logically valid? When should one disregard tiny probabilities of vast utilities and wait for empirical evidence? Yet I think that compared to the alternatives the arguments in favor of friendly AI are water-tight.
The problem why I and other people seem to be reluctant to accept that it is rational to support friendly AI research is that the consequences are unbearable. Robin Hanson recently described the problem:
I believe that people like me feel that to fully accept the importance of friendly AI research would deprive us of the things we value and need.
I feel that I wouldn't be able to justify what I value on the grounds of needing such things. It feels like that I could and should overcome everything that isn't either directly contributing to FAI research or that helps me to earn more money that I could contribute.
Some of us value and need things that consume a lot of time...that's the problem.