Thanks. He says it much more better than I could. He speaks of importance of small problems.
When you are famous it is hard to work on small problems. This is what did Shannon in. After information theory, what do you do for an encore? The great scientists often make this error. They fail to continue to plant the little acorns from which the mighty oak trees grow. They try to get the big thing right off. And that isn't the way things go.
Speaking of which, one thing geniuses do is generate the right problems for themselves, not just choose from already formulated.
Science fiction is full of artificial minds, good and evil. It has minds improving themselves, and a plenty of Frankensteins of all kinds. It doesn't have things like 'a very efficient universal algorithm that given mathematical description of a system and constraints finds values for free parameters that meet constraints', because it is not a plot device. Fiction does not have wolfram alpha in 2010. It has Hal in 2000 . Fiction shuns merely useful in favor of interesting. I would be very surprised if the solution would be among the fictional set. The fictional set is as good place to look in as any, yes, but it is small. edit: On second thought, what I mean is that it would be very bad to be either inspired or 'de-spired' by fiction to any significant extent.
The fictional set is as good place to look in as any, yes, but it is small.
Yes: Humans think in stories but there are far far more concepts that do not make good story than do make it.
My friend, hearing me recount tales of LessWrong, recently asked me if I thought it was simply a coincidence that so many LessWrong rationality nerds cared so much about creating Friendly AI. "If Eliezer had simply been obsessed by saving the world from asteroids, would they all be focused on that?"
Obviously one possibility (the inside view) is simply that rationality compels you to focus on FAI. But if we take the outside view for a second, it does seem like FAI has a special attraction for armchair rationalists: it's the rare heroic act that can be accomplished without ever confronting reality.
After all, if you want to save the planet from an asteroid, you have to do a lot of work! You have to build stuff and test it and just generally solve a lot of gritty engineering problems. But if you want to save the planet from AI, you can conveniently do the whole thing without getting out of bed.
Indeed, as the Tool AI debate as shown, SIAI types have withdrawn from reality even further. There are a lot of AI researchers who spend a lot of time building models, analyzing data, and generally solving a lot of gritty engineering problems all day. But the SIAI view conveniently says this is all very dangerous and that one shouldn't even begin to try implementing anything like an AI until one has perfectly solved all of the theoretical problems first.
Obviously this isn't any sort of proof that working on FAI is irrational, but it does seem awfully suspicious that people who really like to spend their time thinking about ideas have managed to persuade themselves that they can save the entire species from certain doom just by thinking about ideas.