It's conceivable that a large organization could have SIAI folks heading its AI project, but this doesn't seem likely.
I'll give 50% odds of SIAI being subsumed by a larger organization sometime in the next 10 years.
I'm far from being an expert and I don't mean to be derogatory either but I still have this feeling that it is an absurd thought that the SIAI could even approach a narrow AI. I wish someone could write some kind of introductory post that would show that the SIAI can indeed accomplish something like a mathematical definition of friendliness given a few people equipped with laptops. A paragraph in a post by multifoliaterose (a mathematician) does express what I feel:
I don't even see how one would start to research the problem of getting a hypothetical AGI to recognize humans as distinguished beings. [...] Even if one does have a model for a hypothetical AGI, writing code conducive to it recognizing humans as distinguished beings seems like an intractable task.
Nobody is able to even define mathematically what a human being is supposed to be, what about friendliness then? Sounds like magic to me. I hope such thoughts are due to my ignorance of math and general AI research.
ETA I'm talking about something like IBM Watson by 'narrow AI', not some abstract model of intelligence like AIXI which is about as useless regarding real-world AGI as an universal turing machine when it comes to manufacturing efficient computers (as far as I can tell).
I don't even see how one would start to research the problem of getting a hypothetical AGI to recognize humans as distinguished beings. [...] Even if one does have a model for a hypothetical AGI, writing code conducive to it recognizing humans as distinguished beings seems like an intractable task.
IANAAIR, but I doubt anyone at the SIAI would try to program a way of identifying humans directly into an AI. Instead, it would learn what a human is from observations, using its general learning algorithms. Obviously there are many possible ambiguous cases, but we should try to prevent the AI from needing to make these decisions until it has figured out what goals we want it to have.
I wish someone could write some kind of introductory post that would show that the SIAI can indeed accomplish something like a mathematical definition of friendliness
It's not known to be probable, but it's not known to be impossible either, and the only way to mine this possibility is to make an actual effort.
An existing machine - in this "Volvo S60 - Pedestrian Detection" advert - seems to identify humans, and behave nicely towards them.
Mobileye's pedestrian detection technology is based on the use of mono cameras only, using advanced pattern recognition and classifiers with image processing and optic flow analysis. Both static and moving pedestrians can be detected to a range of around 30m using VGA resolution imagers. As higher resolution imagers become available range will scale with imager resolution, making detection ranges of up to 60m feasible.
That's nice but probably not what multifoliaterose had in mind. The company has 200 employees and was founded in 1999. I'd be impressed if the SIAI managed to come up with anything nearly as sophisticated. I suppose such companies would call CEV better science fiction. Or do you think they would hire Yudkowsky, if he wanted to get hired? I'm not sure if he is skilled enough to work for such a company. Have you seen any proof of his math or programming skills that would allow you to conclude that he would be able to come up with such a pedestrian detection software, let alone friendliness? (ETA I believe such questions are justified, after all it is important to assess the ability of the SIAI.)
I am a programmer, and I for one, do not see a very strong connection between the potential for building an AGI and programming ability. An AGI isn't going to come about because you made a really sweet sorting algorithm, it's going to come about because you had a key insight about what thought is (or something along those lines). 1337 programming skillz probably doesn't help a lot with that.
AGI requires John von Neumann or Alan Turing or the like. Any of them would have decent programming expertise today.
The AGI requires something, that would also result in acquisition of familiarity with the tool-set of mankind, including the actual use of computers for reasoning, which requires you to be able to program. It is enough that the programming expertise might be useful, for the upcoming AGI insight maker to become a good programmer.
I am a programmer, and I for one, do not see a very strong connection between the potential for building an AGI and programming ability.
Do you think that intelligence is going to be quite simple with hindsight? Something like Einstein's mass–energy equivalence formula? Because if it is 'modularly' then I don't see how programmers, or mathematicians, won't be instrumental in making progress towards AGI. Take for example IBM Watson:
When a question is put to Watson, more than 100 algorithms analyze the question in different ways, and find many different plausible answers–all at the same time. Yet another set of algorithms ranks the answers and gives them a score. For each possible answer, Watson finds evidence that may support or refute that answer. So for each of hundreds of possible answers it finds hundreds of bits of evidence and then with hundreds of algorithms scores the degree to which the evidence supports the answer. The answer with the best evidence assessment will earn the most confidence. The highest-ranking answer becomes the answer. However, during a Jeopardy! game, if the highest-ranking possible answer isn’t rated high enough to give Watson enough confidence, Watson decides not to buzz in and risk losing money if it’s wrong. The Watson computer does all of this in about three seconds.
It needs a company like IBM to design such a narrow AI. More than 100 algorithms. Could it have been done without a lot of computational and intellectual resources? Can progress be made without tapping into the workings of the human brain, without designing specially optimized hardware, without programming and debugging? Does it really only need some smarts and contemplation to come up with a few key insights to get something that can take over the universe? I'd be interested to learn on how one can arrive at that conclusion.
I agree mathematicians are likely to useful in making AGI. If the folks as SIAI were terrible at math, that would be a bad sign indeed.
I wouldn't say 'simple' but, I would be surprised if it were complex in the same way that Watson is complex. Watson is complex because statistical algorithms can be complex, and Watson has a lot of them. As far as I can tell, there's nothing conceptually revolutionary about Watson, it's just a neat and impressive statistical application. I don't see a strong relationship between Watson-like narrow AI and the goal of AGI.
An AGI might have a lot of algorithms (because intelligence turns out to have a lot of separate components), but the difficulty will be understanding the nature of intelligence and coming up with algorithms and proving the important properties about those algorithms. I wouldn't expect "practical implementation" to be a separate step where you need programmers because I would expect everything to be implemented in some kind of proof environment.
I assume that as genetic and bio-engineering and non-organic augmentation come into play, recognizing humans is going to get harder.
I think there might some slight difference in quality in the problems of observing a wide volume of physical reality by varying means and in varying circumstances and unerringly recognizing certain persistent physical processes as corresponding to agents with moral standing such as living, conscious humans, in order to formulate open-ended plans to guarantee their ethically preferred mode of continued existence, and that of analyzing a video feed for patterns that roughly correspond to a 2D projection of an average-human-shaped object and doing some pathfinding adjustments to avoid bumping into it.
A pertinent question is what problem a government or business (not including a general AI startup) may wish to solve with a general AI that is not more easily solved by developing a narrow AI. 'Easy' here factors in the risk of failure, which will at least be perceived as very high for a general AI project. Governments and businesses may fund basic research into general AI as part of a strategy to exploit high-risk high-reward opportunities, but are unlikely to do it in-house.
One could also try and figure out some prerequisites for a general AI, and see what would lead to them coming into play. So for instance, I'm pretty sure that a general AI is going to have long-term memory. What AIs are going to get long-term memory? A general AI is going to be able to generalize its knowledge across domains, and that's probably only going to work properly if it can infer causation. What AIs are going to need to do that?
How much of the world do you need to understand to make reliably good investments?
Do you want your investment computer to be smart enough to say "there's a rather non-obvious huge bubble in the derivatives based on real estate"? Smart enough to convince you when you don't want to believe it?
The idea of creating an AI seems to be getting more common.
To the extent that creating an AI is made easier by having more resources rather than by having more carefully thought out philosophy, the first AI will be created by a government or a business, not SIAI. I think the more resources side is the way to bet, but I'm open to argument.
If this is correct, the best strategy for Friendliness may be to keep working on the philosophy but not expect to code, and publicize the risks of Unfriendliness, both seriously and humorously.
The latter is based on something Scott Adams said (for what that's worth)-- that no one ever realizes they're the pointy-haired boss, but if anyone says "that plan sounds like something out of Dilbert", the plan is immediately taken out of consideration.
The good news, such as it is, is that the mistakes likely to be made by corporations and governments can be presented as funnier (or at least more entertaining to people who already dislike those institutions) than those likely to be made by people who are unthinkingly trying to create utopia.
ETA: It's conceivable that a large organization could have SIAI folks heading its AI project, but this doesn't seem likely.