= 27d567bc2d48d9f40e9ccc20ea370b15)
You (as a group) need "street cred" to be persuasive. To a typical person you look like a modern day version of a doomsday cult. Publishing recognized AI work would be a good place to start.
The issue is that it is a doomsday cult if one is to expect extreme outlier (on doom belief) who had never done anything notable beyond being a popular blogger, to be the best person to listen to. That is incredibly unlikely situation for a genuine risk. Bonus cultism points for knowing Bayesian inference but not applying it here. Regardless of how real is the AI risk. Regardless of how truly qualified that one outlier may be. It is an incredibly unlikely world-state where the AI risk would be best coming from someone like that. No matter how fucked up is the scientific review process, it is incredibly unlikely that world's best AI talk is someone's first notable contribution.
A well-specified math problem, then. By contrast with fusion or space travel.
how is intelligence well specified compared to space travel? We know physics well enough. We know we want to get from point A to point B. The intelligence: we don't even quite know what do exactly we want from it. We know of some ridiculous towers of exponents slow method, that means precisely nothing.
In response to
Reframing the Problem of AI Progress
but brings forward the date by which we must solve it
Does it really? I already explained that if someone makes an automated engineering tool, all users of that tool are at least as powerful as some (U)FAI based upon this engineering tool. Addition of independent will onto tank doesn't make it suddenly win the war against much larger force of tanks with no independent will.
You are rationalizing the position here. If you actually reason forwards, it is clear that creation of such tools may, instead, be the life-saver when someone who thought he solved morality unleashes some horror upon the world. (Or sometime, hardware gets so good that very simple evolution simulator like systems could self improve to point of super-intelligence by evolving, albeit that is very far off into the future)
Suppose I were to convince you of butterfly effect, and explain that you sneezing could kill people, months later. And suppose you couldn't think that non sneezing has same probability. You'd be trying real hard not to sneeze, for nothing, avoid the sudden bright lights (if you have sneeze reflex on bright lights), and so on.
The engineering super-intelligences don't share our values to such profound extent, as to not even share the desire to 'do something' in the real world. Even the engineering intelligence inside my own skull, as far as I can feel. I build designs in real life, because I have rent to pay, or because I am not sure enough it will work and don't trust the internal simulator that I use for design (i.e. imagining) [and that's because my hardware is very flawed]. This is also the case with all my friends whom are good engineers.
The issue here is that you conflate things into 'human level AI'. There's at least three distinct aspects to AI:
1: Engineering, and other problem solving. This is a creation of designs in abstract design space.
2: Will to do something in real world in real time.
3: Morality.
People here see first two as inseparable, while seeing third as unrelated.
It's worth discussing an issue as important as cultishness every so often, but as you might expect, this isn't the first time Less Wrong has discussed the meme of "SIAI agrees on ideas that most people don't take seriously? They must be a cult!"
ETA: That is, I'm not dismissing your impression, just saying that the last time this was discussed is relevant.
Less Wrong has discussed the meme of "SIAI agrees on ideas that most people don't take seriously? They must be a cult!"
Awesome, it has discussed this particular 'meme', to prevalence of viral transmission of which your words seem to imply it attributes it's identification as cult. Has it, however, discussed good Bayesian reasoning and understood the impact of a statistical fact that even when there is a genuine risk (if there is such risk), it is incredibly unlikely that the person most worth listening to will be lacking both academic credentials and any evidence of rounded knowledge, and also be an extreme outlier on degree of belief? There's also the NPD diagnostic criteria to consider. The probabilities multiply here into an incredibly low probability of extreme on many parameters relevant to cult identification, for a non-cult. (For cults, they don't multiply up because there is common cause.)
edit: to spell out details: So you start with prior maybe 0.1 probability that doomsday salvation group is noncult (and that is massive benefit of the doubt right here), then you look at the founder being such incredibly unlikely combination of traits for a non-cult doomsday caution advocate but such a typical founder for a cult - on multitude of parameters - and then you fuzzily do some knee jerk Bayesian reasoning (which however can be perfectly well replicated using a calculator instead of neuronal signals), and you end up virtually certain it is cult. That's if you can do Bayes without doing it explicitly on calculator. Now, the reason I am here, is that I did not take a good look until very recently because I did not care if you guys are a cult or not - the cults can be interesting to argue with. And EY is not a bad guy at all, don't take me wrong, he himself understands that he's risking making a cult, and trying very hard NOT to make a cult. That's very redeeming. I do feel bad for the guy, he happened to let one odd belief through, and then voila, a cult that he didn't want. Or a semi cult, with some people in it for cult reasons and some not so much. He happened not to have formal education, or notable accomplishments that are easily to know are challenging (like being an author of some computer vision library or what ever really). He has some ideas. The cult-follower-type people are dragged towards those ideas like flies to food.
For more subtle cases though - see, the problem is substitution of 'intellectually omnipotent omniscient entity' for AI. If the AI tells to assassinate foreign official, nobody's going to do that; got to be starting the nuclear war via butterfly effect, and that's pretty much intractable.
I would prefer our only line of defense not be "most stupid solutions are going to look stupid". It's harder to recognize stupid solutions in say, medicine (although there we can verify with empirical data).
It is unclear to me that artificial intelligence adds any risk there, though, that isn't present from natural stupidity.
Right now, look, so many plastics around us, food additives, and other novel substances. Rising cancer rates even after controlling for age. With all the testing, when you have hundred random things a few bad ones will slip through. Or obesity. This (idiotic solutions) is a problem with technological progress in general.
edit: actually, our all natural intelligence is very prone to quite odd solutions. Say, reproductive drive, secondary sex characteristics, yadda yadda, end result, cosmetic implants. Desire to sell more product, end result, overconsumption. Etc etc.
Yup, we seem safe for the moment because we simply lack the ability to create anything dangerous.
Sorry you're being downvoted. It's not me.
Yup, we seem safe for the moment because we simply lack the ability to create anything dangerous.
Actually your scenario already happened... Fukushima reactor failure: they used computer modelling to simulate tsunami, it was 1960s, the computers were science woo, and if computer said so, then it was true.
For more subtle cases though - see, the problem is substitution of 'intellectually omnipotent omniscient entity' for AI. If the AI tells to assassinate foreign official, nobody's going to do that; got to be starting the nuclear war via butterfly effect, and that's pretty much intractable.
It isn't an amazing novel philosophical insight that type-1 agents 'love' to solve problems in the wrong way. It is fact of life apparent even in the simplest automated software of that kind.
Of course it isn't.
Let's just assume that mister president sits on nuclear launch button by accident, shall we?
There are machine learning techniques like genetic programming that can result in black-box models. As I stated earlier, I'm not sure humans will ever combine black-box problem solving techniques with self-optimization and attempt to use the product to solve practical problems; I just think it is dangerous to do so once the techniques become powerful enough.
There are machine learning techniques like genetic programming that can result in black-box models.
Which are even more prone to outputting crap solutions even without being superintelligent.
What if instead of giving the solution "cause nuclear war" it simply returns a seemingly innocuous solution expected to cause nuclear war? I'm assuming that the modelling portion is a black box so you can't look inside and see why that solution is expected to lead to a reduction in global temperatures.
If the software is using models we can understand and check ourselves then it isn't nearly so dangerous.
I'm assuming that the modelling portion is a black box so you can't look inside and see why that solution is expected to lead to a reduction in global temperatures.
Let's just assume that mister president sits on nuclear launch button by accident, shall we?
It isn't an amazing novel philosophical insight that type-1 agents 'love' to solve problems in the wrong way. It is fact of life apparent even in the simplest automated software of that kind. You, of course, also have some pretty visualization of what is the scenario where the parameter was minimized or maximized.
edit: also the answers could be really funny. How do we solve global warming? Okay, just abduct the prime minister of china! That should cool the planet off.
Even strongly superhuman 1 by itself is entirely harmless, even if very general within the problem space of 1.
Type 1 intelligence is dangerous as soon as you try to use it for anything practical simply because it is powerful. If you ask it "how can we reduce global temperatures" and "causing a nuclear winter" is in its solution space, it may return that. Powerful tools must be wielded precisely.
See, that's what is so incredibly irritating about dealing with people who lack any domain specific knowledge. You can't ask it, "how can we reduce global temperatures" in the real world.
You can ask it how to make a model out of data, you can ask it what to do to the model so that such and such function decreases, it may try nuking this model (inside the model), and generate such solution. You got to actually put a lot of effort, like replicating it's in-model actions in real world in mindless manner, for this nuking to happen in real world. (and you'll also have the model visualization to examine, by the way)
= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d){kind=logo}
I think this is a mistaken picture of the intellectual history around AI risk.
Prominent AI folk like Hans Moravec and Marvin Minsky had predicted the eclipse of humanity and humane values (save perhaps as pets/specimens/similar, losing the overwhelming majority of the future) long before Yudkowsky. Moravec in particular published a fair bit of his analysis in his books Mind Children and Robot, including a prediction of the eventual "devouring" of humanity by competitive AI life (stored as data and perhaps occasionally pulled out as simulations). Many other AI researchers endorse this rough picture, although often approvingly, saying that these would be "worthy successors" and "humans aren't worth saving" or "nothing to be done about it" and so forth.
Vinge's (a mathematician as well as a sci-fi novelist) 1993 essay has the phrase "How to survive in the post-human era" in the title, and "If not to be avoided, can events be guided so that we may survive?" and discusses risks to humanity extensively (as large).
I.J. Good mentioned the risk of out-of-control AI in the first public writing on the concept of an intelligence explosion.
Marcus Hutter, Jurgen Schmidhuber, Kevin Warwick, and a number of other AI folk have written about the future of AI and risk of human extinction, etc.
Stephen Omohundro is another successful AI guy who has done work in this area.
The concept of "Friendly AI," i.e. an AI with a clear enough model of human welfare to fairly reliably design successors that get better and better at helping humans, was originally created by the philosopher Nick Bostrom, not Yudkowsky.
Yudkowsky has spent more time on the topic than any of the others on this list, and has specific conclusions that are more idiosyncratic (especially the combination of views on many subjects), but the basic ideas are not so rare or privileged that they do not recur independently among many folk, including subject matter experts.
Problem solving works better when you can flexibly reallocate internal resources to old and new uses that will best meet goal criteria, run experiments (at least internal ones, involving writing and using programs), and identify and pursue subtasks.
If it considers, creates, and runs programs in the course of identifying and evaluating possible improvements (with respect to some criteria of improvement), then it doesn't need to acquire some novel will. What is the necessary difference between creating and running a program that does a specialized search or simulation on local hardware and returns an answer, and a program that transmits an AI into the outside world to acquire resources and return an answer? Even more so, if the AI is designed to communicate with outside sources of information like humans or experimental apparatus and use them as "black-boxes" to acquire information.
If you make a point, and someone raises a complication that undercuts your conclusion, it may look to you like it is an instant rationalization to your novel objection. But in fact the points you raise (simulation-based and heuristic instrumental reasons for AI to be wary of immediately killing humans, wireheading, etc) and the counter-considerations you identify as rationalizations, are old news to many of your interlocutors (people like Wei Dai), and were independently invented (by supposed doomsayers) in the course of searching the possibility space, for both good and bad news. ETA: See Wei Dai's link below.
This is not to say that there aren't rationalization and biases of discourse on this subject around here: there are comments and commenters that clearly illustrate those.
The argument is for the insights coming out of EY , and the privileging that EY is making for those hypotheses originated by others, aka cherrypicking what to advertise. EY is a good writer.
edit: concrete thought example: There is a drug A that undergoes many tests, with some of them evaluating it as better than placebo, some as equal to placebo, and some as worse to placebo. Worst of all, each trial is conducted on 1 person's opinion. Comes in the charismatic pharmaceutical marketer, or charismatic anti-vaccination campaign leader, and starts bringing to attention the negative or positive trials. That is not good. Even if there's both of those people.