A well-specified math problem, then. By contrast with fusion or space travel.
How a software developer became less concerned with AI risks
This will look like self promotion but I feel that you need to know who I am so that you know who is now less concerned about AI risks than before looking into the issue.
I am a software developer and CGI artist, I am 26 years old. I earn my living mostly from selling a computer game I made but also from my 3D rendering software. There is my website where you can see some of my projects. I am rather good at engineering, even if I myself say so; the reason I can say so is that my products are commercially competitive, and were so since I was 21. My dad and both my granddads are/were engineers with impressive credentials; grand-grand-parents worked in related occupations; everyone in the family can fix broken mechanical devices. I am also very interested in science, especially the invention of ways to know more about the world.
I do not care if someone does, or does not, have academic credentials. If I had two job candidates to evaluate, one without high school diploma, and other with PhD, I would compare the work done by the first to the work done by the second (which may include the PhD thesis).
I have originally posted posts shooting down what I thought are not very good 'safe ai' approaches. There is an example. I didn't come here with made up view on AI risk. I am not working or plan to work on AI beyond tools I make which implement things like 'simulated annealing', search for solutions, and so on. I have no immediate plans to work on AGI of any kind, but of course future is not very predictable.
Having given it more thought, and having been exposed to beliefs here, I became considerably less concerned about the AI risk. There's why:
- The arguments for are pretty bad upon closer scrutiny, and are almost certainly rationalizations rather than rationality. Sorry. The 'random mind design space' is probably the worst offender. The second worst is this conflation of will and problem solving while keeping it purely orthogonal from morality.
- It is incredibly unlikely to find yourself in the world where the significant insights about real doomsday is coming from single visionary who did so little that can be unambiguously graded, before coming up with those insights. It seems to me that majority of worlds with awareness of the problem, have one or many very technically accomplished visionaries in place of Yudkowsky. Very simple probabilistic reasoning (Bayesian, if you insist) makes it incredibly unlikely the AI consequences aspect of lesswrong is not a form of doomsday cult - perhaps a cult within a noncult group.
- The unknown-origin beliefs that I have been exposed to previously trace back to bad ideas, and as I update, those unknown-origin ideas get lower weight.
- One can make equally good arguments for the opposite point of view; this is strong indication that something is very wrong with the argument structure. I posted a graph on the argument structure a couple days back.
- The response to counter-arguments is that of rationalization, not rationality - the arguments are being 'fixed' in precisely the way in which you don't when you aren't rationalizing. For example, if I point out that the AI has good reasons not to kill us all due to it not being able to determine if it is within top level world or a simulator or within engineering test sim. It is immediately conjectured that we will still 'lose' something because it'll take up some resources in space. That is rationalization. Privileging a path of thought. The botched FAI attempts have their specific risk - euthanasia, wireheading, and so on, which don't exist for an AI that is not explicitly friendly.
- There isn't a solid consequentialist reason to think that FAI effort decreases chance of doomsday as opposed to absence of FAI effort. It may increase the chances as easily as decrease.
- It appears to me that will to form most accurate beliefs about the real world, and implement solutions in the real world, is orthogonal to problem solving itself. It is certainly the case for me. My ability to engineer solutions is independent of my will, and while I have a plenty of solutions I implemented, I have very huge number in the desk drawer. No thought is given to this orthogonality. Some of the brightest people work purely within idea space; we call them 'mathematicians'.
- Foom scenario is especially odd in light of the above. Why would optimizing compiler that can optimize it's ability to optimize, suddenly emerge will? It could foom all right, but it wouldn't get out and start touching itself from outside; and if it would, it would wirehead rather than add more hardware; and it would be incredibly difficult to prevent it from doing so.
- Unless the work is in fact focussed in some secret FAI effort, it seems likely that some automated software development tool would foom, reaching close to absolute maximum optimality on certain hardware. But will remain a tool. Availability of such ultra optimal tools in all aspects of software and hardware design would greatly decrease the advantage that self willed UFAI might have.
- The foom looks like a very severely privileged non presently testable hypothesis. I discard such hypotheses unless I generate them myself and can guess how privileged they are. I don't like when someone picks up untestable hypotheses out of scifi. That is a very bad habit. Especially for Bayesians. You can be off on your priors by ten orders of magnitude, or even a hundred orders of magnitude.
There's a story from my past.
I had very intelligent friend as a child - very high IQ - who, if we were to discuss my experiments, would come up with ideas like e.g. that you could make high voltage by connecting wall outlets in series, which instantly obviously will at best blow the fuse. No, we didn't make it but I literally couldn't explain the problem to him without drawing the circuit, because he abstracted the outlet as ideal voltage source and wouldn't step back from that easily. I remember it very well because it was most puzzling example of an incredibly stupid cognitive mishap by someone with very high IQ . He was quite prone to such ideas; good thing he didn't try them in practice. Such cognitive mishaps are the cause behind most if not all cases of fatally misbehaving technology. I love reading about misbehaving tech, like Chernobyl disaster. It's not the known unknowns, for the most part, that kill people. Not the things that someone sees from afar. It's the unknown unknowns. Carbon tipped control rods - which one can abstractly think would increase controllability and thus safety, blew up the Chernobyl power plant. EY very strongly pattern-matches to this friend of mine, and focusses very hard on the known unknowns aspect of the problem about which we know very little - which can easily steer one into a very dangerous zone full of unknown unknowns - the not-quite-FAIs that euthanize us or worse.
I originally dismissed EY as harmless due to inability to make an AI, but in so much as there's probability that this judgement is wrong, or some engineers are hired, there is an open letter: I urge EY: please, do some testing on yourself to see how good are you at foreseeing issues like this wall socket example, in more complex situations. Participate in programming contests or something, and see how your software misbehaves on you when you are very sure it won't. This will also win you street cred. I would take you far more seriously if you spent 1 week of your time to get into first 5 on a marathon contest on TopCoder . The TopCoder is slightly evil, but the benefit is larger. (The contest specific experience is unimportant. I got second place first time I tried; I only wasted 4 days, not full week. I never did a programming contest before that).
Now, do not think of it in terms of fixing the good idea's argument, please. Treat it as evidence that the idea is, actually, bad, and process it as to make a better idea - which may or may not coincide with original idea. You can't right now know if your idea is in fact good or not - rather than fixing you should make a new idea. To do anything else is not rationality. It is rationalization. It is to become even more wrong by making even more privileged hypotheses, and make even worse impression on the engineers whom you try to convince.
You (LW) may dislike this. You can provide me with a poll results informing me that you dislike this, if you wish (This is pretty silly if you ask me; you think you are rating me, but clearly, if I am not mentally handicapped individual, all that does is providing me with pieces of information which I can use to many purposes besides self evaluation; I self evaluate by trying myself on practical problems, or when I actually care.).
If you care for future of mankind, and if you believe in AI risks, and if a software developer, after an encounter with you, becomes *less* worried of the AI risk, then clearly you are doing something wrong. I never knew any of you guys before, never met any of you in real life, and have no prior grudges. I am blind to sound of your voice, look on your face, and so on. You are purely represented to me by how you present your ideas.
If you are an engineering person and lesswrong is making you less concerned about the AI risk - let them know. That certainly won't hurt.
(and why I am posting this: looking at the donations received by SIAI and having seen talk of hiring software developers, I got pascal-wagered into explaining it)
how is intelligence well specified compared to space travel? We know physics well enough. We know we want to get from point A to point B. The intelligence: we don't even quite know what do exactly we want from it. We know of some ridiculous towers of exponents slow method, that means precisely nothing.
but brings forward the date by which we must solve it
Does it really? I already explained that if someone makes an automated engineering tool, all users of that tool are at least as powerful as some (U)FAI based upon this engineering tool. Addition of independent will onto tank doesn't make it suddenly win the war against much larger force of tanks with no independent will.
You are rationalizing the position here. If you actually reason forwards, it is clear that creation of such tools may, instead, be the life-saver when someone who thought he solved morality unleashes some horror upon the world. (Or sometime, hardware gets so good that very simple evolution simulator like systems could self improve to point of super-intelligence by evolving, albeit that is very far off into the future)
Suppose I were to convince you of butterfly effect, and explain that you sneezing could kill people, months later. And suppose you couldn't think that non sneezing has same probability. You'd be trying real hard not to sneeze, for nothing, avoid the sudden bright lights (if you have sneeze reflex on bright lights), and so on.
The engineering super-intelligences don't share our values to such profound extent, as to not even share the desire to 'do something' in the real world. Even the engineering intelligence inside my own skull, as far as I can feel. I build designs in real life, because I have rent to pay, or because I am not sure enough it will work and don't trust the internal simulator that I use for design (i.e. imagining) [and that's because my hardware is very flawed]. This is also the case with all my friends whom are good engineers.
The issue here is that you conflate things into 'human level AI'. There's at least three distinct aspects to AI:
1: Engineering, and other problem solving. This is a creation of designs in abstract design space.
2: Will to do something in real world in real time.
3: Morality.
People here see first two as inseparable, while seeing third as unrelated.
It's worth discussing an issue as important as cultishness every so often, but as you might expect, this isn't the first time Less Wrong has discussed the meme of "SIAI agrees on ideas that most people don't take seriously? They must be a cult!"
ETA: That is, I'm not dismissing your impression, just saying that the last time this was discussed is relevant.
Less Wrong has discussed the meme of "SIAI agrees on ideas that most people don't take seriously? They must be a cult!"
Awesome, it has discussed this particular 'meme', to prevalence of viral transmission of which your words seem to imply it attributes it's identification as cult. Has it, however, discussed good Bayesian reasoning and understood the impact of a statistical fact that even when there is a genuine risk (if there is such risk), it is incredibly unlikely that the person most worth listening to will be lacking both academic credentials and any evidence of rounded knowledge, and also be an extreme outlier on degree of belief? There's also the NPD diagnostic criteria to consider. The probabilities multiply here into an incredibly low probability of extreme on many parameters relevant to cult identification, for a non-cult. (For cults, they don't multiply up because there is common cause.)
edit: to spell out details: So you start with prior maybe 0.1 probability that doomsday salvation group is noncult (and that is massive benefit of the doubt right here), then you look at the founder being such incredibly unlikely combination of traits for a non-cult doomsday caution advocate but such a typical founder for a cult - on multitude of parameters - and then you fuzzily do some knee jerk Bayesian reasoning (which however can be perfectly well replicated using a calculator instead of neuronal signals), and you end up virtually certain it is cult. That's if you can do Bayes without doing it explicitly on calculator. Now, the reason I am here, is that I did not take a good look until very recently because I did not care if you guys are a cult or not - the cults can be interesting to argue with. And EY is not a bad guy at all, don't take me wrong, he himself understands that he's risking making a cult, and trying very hard NOT to make a cult. That's very redeeming. I do feel bad for the guy, he happened to let one odd belief through, and then voila, a cult that he didn't want. Or a semi cult, with some people in it for cult reasons and some not so much. He happened not to have formal education, or notable accomplishments that are easily to know are challenging (like being an author of some computer vision library or what ever really). He has some ideas. The cult-follower-type people are dragged towards those ideas like flies to food.
For more subtle cases though - see, the problem is substitution of 'intellectually omnipotent omniscient entity' for AI. If the AI tells to assassinate foreign official, nobody's going to do that; got to be starting the nuclear war via butterfly effect, and that's pretty much intractable.
I would prefer our only line of defense not be "most stupid solutions are going to look stupid". It's harder to recognize stupid solutions in say, medicine (although there we can verify with empirical data).
It is unclear to me that artificial intelligence adds any risk there, though, that isn't present from natural stupidity.
Right now, look, so many plastics around us, food additives, and other novel substances. Rising cancer rates even after controlling for age. With all the testing, when you have hundred random things a few bad ones will slip through. Or obesity. This (idiotic solutions) is a problem with technological progress in general.
edit: actually, our all natural intelligence is very prone to quite odd solutions. Say, reproductive drive, secondary sex characteristics, yadda yadda, end result, cosmetic implants. Desire to sell more product, end result, overconsumption. Etc etc.
Yup, we seem safe for the moment because we simply lack the ability to create anything dangerous.
Sorry you're being downvoted. It's not me.
Yup, we seem safe for the moment because we simply lack the ability to create anything dangerous.
Actually your scenario already happened... Fukushima reactor failure: they used computer modelling to simulate tsunami, it was 1960s, the computers were science woo, and if computer said so, then it was true.
For more subtle cases though - see, the problem is substitution of 'intellectually omnipotent omniscient entity' for AI. If the AI tells to assassinate foreign official, nobody's going to do that; got to be starting the nuclear war via butterfly effect, and that's pretty much intractable.
It isn't an amazing novel philosophical insight that type-1 agents 'love' to solve problems in the wrong way. It is fact of life apparent even in the simplest automated software of that kind.
Of course it isn't.
Let's just assume that mister president sits on nuclear launch button by accident, shall we?
There are machine learning techniques like genetic programming that can result in black-box models. As I stated earlier, I'm not sure humans will ever combine black-box problem solving techniques with self-optimization and attempt to use the product to solve practical problems; I just think it is dangerous to do so once the techniques become powerful enough.
There are machine learning techniques like genetic programming that can result in black-box models.
Which are even more prone to outputting crap solutions even without being superintelligent.
What if instead of giving the solution "cause nuclear war" it simply returns a seemingly innocuous solution expected to cause nuclear war? I'm assuming that the modelling portion is a black box so you can't look inside and see why that solution is expected to lead to a reduction in global temperatures.
If the software is using models we can understand and check ourselves then it isn't nearly so dangerous.
I'm assuming that the modelling portion is a black box so you can't look inside and see why that solution is expected to lead to a reduction in global temperatures.
Let's just assume that mister president sits on nuclear launch button by accident, shall we?
It isn't an amazing novel philosophical insight that type-1 agents 'love' to solve problems in the wrong way. It is fact of life apparent even in the simplest automated software of that kind. You, of course, also have some pretty visualization of what is the scenario where the parameter was minimized or maximized.
edit: also the answers could be really funny. How do we solve global warming? Okay, just abduct the prime minister of china! That should cool the planet off.
Even strongly superhuman 1 by itself is entirely harmless, even if very general within the problem space of 1.
Type 1 intelligence is dangerous as soon as you try to use it for anything practical simply because it is powerful. If you ask it "how can we reduce global temperatures" and "causing a nuclear winter" is in its solution space, it may return that. Powerful tools must be wielded precisely.
See, that's what is so incredibly irritating about dealing with people who lack any domain specific knowledge. You can't ask it, "how can we reduce global temperatures" in the real world.
You can ask it how to make a model out of data, you can ask it what to do to the model so that such and such function decreases, it may try nuking this model (inside the model), and generate such solution. You got to actually put a lot of effort, like replicating it's in-model actions in real world in mindless manner, for this nuking to happen in real world. (and you'll also have the model visualization to examine, by the way)
Subscribe to RSS Feed
= f037147d6e6c911a85753b9abdedda8d)
You (as a group) need "street cred" to be persuasive. To a typical person you look like a modern day version of a doomsday cult. Publishing recognized AI work would be a good place to start.
The issue is that it is a doomsday cult if one is to expect extreme outlier (on doom belief) who had never done anything notable beyond being a popular blogger, to be the best person to listen to. That is incredibly unlikely situation for a genuine risk. Bonus cultism points for knowing Bayesian inference but not applying it here. Regardless of how real is the AI risk. Regardless of how truly qualified that one outlier may be. It is an incredibly unlikely world-state where the AI risk would be best coming from someone like that. No matter how fucked up is the scientific review process, it is incredibly unlikely that world's best AI talk is someone's first notable contribution.