= 27d567bc2d48d9f40e9ccc20ea370b15)
Rain comments on Tallinn-Evans $125,000 Singularity Challenge - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (369)
The rational reasons to signal are outlined in the post Why Our Kind Can't Cooperate, and there are more good articles with the Charity tag.
My personal reasons for supporting SIAI are outlined entirely in this comment.
Please inform me if anyone knows of a better charity.
As long as you presume that the SIAI saves a potential galactic civilization from extinction (i.e. from being created), and assign a high enough probably to that outcome, nobody is going to be able to inform you of a charity with an higher payoff. At least as long as no other organization is going to make similar claims (implicitly or explicitly).
If you don't mind I would like you to state some numerical probability estimates:
I'd also like you to tackle some problems I see regarding the SIAI in its current form:
Transparency
How do you know that they are trying to deliver what they are selling? If you believe the premise of AI going FOOM and that the SIAI is trying to implement a binding policy based on which the first AGI is going to FOOM, then you believe that the SIAI is an organisation involved in shaping the future of the universe. If the stakes are this high there does exist a lot of incentive for deception. Can you conclude that because someone writes a lot of ethical correct articles and papers that that output is reflective of their true goals?
Agenda and Progress
The current agenda seems to be very broad and vague. Can the SIAI make effective progress given such an agenda compared to specialized charities and workshops focusing on more narrow sub-goals?
As multifoliaterose implied here, at the moment the task to recognize humans as distinguished beings already seems to be too broad a problem to tackle directly. Might it be more effective, at this point, to concentrate on supporting other causes leading towards the general goal of AI associated existential risk mitigation?
Third Party Review
Without being an expert and without any peer review, how sure can you be about the given premises (AI going FOOM etc.) and the effectiveness of their current agenda?
Also what conclusion should one draw from the fact that at least 2 people who have been working for the SIAI, or have been in close contact with it, do disagree with some of the stronger claims. Robin Hanson seems not to be convinced that donating to the SIAI is an effective way to mitigate risks from AI? Ben Goertzel does not believe into the scary idea. And Katja Grace thinks AI is no big threat.
More
My own estimations
Therefore that a donation to the SIAI does pay off: 0.0000003%
I consider the above form of futurism to be the "narrow view". It considers too few possibilities over too short a timespan.
I'm not academic enough to provide the defense you're looking for. Instead, I'll do what I did at the end of the above linked thread, and say you should read more source material. And no, I don't know what the best material is. And yes, this is SIAI's problem. They really do suck at marketing. I think it'd be pretty funny if they failed because they didn't have a catchy slogan...
I will give one probability estimate, since I already linked to it: SIAI fails in their mission AND all homo sapiens are extinct by the year 2100: 90 percent. I'm donating in the hopes of reducing that estimate as much as possible.
One of my main problems regarding risks from AI is that I do not see anything right now that would hint at the possibility of FOOM. I am aware that you can extrapolate from the chimpanzee-human bridge. But does the possibility of superchimpanzee-intelligence really imply superhuman-intelligence? Even if that was the case, which I consider sparse evidence to neglect other risks for, I do not see that it implies FOOM (e.g. vast amounts of recursive self-improvement). You might further argue that even human-level intelligence (EMS or AI) might pose a significant risk when speed-up or by means of brute-force. In any case, I do believe that the associated problems to create any such intelligence are vastly greater than the problem to limit an intelligence, its scope of action. I believe that it is reasonable to assume that there will be a gradual development with many small-impact mistakes that will lead to a thoroughly comprehension of intelligence and its risks before any superhuman-intelligence could pose an existential risk.
I see foom as a completely separate argument from FAI or AGI or extinction risks. Certainly it would make things more chaotic and difficult to handle, increasing risk and uncertainty, but it's completely unnecessary for chaos, risk, and destruction to occur - humans are quite capable of that on their own.
Once an AGI is "out there" and starts getting copied (assuming no foom), I want to make sure they're all pointed in the right direction, regardless of capabilities, just as I want that for nuclear and other weapons. I think there's a possibility we'll be arguing over the politics of enemy states getting an AGI. That doesn't seem to be a promising future. FAI is arms control, and a whole lot more.
I do not see that. The first AGI will likely be orders of magnitude slower (not less intelligent) than a standard human and be running on some specialized computational substrate (supercomputer). If you remove FOOM from the equation then I see many other existential risks being as dangerous as AI associated risks.
Again, a point-in-time view. Maybe you're just not playing it out in your head like I am? Because when you say, "the first AGI will likely be orders of magnitude slower", I think to myself, uh, who cares? What about the one built three years later that's 3x faster and runs on a microcomputer? Does the first one being slow somehow make that other one less dangerous? Or that no one else will build one? Or that AGI theory will stagnate after the first artificial mind goes online? (?!?!)
Why does it have to happen 'in one day' for it to be dangerous? It could take a hundred years, and still be orders of magnitude more dangerous than any other known existential risk.
Yes, because I believe that the development will be gradually enough to tackle any risks on the way to a superhuman AGI, if superhuman capability is possible at all. There are certain limitations. Shortly after the invention of rocket science people landed on the moon. But the development eventually halted or slowed down. We haven't reached other star systems yet. By that metaphor I want highlight that I am not aware of good arguments or other kinds of evidence indicating that an AGI would likely result in a run-away risk at any point of its development. It is possible but I am not sure that because of its low-probability we can reasonable neglect other existential risks. I believe that once we know how to create artificial intelligence capable of learning on a human level our comprehension of its associated risks and ability to limit its scope will have increased dramatically as well.
You're using a different definition of AI than me. I'm thinking of 'a mind running on a computer' and you're apparently thinking of 'a human-like mind running on a computer', where 'human-like' includes a lot of baggage about 'what it means to be a mind' or 'what it takes to have a mind'.
I think any AI built from scratch will be a complete alien, and we won't know just how alien until it starts doing stuff for reasons we're incapable of understanding. And history has proven that the more sophisticated and complex the program, the more bugs, and the more it goes wrong in weird, subtle ways. Most such programs don't have will, intent, or the ability to converse with you, making them substantially less likely to run away.
And again, you're positing that people will understand, accept, and put limits in place, where there's substantial incentives to let it run as free and as fast as possible.
Sorry, I meant human-level learning capability when I said human like.
We have had quite a bit of self-improvement so far - according to my own:
http://www.alife.co.uk/essays/the_intelligence_explosion_is_happening_now/
Progress is accelerating in a sense - due to synergy between developments making progress easier.
...when progress is at its fastest, things might well get pretty interesting. Much in the way of "existential" risk seems pretty unlikely to me - but with speculative far-future events, it is hard to be certain.
What does look as though it will go up against the wall are unmodified human beings - and other multicellular DNA-based lifeforms. There is no way these can compete against engineering and intelligent design - and they look like an unsuitable foundation for building directly on top of.
Some will paint that as an apocalypse - though to me it looks like a sensible, obvious and practically inevitable move. The most reasonable hope for the continued existence of biological humans is in future equivalents of museums and historical simulations - IMO.
To restate my original question, is there anyone out there doing better than your estimated 0.0000003%? Even though the number is small, it could still be the highest.
None whose goal is to save humanity from an existential risk. Although asteroid surveillance might come close, I'm not sure. It is not my intention to claim that donating to the SIAI is worthless, I believe that the world does indeed need an organisation that does tackle the big picture. In other words, I am not saying that you shouldn't be donating to the SIAI, I am happy someone does (if only because of LW). But the fervor in this thread seemed to me completely unjustified. One should seriously consider if there are other groups worthy of promotion or if there should be other groups doing the same as the SIAI or being dealing with one of its sub-goals.
My main problem is how far I should go to neglect other problems in favor of some high-impact low-probability event. If your number of possible beings of human descent is high enough, and you assign each being enough utility, you can outweigh any low probability. You could probably calculate not to help someone who is drowning because 1.) you'd risk your own life and all the money you could make to donate to the SIAI 2.) in that time you could tell 5 people about existential risks from AI. I am exaggerating to highlight my problem. I'm just not educated enough yet, I have to learn more math, especially probability. Right now I feel that it is unreasonable to donate my whole money (or a lot) to the SIAI.
It really saddens me to see how often LW perceives any critique of the SIAI as ill-intentioned. As if people want to destroy the world. There are some morons out there, but most really would like to save the world if possible. They just don't see that the SIAI is a reasonable choice to do so.
I agree with SIAI's goals. I don't see it as "fervor". I see it as: I can do something to make this world a better place (according to my own understanding, in a better way than any other possible), therefore I will do so.
I compartmentalize. Humans are self-contradictory in many ways. I can send my entire bank account to some charity in the hopes of increasing the odds of friendly AI, and I can buy a hundred dollar bottle of bourbon for my own personal enjoyment. Sometimes on the same day. I'm not ultra-rational or pure utilitarian. I'm a regular person with various drives and desires. I save frogs from my stairwell rather than driving straight to work and earning more money. I do what I can.
I have seriously considered it. I have looked for such groups, here and elsewhere, and no one has ever presented a contender. That's why I made my question as simple and straightforward as possible: name something more important. No one's named anything so far, and I still read for many hours each week on this and other such topics, so hopefully if one arises, I'll know and be able to evaluate it.
I donate based on relative merit. As I said at the end of my original supporting post: so far, no one else seems to come close to SIAI. I'm comfortable with giving away a large portion of my income because I don't have much use for it myself. I post it here because it encourages others to give of themselves. I think it's the right thing to do.
I know it's hard to see why. I wish they had better marketing materials. I was really hoping the last challenge, with projects like a landing page, a FAQ, etc., would make a difference. So far, I don't see much in the way of results, which is upsetting.
I still think it's the right place to put my money.
If you're going to do this sort of explicit decomposition at all, it's probably also worth thinking explicitly about the expected value of a donation. That is: how much does your .0001 estimate of SIAI's chance of preventing a humanity-destroying AI go up or down based on an N$ change in its annual revenue?
Thanks, you are right. I'd actually do a lot more but I feel I am not yet ready to tackle this topic mathematically. I only started getting into math in 2009. I asked several times for an analysis with input variables I could use to come up with my own estimations of the expected value of a donation to the SIAI. I asked people who are convinced of the SIAI to provide a decision procedure on how they were convinced. I asked them to lay it open to public inspection so people could reassess the procedure and calculations to compute their own conclusion. In response they asked me to do so myself. I do not take it amiss, they do not have to convince me. I am not able to do so yet. But while learning math I try to encourage other people to think about it.
I feel that this deserves a direct answer. I think it is not just about money. The question would be, what would they do with it, would they actually hire experts? I will assume the best-case scenario here.
If the SIAI would be able to obtain a billion dollars I'd estimate the chance of the SIAI to prevent a FOOMing uFAI 10%.
This part is the one that seems the most different from my own probabilities:
So, do you think the default case is a friendly AI? Or at least innocuous AI? Or that friendly AI is easy enough so that whoever first makes a fooming AI will get the friendliness part right with no influence from the SIAI?
No, I do not believe that the default case is friendly AI. But I believe that AI going FOOM is, if possible at all, very hard to accomplish. Surely everyone agrees here. But at the moment I do not share the opinion that friendliness, that is to implement scope boundaries, is a very likely failure mode. I see it this way, if one can figure out how to create an AGI that FOOM's (no I do not think AGI implies FOOM) then you have a thorough comprehension of intelligence and its associated risks. I just don't see that a group of researchers (I don't believe a mere group is enough anyway) will be smart enough to create an AGI that does FOOM but somehow fail to limit its scope. Please consider reading this comment where I cover this topic in more detail. That is why I believe that only 5% of all AI's going FOOM will be an existential risk to all of humanity. That is my current estimation, I'll of course update on new evidence (e.g. arguments).
I looked at http://lesswrong.com/lw/wf/hard_takeoff/
I was left pretty puzzled about what "AI go FOOM" was actually intended to mean. The page shies away from making any kind of quantifiable statement.
You seem to be assigning probabilities to this - as though it is a well defined idea - but what is it supposed to mean?
I know (I don't), but since I asked Rain to assign probabilities to it I felt that I had to state my own as well. I asked him to do so because I read that some people are arguing in favor of making probability estimates, to say a number. But since I haven't come across much analysis that actually does state numbers I thought I'd ask a donor who contributed the current balance of his bank account.
My understanding is it means "the AI gets to a point where software improvements allow it to outpace us and trick us into doing anything it wants us to, and understand nanotechnology at a scale that it soon has unlimited material power."
Instead of 1e-4 I'd probably put that at 1e-6 to 1e-9, but I have little experience accurately estimating very low probabilities.
(The sticking point of my interpretation is something that seems glossed over in the stuff I've read about it- that the AI only has complete access to software improvements. If it's working on chips made of silicon, all it can do is tell us better chip designs (unless it's hacked a factory, and is able to assemble itself somehow). Even if it's as intelligent as EY imagines it can be, I don't see how it could derive GR from a webcam quality picture; massive intelligence is no replacement for scant evidence. Those problems can be worked around- if it has access to the internet, it's got a lot of evidence and a lot of power- but suggest that in some limited cases FOOM is very improbable.)
I am pretty sure that the "FOOM" term is an attempt to say something about the timescale of the growth of machine intelligence. So, I am sceptical about definitions which involve the concept of trickery. Surely rapid growth need not necessarily involve trickery. My FOOM sources don't seem to mention trickery. Do you have any references relating to the point?
The bit about "trickery" was probably just referencing the weaknesses of AI boxing. You are correct that it's not essential to the idea of hard takeoff.