= 27d567bc2d48d9f40e9ccc20ea370b15)
somervta comments on Open thread, September 2-8, 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (376)
He doesn't really make any relevant points.
The closest is this:
Which is really just an assertion that you won't get FOOM (I mean, no one thinks it'll take less time than it takes you to hit Ctrl-C, but that's just hyperbole for writing style). He doesn't argue for that claim, he doesn't address any of the arguments for FOOM (most notably and recently: IEM).
Ah, thanks, better understand your position now. I will endeavor to read IEM (if it isn't too stocked with false presuppositions from the get go).
I agree the essay did not endeavor to disprove FOOM, but let's say it's just wrong on that claim, and that FOOM is really a possibility -- then are you saying you'd rather let the military AI go FOOM than something homebrewed? Or are you claiming that it's possible to reign in military efforts in this direction (world round)? Or give me a third option if neither of those applies.
AS to the 'third option', most work that I'm aware of falls into either educating people about AI risk, or trying to solve the problem before someone else builds an AGI. Most people advocate both.
FAI proponents (Of which I am one, yes) tend to say that, ceteris paribus, an AGI which is constructed without first 'solving FAI'* will be 'Unfriendly', military or otherwise. This would be very very bad for humans and human values.
*There is significant disagreement over what exactly this consists of and how hard it will be.
Do you think people who can't implement AGI can solve FAI?
The problem of Friendliness can be worked on before the problem of AGI has been solved, yes
Which do you think is more likely: That you will die of old age, or of unfriendly-AI? (Serious question, genuinely curious.)
I have lots of uncertainty around these sorts of question, especially considering the timeline-dependency and the how-likely-are-MIRI(and others)-to-succed-at-avoiding-UFAI. Suffice to say that it's not an obvious win for 'old age', (which for me is hopefully >50 years away).
Let's imagine you solve FAI tomorrow, but not AGI. (I see it as highly improbable that anyone will meaningfully solve FAI before solving AGI, but let's explore that optimistic scenario.) Meanwhile, various folks and institutions out there are ahead of you in AGI research by however much time you've spent on FAI. At least one of them won't care about FAI.
I have a hard time imagining any outcome from that scenario that doesn't involve you wishing you'd been working on AGI and gotten there first. How do you imagine the outcome?
"worked on" != "solved"
(In addition, MIRI claims that a FAI could be easier to implement than an AGI in general- i.e. that if you solve the philosophical difficulties regarding FAI, this also makes it easier to create an AGI in general. For example, MIRI's specific most-likely scenario for the creation of an AGI is a sub-human AI that self-modifies to become smarter very quickly; MIRI's research on modeling self-modification, while aimed at solving one specific problem that stands in the way of Friendliness, also has potential applications towards understanding self-modification in general.)
drethlin nailed it - If I counterfactually-had spent that time working on AGI, I wouldn't have solved Friendliness, and (unless someone else had solved FAI without me) my AGI would be just as Unfriendly in expectation as the competitors.
If FAI is solved first, however, it increases the probability that the first AGI will be Friendly. Depending on the nature of the solution (how much of it is something that can be published so others can use it with their AGIs?), this could happen through AGI development by people already convinced of the problem, or it could be 'added on' to existing AGi projects.
See reply below to drethlin.
why would he wish that? His unfriendly AI that he'd been working on will probably just kill him.
Sigh.
Ok, I see the problem with this discussion, and I see no solution. If you understood AGI better, you would understand why your reply is like telling me I shouldn't play with electricity because Zeus will get angry and punish the village. But that very concern prevents you from understanding AGI better, so we are at an impasse.
It makes me sad, because with the pervasiveness of this superstition, we've lost enough minds from our side that the military will probably beat us to it.
The other thing is: "Our Side" is not losing minds. People are going to try to make AGI regardless of friendliness. but almost no one anywhere has ever heard of AI friendliness and even fewer give a shit. That means that the marginal person working on friendliness is HUGELY more valuable. And if someone discovers friendliness, guess what? The military are going to want it too! Maybe someone actually insane would not, but any organization that has goals and cares about humans at all will be better off with a friendly AI than not.
"but almost no one anywhere has ever heard of AI friendliness"
Ok, if this is your vantage point, I understand better. I must hang in the wrong circles 'cause I meet far more FAI than AGI folks.
You shouldn't play with radiation because you don't understand it and trying to build a bomb might get you and everyone else killed. This isn't a question of superstition, you fool, it's a question of NOT just throwing a bunch of radioactive material in a pile to see what happens, only when you fuck it up you don't just kill yourself like Marie Curie or possibly blow up a few square miles like if the Manhattan Project hadn't been careful enough. You fuck over EVERYTHING.
Appologies for the provacative phrasing--I was (inadvertently) asking for a heated reply...
But to clarify the point in light of your response (which no doubt will get another heated reply, though honestly trying to convey the point w/out provoking...):
Piles of radioactive material is not a good analogy here. But I think it's appearance here is a good illustration of the very thing I'm hoping to convey: There are a lot of (vague, wrong) theories of AGI which map well to the radioactive pile analogy. Just put enough of the ingredients together in a pile, and FOOM. But the more you actually work on AGI, the more you realize how heuristic, incremental, and data bound it is; how a fantastic solution to monkey problems (vision, planning, etc) confers only the weakest ability in symbolic domains, and that, for instance, NP problems most likely remain NP hard regardless of intelligence, and their solutions are limited by time, space, and energy constraints--not cleverness. Can a hyper-intelligent AI improve upon hardware design, etc, etc? Sure! But the whole system (of progress) we're speaking of is a large complex system of differential equations with many bottlenecks, at least some of which aren't readily amendable to hyper-exponential change. Will there be a point where things are out of our (human) hands? Yes. Will it happen over night? No.
The radioactive pile analogy fails because AGI will not be had by heaping a bunch of stuff in a pile--it will be had through extensive engineering and design. It will progress incrementally, and it will be bounded by resource constraints--for a long long time.
A better analogy might be to building a fusion reactor. Sure, you have to be careful, especially in the final design, construction, and execution of the full scale device, but there's a huge amount of engineering to be done before you get anywhere near having to worry about that, and tons of smaller experiments that need to be proven first, and so on. And you learn as you go, and after years of work you start getting closer and you know a shitload about the technology and what it's quirks and hazards are and what's easy to control, and so on.
And when you're there -- well along the way, and you understand the technology to a deep level even if you haven't quite figured out how to make the sustainable fusion reactor yet -- it's pretty insulting/annoying when someone who doesn't have any practical(!) grasp of the matter comes along and tells you you shouldn't be working on this because they haven't figured out how to make it safe yet! And because they think that if you put too much stuff in a pile, it will go boom! (Sigh!) You're on your way to making clean energy before the peak oil appocolypse or whatever, and they're working against you (if only they knew what their efforts were costing the world).
What do you do there? They're fearful because they don't understand, and the particulars of their fear are, really, superstition (in the sense that they are not founded on a solid understanding, but quite specifically on a lack thereof). You want to say: get up to speed, and you can help us make this work and make it safe (and when you understand it better, you'll start to actually understand how that might be done--and how much less of an explosive problem it is than you think). But GTF out of my way if you're just going to pontificate from ignorance and try to dictate how I do my job from there. No matter how long you sofa-think about how to keep the pile-o-stuff from going bad-FOOM, your answers are never going to mesh with reality because you've got way too many false premises that need to be sifted out first (through actual experience in the topic). [And, sorry, but no matter how big Eliezer's cloud of self-citations is, that's just someone else's sofa-think, not actual experience.]
Personally, I do not think FAI is a hard problem (a highly educated opinion, not offhand dismissal). But I also know that UAI is going to happen eventually (intentionally), no matter how many conferences y'all have. And I also know the odds are highest of all we'll all die of old age because AI didn't happen well enough soon enough. But I understand if you disagree.
Just to follow up, I'm seeing nothing new in IEM (or if it's there it's too burried in "hear me think" to find--Eliezer really would benefit from pruning down to essentials). Most of it concerns the point where AGI approaches or exceeds human intelligence. There's very little to support concern for the long ramp up to that point (other than some matter of genetic programming, which I haven't the time to address here). I could go on rather at length in rebuttal of the post-human-intelligence FOOM theory (not discounting it entirely, but putting certain qualitative bounds on it that justify the claim that FAI will be most fruitfully pursued during that transition, not before it), but for the reasons implied in the original essay and in my other comments here, it seems moot against the overriding truth that AGI is going to happen without FAI regardless--which means our best hope is to see AGI+FAI happen first. If it's really not obvious that that has to lead with AGI, then tell me why.
Does anybody really think they are going to create an AGI that will get out of their hands before they can stop it? That they will somehow bypass ant, mouse, dog, monkey, and human and go straight to superhuman? Do you really think that you can solve FAI faster or better than someone who's invented monkey-level AI first?
I feel most of this fear is risidual leftovers from the self-modifying symbolic-program singularity FOOM theories that I hope are mostly left behind by now. But this is just the point -- people who don't understand real AGI don't understand what the real risks are and aren't (and certainly can't mediate them).
Self-modifying AI is the point behind FOOM. I'm not sure why you're connecting self-modification/FOOM/singularity with symbolic programming (I assume you mean GOFAI), but everyone I'm aware of who thinks FOOM is plausible thinks it will be because of self-modification.
Yes, I understand that. But it matters a lot what premises underlie AGI how self-modification is going to impact it. The stronger fast-FOOM arguments spring from older conceptions of AGI. Imo, a better understanding of AGI does not support it.
Thanks much for the interesting conversation, I think I am expired.
I don't think anyone is saying that an 'ant-level' AGI is a problem. The issue is with 'relatively-near-human-level' AGI. I also don't think there's much disagreement about whether a better understanding of AGI would make FAI work easier. People aren't concerned about AI work being done today, except inasmuch as it hastens better AGI work done in the future.
"I mean, no one thinks it'll take less time than it takes you to hit Ctrl-C" -- by the way, are you sure about this? Would it be more accurate to say "before you realize you should hit control-C"? Because it seems to me, if it aint goin' FOOM before you realize you should hit control-C (and do so) then.... it aint goin' FOOM.
More importantly: If someone who KNOWS how important stopping it is sitting at the button, then they're more likely to stop it, but if someone is like "it's getting more powerful and better optimized! Let's see how it looks in a week!" is in charge, then problems.
Well, then, I hope it's someone like you or me that's at the button. But that's not going to be the case if we're working on FAI instead of AGI, is it...
What do you think Ctrl-C does in this?
"The Intelligence Explosion Thesis says that an AI can potentially grow in capability on a timescale that seems fast relative to human experience due to recursive self-improvement. This in turn implies that strategies which rely on humans reacting to and restraining or punishing AIs are unlikely to be successful in the long run, and that what the first strongly self-improving AI prefers can end up mostly determining the final outcomes for Earth-originating intelligent life. " -- Eliezer Yudkowsky, IEM.
I.e., Eliezer thinks it'll take less time than it takes you to hit Ctrl-C. (Granted it takes Eliezer a whole paragraph to say what the essay captures in a phrase, but I digress.)
Eliezer's position is somewhat more nuanced than that. He admits a possibility of a FOOM timescale on the order of seconds, but a timescale on the order of weeks/months/years is also in line with the IE thesis.
How would you know that you have to press Crtl-C? What observation would you need to make?