= 27d567bc2d48d9f40e9ccc20ea370b15)
brandyn comments on Open thread, September 2-8, 2013 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (376)
He's ignoring everything Friendly AI Proponents have said on the issue , and is attacking a strawman instead of the real reasons FAI people think it's a problem.
Trying to understand here. What's the strawman in this case?
Can you point me to an essay that addresses the points in this one?
He doesn't really make any relevant points.
The closest is this:
Which is really just an assertion that you won't get FOOM (I mean, no one thinks it'll take less time than it takes you to hit Ctrl-C, but that's just hyperbole for writing style). He doesn't argue for that claim, he doesn't address any of the arguments for FOOM (most notably and recently: IEM).
Ah, thanks, better understand your position now. I will endeavor to read IEM (if it isn't too stocked with false presuppositions from the get go).
I agree the essay did not endeavor to disprove FOOM, but let's say it's just wrong on that claim, and that FOOM is really a possibility -- then are you saying you'd rather let the military AI go FOOM than something homebrewed? Or are you claiming that it's possible to reign in military efforts in this direction (world round)? Or give me a third option if neither of those applies.
AS to the 'third option', most work that I'm aware of falls into either educating people about AI risk, or trying to solve the problem before someone else builds an AGI. Most people advocate both.
FAI proponents (Of which I am one, yes) tend to say that, ceteris paribus, an AGI which is constructed without first 'solving FAI'* will be 'Unfriendly', military or otherwise. This would be very very bad for humans and human values.
*There is significant disagreement over what exactly this consists of and how hard it will be.
Do you think people who can't implement AGI can solve FAI?
The problem of Friendliness can be worked on before the problem of AGI has been solved, yes
Which do you think is more likely: That you will die of old age, or of unfriendly-AI? (Serious question, genuinely curious.)
I have lots of uncertainty around these sorts of question, especially considering the timeline-dependency and the how-likely-are-MIRI(and others)-to-succed-at-avoiding-UFAI. Suffice to say that it's not an obvious win for 'old age', (which for me is hopefully >50 years away).
Let's imagine you solve FAI tomorrow, but not AGI. (I see it as highly improbable that anyone will meaningfully solve FAI before solving AGI, but let's explore that optimistic scenario.) Meanwhile, various folks and institutions out there are ahead of you in AGI research by however much time you've spent on FAI. At least one of them won't care about FAI.
I have a hard time imagining any outcome from that scenario that doesn't involve you wishing you'd been working on AGI and gotten there first. How do you imagine the outcome?
"worked on" != "solved"
(In addition, MIRI claims that a FAI could be easier to implement than an AGI in general- i.e. that if you solve the philosophical difficulties regarding FAI, this also makes it easier to create an AGI in general. For example, MIRI's specific most-likely scenario for the creation of an AGI is a sub-human AI that self-modifies to become smarter very quickly; MIRI's research on modeling self-modification, while aimed at solving one specific problem that stands in the way of Friendliness, also has potential applications towards understanding self-modification in general.)
drethlin nailed it - If I counterfactually-had spent that time working on AGI, I wouldn't have solved Friendliness, and (unless someone else had solved FAI without me) my AGI would be just as Unfriendly in expectation as the competitors.
If FAI is solved first, however, it increases the probability that the first AGI will be Friendly. Depending on the nature of the solution (how much of it is something that can be published so others can use it with their AGIs?), this could happen through AGI development by people already convinced of the problem, or it could be 'added on' to existing AGi projects.
See reply below to drethlin.
why would he wish that? His unfriendly AI that he'd been working on will probably just kill him.
Sigh.
Ok, I see the problem with this discussion, and I see no solution. If you understood AGI better, you would understand why your reply is like telling me I shouldn't play with electricity because Zeus will get angry and punish the village. But that very concern prevents you from understanding AGI better, so we are at an impasse.
It makes me sad, because with the pervasiveness of this superstition, we've lost enough minds from our side that the military will probably beat us to it.
Just to follow up, I'm seeing nothing new in IEM (or if it's there it's too burried in "hear me think" to find--Eliezer really would benefit from pruning down to essentials). Most of it concerns the point where AGI approaches or exceeds human intelligence. There's very little to support concern for the long ramp up to that point (other than some matter of genetic programming, which I haven't the time to address here). I could go on rather at length in rebuttal of the post-human-intelligence FOOM theory (not discounting it entirely, but putting certain qualitative bounds on it that justify the claim that FAI will be most fruitfully pursued during that transition, not before it), but for the reasons implied in the original essay and in my other comments here, it seems moot against the overriding truth that AGI is going to happen without FAI regardless--which means our best hope is to see AGI+FAI happen first. If it's really not obvious that that has to lead with AGI, then tell me why.
Does anybody really think they are going to create an AGI that will get out of their hands before they can stop it? That they will somehow bypass ant, mouse, dog, monkey, and human and go straight to superhuman? Do you really think that you can solve FAI faster or better than someone who's invented monkey-level AI first?
I feel most of this fear is risidual leftovers from the self-modifying symbolic-program singularity FOOM theories that I hope are mostly left behind by now. But this is just the point -- people who don't understand real AGI don't understand what the real risks are and aren't (and certainly can't mediate them).
Self-modifying AI is the point behind FOOM. I'm not sure why you're connecting self-modification/FOOM/singularity with symbolic programming (I assume you mean GOFAI), but everyone I'm aware of who thinks FOOM is plausible thinks it will be because of self-modification.
Yes, I understand that. But it matters a lot what premises underlie AGI how self-modification is going to impact it. The stronger fast-FOOM arguments spring from older conceptions of AGI. Imo, a better understanding of AGI does not support it.
Thanks much for the interesting conversation, I think I am expired.
I don't think anyone is saying that an 'ant-level' AGI is a problem. The issue is with 'relatively-near-human-level' AGI. I also don't think there's much disagreement about whether a better understanding of AGI would make FAI work easier. People aren't concerned about AI work being done today, except inasmuch as it hastens better AGI work done in the future.
"I mean, no one thinks it'll take less time than it takes you to hit Ctrl-C" -- by the way, are you sure about this? Would it be more accurate to say "before you realize you should hit control-C"? Because it seems to me, if it aint goin' FOOM before you realize you should hit control-C (and do so) then.... it aint goin' FOOM.
More importantly: If someone who KNOWS how important stopping it is sitting at the button, then they're more likely to stop it, but if someone is like "it's getting more powerful and better optimized! Let's see how it looks in a week!" is in charge, then problems.
Well, then, I hope it's someone like you or me that's at the button. But that's not going to be the case if we're working on FAI instead of AGI, is it...
What do you think Ctrl-C does in this?
"The Intelligence Explosion Thesis says that an AI can potentially grow in capability on a timescale that seems fast relative to human experience due to recursive self-improvement. This in turn implies that strategies which rely on humans reacting to and restraining or punishing AIs are unlikely to be successful in the long run, and that what the first strongly self-improving AI prefers can end up mostly determining the final outcomes for Earth-originating intelligent life. " -- Eliezer Yudkowsky, IEM.
I.e., Eliezer thinks it'll take less time than it takes you to hit Ctrl-C. (Granted it takes Eliezer a whole paragraph to say what the essay captures in a phrase, but I digress.)
Eliezer's position is somewhat more nuanced than that. He admits a possibility of a FOOM timescale on the order of seconds, but a timescale on the order of weeks/months/years is also in line with the IE thesis.
How would you know that you have to press Crtl-C? What observation would you need to make?