Open thread, September 2-8, 2013

David_Gerard

If it's worth saying, but not worth its own post (even in Discussion), then it goes here.

Let's imagine you solve FAI tomorrow, but not AGI. (I see it as highly improbable that anyone will meaningfully solve FAI before solving AGI, but let's explore that optimistic scenario.) Meanwhile, various folks and institutions out there are ahead of you in AGI research by however much time you've spent on FAI. At least one of them won't care about FAI.

I have a hard time imagining any outcome from that scenario that doesn't involve you wishing you'd been working on AGI and gotten there first. How do you imagine the outcome?

"worked on" != "solved"

(In addition, MIRI claims that a FAI could be easier to implement than an AGI in general- i.e. that if you solve the philosophical difficulties regarding FAI, this also makes it easier to create an AGI in general. For example, MIRI's specific most-likely scenario for the creation of an AGI is a sub-human AI that self-modifies to become smarter very quickly; MIRI's research on modeling self-modification, while aimed at solving one specific problem that stands in the way of Friendliness, also has potential applications towards understanding self-modification in general.)

2somervta13y

drethlin nailed it - If I counterfactually-had spent that time working on AGI, I wouldn't have solved Friendliness, and (unless someone else had solved FAI without me) my AGI would be just as Unfriendly in expectation as the competitors. If FAI is solved first, however, it increases the probability that the first AGI will be Friendly. Depending on the nature of the solution (how much of it is something that can be published so others can use it with their AGIs?), this could happen through AGI development by people already convinced of the problem, or it could be 'added on' to existing AGi projects.

2drethelin13y

why would he wish that? His unfriendly AI that he'd been working on will probably just kill him.