Vladimir_Nesov16 May 2012 10:47:25PM* 1 point [-]

There's been video or two where Eliezer was called "world's foremost expert on recursive self improvement"

This usually happens when the person being introduced wasn't consulted about the choice of introduction.

TheOtherDave16 May 2012 10:47:14PM4 points [-]

2%?
Seriously?
I am curious as to why your estimate is so high.

nyan_sandwich16 May 2012 10:46:26PM0 points [-]

You are not trying very hard. You missed the thrid alternative: Use your crazy fear mongering to convince him to stop, not just avoid SI.

I hope you're not just using this as a rhetorical opportunity to spread fear about SI.

amcknight16 May 2012 10:43:23PM0 points [-]

By the way, I still can't stop thinking about that post after 6 months. I think it's my favorite wild-idea scenario I've ever heard of.

nyan_sandwich16 May 2012 10:41:11PM-1 points [-]

If you are not totally incompetent or lying out of your ass, please stop. Do not turn it on. At least consult SI.

othercriteria16 May 2012 10:33:06PM2 points [-]

Oh, I see that I misinterpreted FinalState's statement

A paper? I'll write that in a few minutes after I finish the implementation. link

as an indication of only being a few minutes away from having a working implementation.

dlthomas16 May 2012 10:32:45PM* 0 points [-]

Why isn't it a minor nitpick? I mean, we use dimensioned constants in other areas; why, in principle, couldn't the equation be E=mc * (1 m/s)? If that was the only objection, and the theory made better predictions (which, obviously, it didn't, but bear with me), then I don't see any reason not to adopt it. Given that, I'm not sure why it should be a significant objection.

Edited to add: Although I suppose that would privilege the meter and second (actually, the ratio between them) in a universal law, which would be very surprising.

Bugmaster16 May 2012 10:31:41PM0 points [-]

I try to treat all comments with some degree of seriousness, which can be expressed as a floating-point number between 0 and 1 :-)

private_messaging16 May 2012 10:31:38PM* 0 points [-]

The thread about EY's failure to make make many falsifiable predictions is better ad hominem

I meant to provide priors for the expected value of communication with SI. Sorry, can't be done in non ad hominem way. There's been video or two where Eliezer was called "world's foremost expert on recursive self improvement", which normally implies making something self improve.

the speculation about launching terrorist attacks on fab plants is a much more compelling display of potential risk to life and property.

Ahh right, should of also linked this one. I see it was edited replacing 'we' with 'world government' and 'sabotage' with sanctions and military action. BTW that speculation is by gwern, is he working at SIAI?

What probability would you give to FinalState's assertion of having a working AGI?

AGI is ill defined. Of something that would foom as to pose potential danger, infinitesimally small.

Ultimately: I think risk to his safety is small, and payoff is negligible, while the risk from his software is pretty much nonexistent.

Bugmaster16 May 2012 10:29:35PM1 point [-]

I'm not private_messaging, but I think he has a marginally valid point, even though I disagree with his sensational style.

I personally would estimate FinalState's chances of building a working AGI at approximately epsilon, given the total absence of evidence. My opinion doesn't really matter, though, because I'm just some guy with a LessWrong account.

The SIAI folks, on the other hand, have made it their mission in life to prevent the rise of un-Friendly AGI. Thus, they could make FinalState's life difficult in some way, in order to fulfill their core mission. In effect, FinalState's post could be seen as a Pascal's Mugging attempt vs. SIAI.

paper-machine16 May 2012 10:25:34PM* -1 points [-]

What probability would you give to FinalState's assertion of having a working AGI?

0%, since it apparently isn't finished yet.

Will it be finished in a year? 2%, as all other attempts that have reached the "final stages" have failed to build working AGI. The most credible of those attempts were done with groups; it appears FinalState is working alone.

shminux16 May 2012 10:25:01PM6 points [-]

I am in the final implementation stages of the general intelligence algorithm.

it's both amusing and disconcerting that people on this forum treat such a comment seriously.

amcknight16 May 2012 10:22:17PM0 points [-]

Let me try to strengthen your point. If an agent with goal G1 acquires sufficient "philosophical ability", that it concludes that goal G is the right goal to have, that means that it decided that the best way to achieve goal G1 is to pursue what it thinks is the "right goal to have". This would require it to take a kind of normative stance on goal fulfillment, which would require it to have normative machinery, which would need to be implemented in the agents mind. Is it impossible to create an agent without normative machinery of this kind? Does philosophical ability depend directly on normative machinery?

othercriteria16 May 2012 10:20:28PM* 1 point [-]

At least try harder in you fear-mongering. The thread about EY's failure to make make many falsifiable predictions is better ad hominem and the speculation about launching terrorist attacks on fab plants is a much more compelling display of potential risk to life and property.

I agree that this is not a game, although you should note that you are doing EY/SIAI/LessWrong's work for it by trying to scare FinalState.

What probability would you give to FinalState's assertion of having a working AGI?

Bugmaster16 May 2012 10:20:16PM2 points [-]

I don't really see what the risk is...

As far as I understand, the SIAI folks believe that the risk is, "you push the Enter key, your algorithm goes online, bootstraps itself to transhuman superintelligence, and eats the Earth with nanotechnology" (nanotech is just one possibility among many, of course). I personally don't believe we're in any danger of that happening any time soon, but these guys do. They have made it their mission in life to prevent this scenario from happening. Their mission and yours appear to be in conflict.

paper-machine16 May 2012 10:17:57PM* 0 points [-]

Yes, the most that has ever happened to anyone who talked to EY about building an AGI is some mild verbal/textual abuse.

I agree with gwern's assessment of your arguments.

EDIT: Also, I am not affiliated with the SI.

Wei_Dai16 May 2012 10:15:20PM0 points [-]

I wrote earlier

"ability to improve decision theory via philosophical reasoning" (as opposed to CDT-AI changing into XDT<CDT> and then being stuck with that)

XDT<CDT> (or in Eliezer's words, "crippled and inelegant form of TDT") is closer to TDT but still worse. For example, XDT<CDT> would fail to acausally control/trade with other agents living before the time of its self-modification, or in other possible worlds.

private_messaging16 May 2012 10:09:51PM* -2 points [-]

The Roko incident allows to evaluate the sanity of people he'd be talking to.

Other relevant link:

http://acceleratingfuture.com/sl4/archive/0501/10613.html

"And if Novamente should ever cross the finish line, we all die."

Ultimately, you can present your arguments, I can present my arguments, and then he can decide, to talk to you guys, or not.

Jonathan_Graehl16 May 2012 10:04:10PM0 points [-]

Wow. Coordination is hard ;)

Your explanation is more or less what I'd gathered from your earlier statement. It makes sense.

The org. that can convince passionate supporters of the cause to work for $ and donate may be different from the one that can get the most mainstream donations.

paper-machine16 May 2012 10:03:02PM2 points [-]

How highly educated?

You've already gone down this road with Wei Dai. More FUD.

View more: Next