NancyLebovitz comments on The UFAI among us - Less Wrong

1 Post author: PhilGoetz 08 February 2011 11:29PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (86)

Tags: ai business law cev fai

Comments (86)

Sort By: Best

You are viewing a single comment's thread.

Comment author: NancyLebovitz 08 February 2011 11:37:00PM 6 points [-]

Should other large human organizations like governments and some religions also count as UFAIs?

Comment author: Eliezer_Yudkowsky 09 February 2011 06:43:46PM 10 points [-]

Yes, I find it quite amusing that some people of a certain political bent refer to "corporations" as superintelligences, UFAIs, etcetera, and thus insist on diverting marginal efforts that could have been directed against a vastly underaddressed global catastrophic risk to yet more tugging on the same old rope that millions of other people are pulling on, based on their attempt to reinterpret the category-word; and yet oddly enough they don't think to extend the same anthropomorphism of demonic agency to large organizations that they're less interested in devalorizing, like governments and religions.

Comment author: [deleted] 11 February 2011 09:01:18AM 2 points [-]

Maybe those people are prioritising the things that seem to affect their lives? I can certainly see exactly the same argument about government or religion as about corporations, but currently the biggest companies (the Microsofts and Sonys and their like) seem to have more power than even some of the biggest governments.

Comment author: anonym 13 February 2011 08:18:21PM 1 point [-]

There is also the issue of legal personality, which applies to corporations and not to governments or religions.

The corporation actually seems to me a great example of a non-biological, non-software optimization process, and I'm surprised at Eliezer's implicit assertion that there is no significant difference between corporations, governments, and religions with respect to their ability to be unfriendly optimization processes, other than that some people of a certain political bent have a bias to think about corporations differently than other institutions like governments and religions.

Comment author: NancyLebovitz 10 February 2011 12:03:46AM 0 points [-]

I think such folks are likely to trust governments too much. They're more apt to oppose specific religious agendas than to oppose religion as such, and I actually think that's about right most of the time.

Comment author: CronoDAS 08 February 2011 11:44:05PM 2 points [-]

Probably.

Comment author: PhilGoetz comment score below threshold [+] (15 children)

Comment author: Nick_Tarleton 12 February 2011 08:30:58AM 2 points [-]

We shouldn't assume that every AI not designed using rigorous and deliberate FAI methodology is a UFAI. That's a rhetorical flourish, not a documented fact.

Neither; it's the conclusion of a logical argument (which is, yes, weaker than a documented fact).

Comment author: PhilGoetz 14 February 2011 03:58:53AM -1 points [-]

Nick, I disagree. You are saying there is a logical argument that concludes such AIs will be unfriendly with 100% probability. That just isn't true, or even close to true.

Furthermore, even if there were an argument using these concepts that concluded something with 100% probability, the concepts of UFAI and FAI are not well-defined enough to draw the conclusion above.

I think you're using the word "assume" here to mean something more like, "We should not build AIs without FAI methodology." That's a very very different statement! That's a conclusion based on using expectation-maximization over all possible outcomes. What I am saying is that we should not assume that, in all possible outcomes, the AI comes out unfriendly.

Comment author: wedrifid 14 February 2011 04:18:26AM * 0 points [-]

You are saying there is a logical argument that concludes such AIs will be unfriendly with 100% probability.

No, Nick is not saying that.

Comment author: PhilGoetz 20 February 2011 07:51:15PM * -2 points [-]

Yes, he is. He said there is a logical argument that concludes that we should assume that every AI not designed using rigorous and deliberate FAI methodology is a UFAI. "Assume" means "assign 100% probability". What other meaning did you have in mind?

Comment author: Dorikka 09 February 2011 12:30:53AM -2 points [-]

We shouldn't assume that every AI not designed using rigorous and deliberate FAI methodology is a UFAI.

Why not? It's an assumption which may be slightly overcautious, but I would far rather be slightly overcautious than to increase the risk that an AI is going to smiley-tile the universe. Until we have a more precise idea of what AI-not-designed-using-rigorous-and-deliberate-FAI-methodology is not a UFAI, I see no reason to abandon the current hypothesis.

Comment author: David_Gerard 09 February 2011 09:01:14AM 6 points [-]

Because it fails to quite match reality. e.g. charitable corporations can behave pathologically (falling prey to the Iron Law of Institutions), but are generally qualitatively less unFriendly than the standard profit-making corporation.

Comment author: PhilGoetz 09 February 2011 12:43:43AM 4 points [-]

If you believe it is overcautious, then you believe it is wrong. If you are worried about smiley-tiling, then you get the right answer by assigning the right value to that outcome. Not by intentionally biasing your decision process.

Comment author: Dorikka 09 February 2011 12:50:37AM 0 points [-]

I say 'may be slightly overcautious' contingent on it being wrong -- I'm saying that if it is wrong, it's a sort of wrong which will result in less loss in utility than would being wrong in the other direction.

If you're an agent with infinite computing power, you can investigate all hypotheses further to make sure that you're right. Humans, however, are forced to devote time and effort to researching those things which are likely to yield utility, and I think that the current hypothesis sounds reasonable unless you have evidence that it is wrong.

Comment author: sark 09 February 2011 06:06:13PM 2 points [-]

The erring on the side of caution only enters when you have to make a decision. Your pre-action estimate should be clean of this.

Comment author: PhilGoetz 14 February 2011 04:03:01AM * -2 points [-]

You should not err on the side of caution if you are a Bayesian expectation-maximizer!

But I think what you're getting at, which is the important thing, is that people say "Assume X" when they really mean "My computation of the expected value times probability over all possible outcomes indicates X is likely, and I'm too lazy to remember the details, or I think you're too stupid to do the computation right; so I'm just going to cache 'assume X' and repeat that from now on". They ruin their analysis because they're lazy, and don't want to do more analysis than they would need to do in order to decide what action to take if they had to make the choice today. Then the lazy analysis done with poor information becomes dogma. As in the example above.

Comment author: wedrifid 14 February 2011 04:22:36AM 0 points [-]

Then the lazy analysis done with poor information becomes dogma. As in the example above.

I downvoted this sentence.

Comment author: PhilGoetz 20 February 2011 07:48:09PM * -1 points [-]

Instead of downvoting a comment for referring to another comment that you disagree with, I think you should downvote the original comment.

Better yet, explain why you downvoted. Explaining what you downvoted is going halfway, so I half-appreciate it.

I can't express strongly enough my dismay that here, on a forum where people are allegedly devoted to rationality, they still strongly believe in making some assumptions without justification.

continue this thread »

Comment author: wedrifid 12 February 2011 08:34:44AM 0 points [-]

That's a rhetorical flourish, not a documented fact.

Nothing indicates a rhetorical flourish like the phrase 'rhetorical flourish'.

Comment author: PhilGoetz 14 February 2011 03:59:18AM -2 points [-]

Proof that conformist mindless dogma is alive and well at LW...

Comment author: Alexandros 09 February 2011 09:09:01AM 0 points [-]

Funny you should mention that. Just yesterday I added on my list of articles-to-write one by the title of "Religions as UFAI". In fact, I think the comparison goes much deeper than it does for corporations.

Comment author: timtyler 15 May 2012 11:02:46AM 0 points [-]

Some corporations may become machine intelligences. Religions - probably not so much.

NancyLebovitz comments on The UFAI among us - Less Wrong

Article Navigation

Comments (86)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (86)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login