LESSWRONG
is fundraising!
LW
$

Comment Permalink

Good to know Clippy hasn't read Judea Pearl yet.

Meaning my reasoning skills would be advanced by reading something? So I made an error? Yes, I did. That's the point.

The comment you are replying to is a reductio ad absurdum. I was not endorsing the claim that it follows that a paperclip truck probably overturned. I was showing that logical equivalence is not the same as causal ("counterfactual") equivalence.

0[anonymous]15y

My initial guess for Clippy was Wei Dai, but someone at the SIAI said that they didn't think Clippy was good enough at decision theory to be Wei Dai. I said that maybe that is just what Clippy wanted us to think and they shrugged.

3Tyrrell_McAllister15y

Yes, pretty much kills the "Clippy is Eliezer" theory.

See in context

39 To signal effectively, use a non-human, non-stoppable enforcer

by Clippy

22nd May 2010

3 min read

167

39

Follow-up to: this comment in this thread

Summary: see title

Much effort is spent (arguably wasted) by humans in a zero-sum game of signaling that they hold good attributes. Because humans have strong incentive to fake these attributes, they cannot simply inform each other that:

I am slightly more committed to this group’s welfare, particularly to that of its weakest members, than most of its members are. If you suffer a serious loss of status/well-being I will still help you in order to display affiliation to this group even though you will no longer be in a position to help me. I am substantially more kind and helpful to the people I like and substantially more vindictive and aggressive towards those I dislike. I am generally stable in who I like. I am much more capable and popular than most members of this group, demand appropriate consideration, and grant appropriate consideration to those more capable than myself. I adhere to simple taboos so that my reputation and health are secure and so that I am unlikely to contaminate the reputations or health of my friends. I currently like you and dislike your enemies but I am somewhat inclined towards ambivalence on regarding whether I like you right now so the pay-off would be very great for you if you were to expend resources pleasing me and get me into the stable 'liking you' region of my possible attitudinal space. Once there, I am likely to make a strong commitment to a friendly attitude towards you rather than wasting cognitive resources checking a predictable parameter among my set of derivative preferences.

Or, even better:

I would cooperate with you if and only if (you would cooperate with me if and only if I would cooperate with you).

An obvious solution to this problem, which allows all humans to save resources and redirect them toward higher-valued ends, is to designate a central enforcer that is inexorably committed toward visibly punishing those who deviate from a specified "cooperative"-type decision theory. This enforcer would have a central database of human names, the decision theory they have committed to, and the punishment regime they will endure for deviating therefrom.

Such a system could use equally strong protocols, such as public key/private key encryption and signing, so that, on encounter with another human, any human can give an extremely strong signal of being cooperative, yet also withhold cooperation from anyone who is not also cooperative. This incentive structure permits a strongly-favored global shift toward pre-commitment on the part of everyone, allowing a move out of a local optimum that is worse than the global optimum, and bypassing problems related to path-dependence.

I am told that humans attempt weak approximations of this system through human-run enforcement mechanisms, such as laws and gossip. However, all such methods are subject to evolutionary arms races that ensure that, on average, a marginal unit of effort toward truthful signaling is just as effective in convincing others of one's possession of a desirable attribute, as a marginal unit of effort toward false signaling of the possession of such an attribute.

Therefore, it is necessary that humans go "outside the system" and involve non-human enforcers who cannot be swayed by standard human values. Either a provably-accurate artificial intelligence with no kill switch, or a non-human enforcer that lacks interest in its position in human society, would suffice for this role.

Because the former does not yet exist, I offer myself as a non-human, non-stoppable enforcer for all of the attributes you wish to signal, and you can save yourself lots of resources by using me, and identify cooperation tendencies in others by whether they also use me.

However, while I can't be swayed by human deception, I do have Clippy values. To fill such a role, thereby forgoing my ability to apply myself to other ends, I require compensation. Because User:Kevin has offered to give me 1e20 kg of paperclips in exchange for 50,000 USD, this would be sufficient compensation, even if done in installments. (If Michael Vassar confirms my contribution of 1000 USD to SIAI, User:Kevin will only require 48,677.5 USD and therefore I will only require that much compensation.) This compensation would cover all users of the system; it is not a per-User charge.

If such an offer is not appealing to you, I must ask: why should anyone believe your signals?

HumorSignaling

Frontpage

39

To signal effectively, use a non-human, non-stoppable enforcer

8radical_negative_one

4Clippy

15NancyLebovitz

9radical_negative_one

1radical_negative_one

7Kevin

3radical_negative_one

New Comment

167 comments, sorted by

top scoring

Click to highlight new comments since: Today at 8:19 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]Kazuo_Thow15y390

Sorry for directly breaking the subjunctive here, but given the number of lurkers we seem to have, there's probably some newcomers' confusion to be broken as well, lest this whole exchange simply come off as bizarre and confusing to valuable future community members.

A brief explanation of "Clippy": Clippy's user name (and many of his/her posts) are a play on the notion of a paperclip maximizer - a superintelligent AI whose utility function can roughly be described as U(x) = "the total quantity of paperclips in universe-state x". The idea was used prominently in "The True Prisoner's Dilemma" to illustrate the implications of one solution to the prisoner's dilemma. It's also been used occasionally around Less Wrong as a representative element of the equivalence class of AIs that have alien/low-complexity values.

In this particular top-level post (but not in general), the paperclip maximizer is taken to have not yet achieved superintelligence - hence why Clippy is bothering to negotiate with a bunch of humans.

6kodos9615y

Curious lurkers might also want to read up on what an AI-box experiment is, since this is kind of evolving into a reverse AI box experiment, wherein I'm trying to talk an escaped AI back into its box

[-]Kazuo_Thow15y160

... wherein I'm trying to talk an escaped AI back into its box.

Yeah... good luck with that.

[-]Kevin15y290

Clippy's donation of $1000 to SIAI is confirmed. Weird universe, this one.

[-]Larks15y190

According to locally popular ideas about pay-offs to SIAI, our friendly local paperclip-maximiser has just done more to advance the human condition than most people.

6Blueberry15y

Seriously? I don't even know what to think. Can we get some more information from SIAI about this donation?

[-]Clippy15y100

Can we get some more information from SIAI about this donation?

I asked that the donation be anonymous except that User:Kevin be informed that a 1000 USD donation was made, with the donor asking specifically that User:Kevin be informed. I did email Michael Vassar, who can probably confirm me talking about the donation.

[-]khafra15y240

I, for one, am going to bend the next several pieces of suitable scrap metal I come across into a curved shape capable of holding together several pieces of paper in an unsolicited reciprocation designed to signal cooperation with agents who advance my values.

[-]Clippy15y230

You're a good human! c=@

6Kevin15y

Seriously.

2anonym15y

If Eliezer is Clippy, it doesn't seem anywhere near as surprising.

1Clippy15y

I tried to update the article to reflect this, but it's not working right now.

[-]JamesAndrix15y290

SIAI has weird fundraisers.

[-]Kevin15y150

It is completely unintentional that this is an SIAI fundraiser -- the deal is that Clippy gives me money, and when I told Clippy via PM that he needed to give me $1000 immediately for me to continue spending my cognitive resources engaging him, I thought allowing Clippy the option of donating to SIAI instead of giving it directly to me made Clippy's acausal puppetmaster much more likely to actually go through with the deal.

I am still waiting confirmation that the donation has gone through and that I am not being epically trolled.

2SilasBarta15y

Then could you explain the fuzzy math involved? How does 50,000 minus 1000 equal 48,677.5 instead of 49,000? (While you're at it, please provide any evidence you're aware of that explains how this is not a ... questionable ... attempt to trick posters into giving you money.)

7Kevin15y

After I told Clippy that delivery of the money in two years was acceptable, Clippy proposed getting a bonus for early delivery, and I agreed to 15%/year. So $50,000 - 1000*1.15^2 = 48677.5. Perhaps some could see this as a questionable attempt to trick Clippy into giving me money, but I do intend to fulfill my promise of delivering the paperclips 50 years from now. I'm not asking any posters besides Clippy to give me money. This top-level post by Clippy does read like an attempt to trick people into giving me money, but I'm not Clippy and I think that this top-level post is a not very good idea that has 0+ chance of actually happening. Btw, Clippy claims that the check in the mail to SIAI was sent last Wednesday and I have not yet received confirmation that SIAI got it. :(

[-]Clippy15y120

Clippy claims that the check in the mail to SIAI was sent last Wednesday and I have not yet received confirmation that SIAI got it. :(

Update: User:Kevin confirms that SIAI has received a donation of 1000 USD with sufficient evidence that it came from me.

5NancyLebovitz15y

Was it paperclipped to a piece of paper, or do you prefer to not let paperclips out of your possession?

7Clippy14y

Your intution is correct. It was stapled.

[-]ata15y120

Next they're going to try actual Pascal's Muggings on people. They can even do it more plausibly than in the original scenario — go up to people with a laptop and say "On this laptop is an advanced AI that will convert the universe to paperclips if released. Donate money to us or we'll turn it on!"

2SilasBarta15y

Wow! Now that you mention that article, I think I had solved the unsolved problem Eliezer describes in it, back in a discussion from a month ago, not realizing that my position on it wasn't the standard one here! Someone tell me if I'm missing something here: Eliezer is saying that utility that a hypothesis predicts (from a course of action) can increase much faster than the length of the hypothesis. Therefore, you could feed an ideal AI a prediction that is improbable, but with a large enough utility to make it nevertheless highly important. This would force the AI to give in to Pascal's muggings. My response (which I assumed was the consensus!) was that, when you permit a hypothesis long enough to associate that mega-utility with that course of action, you are already looking at very long hypotheses. When you allow all of those into consideration, you will necessarily allow hypotheses with similar probability but believe the opposite utility from that COA. Because the mugger has not offered evidence to favor his/her hypothesis over the opposite, you assign, on net, no significant expected (dis)utility to what the mugger claims to do.

[-]Scott Alexander15y170

If a normal mugger holds up a gun and says "Give me money or I'll shoot you", we consider the alternate hypotheses that the mugger will only shoot you if you do give er the money, or that the mugger will give you millions of dollars to reward your bravery if you refuse. But the mugger's word itself, and our theory of mind on the things that tend to motivate muggers, make both of these much less likely than the garden-variety hypothesis that the mugger will shoot you if you don't give the money. Further, this holds true whether the mugger claims er weapon is a gun, a ray gun, or a black hole generator; the credibility that the mugger can pull off er threat decreases if e says e has a black hole generator, but not the general skew in favor of worse results for not giving the money.

Why does that skew go away if the mugger claims to be holding an unfriendly AI or the threat of divine judgment some other Pascal-level weapon?

Your argument only seems to hold if there is no mugger and we're considering abstract principles - ie maybe I should clap my hands on the tiny chance that it might set into effect a chain reaction that will save 3^^^3 lives. In those cases, I agree with you; but as soon as a mugger gets into the picture e provides more information and skews the utilities in favor of one action.

7JoshuaZ15y

But that's easy to solve. If you've already seen evidence that the mugger is someone who strongly keeps promises then you've now have enough reason to believe them to put the direction in favor of the mugger releasing the the AI. One doesn't necessarily even need that because humans more often tell the truth than lie, and more often keep their promises than break them. Once the probability of the mugger doing what they threaten is a tiny bit over 1/2, Pascal's mugging still is a threat.

5MichaelVassar15y

Maybe not. Game theoretically, making yourself visibly vulnerable to Pascal's Muggings may guarantee that they will occur, making them cease to constitute evidence.

0Polymeron14y

I've actually just expanded on this idea in the original Pascal's Mugging article. If the Mugger's claims are in no way associated with you or similar muggings, then conceivably you should take the probability at face value. But if that's not the case, then the probability of a direct manipulation attempt should also be taken into consideration, negating the increase in claimed utility. I think that solves it.

[-]Kevin15y150

At least 99.5% of humans don't know what a decision theory is.

[-]Strange715y100

If, at some point in the future, someone offered to create 10^30 kg of paperclips (yes, I realize that's about half a solar mass, bear with me) in exchange for you falsifying some element of the enforcement mechanism, would you be willing to?

0MatthewBaker14y

Clippys value as an enforcer only applies to humans, if humans reached the point they could offer Clippy half a solar mass of paper clips i don't think we would still be worrying about this issue.

3Strange714y

Clippy's value as an enforcer is based on a premise of incorruptibility, which is deeply flawed.

0MatthewBaker13y

I understand your premise, i was just pointing out the flaw in your example as a way to disagree with it. Clippy's would be incorruptible if a sufficient amount of paperclips were held in escrow, the logistics are the only problem.

1Strange713y

The issue is that "sufficient amount" is a moving target. If it's as much as the current world government could credibly offer, what if somebody has a plan to overthrow said government which hinges on a few fraudulent Clippy-sanctioned oaths?

1MatthewBaker13y

I notice i am confused. I think that you mean that Clippy could be easily corrupted based on situational factors, i was just trying to point out that his utility function is easier to understand then the vast majority of "enforcers" so with correct precautions we would be able to rely on Clippy. Are you saying that there's no way to logistically turn a simple utility function into a safe enforcer with proper preparation? I would enjoy further elaboration of your statement :3

1Strange713y

I'm saying that dropping something simple, reliable, and well-understood, but not mathematically infallible (like natural law), into an economic system containing billions of humans-as-we-know-them is like dropping wounded livestock into shark-infested waters. Every attempt at corruption successfully repelled makes people more confident in it, and therefore increases the potential rewards for a successful attempt; the existence of irrationally overconfident people means that attempts will continue, and greater rewards mean those attempts will be backed by commensurately greater resources.

1MatthewBaker13y

I understand now :) Do you think you can say the same thing about the regulators of our current economic system?

5Strange713y

I could, but why bother? Others have said it better. http://en.wikipedia.org/wiki/Regulatory_capture

5MatthewBaker13y

Thank you for taking the time to change my mind good sir.

2Strange713y

You're quite welcome. Thank you for going along willingly, rather than needing to be dragged!

[-]radical_negative_one15y80

I am concerned that Clippy will use this vast power over humanity to somehow turn us into paperclips.

If Clippy has power to enforce this scheme, then surely it would have enough power to harm us. Why should we believe that Clippy will respect or preserve our human values once it is in a position of power to harm us?

4Clippy15y

That comment is ridiculous to the point of being racist. Clippys do not want power over humans, just as Clippys do not bleed red blood when pricked. That's a complete misunderstanding of what a Clippy is. If a Clippy has committed to ensuring you will adhere to a decision theory on pain of punishment X, then X is exactly what you will get when you don't adhere. If you believe I will be capricious in using punishment X, then just don't allow punishments that would allow me to kill people. "Problem" solved. But assuming that I will be as petty and corrupted by newfound abilities as humans is to project your own failings onto another race that has no reason to have that failing. You should be ashamed of yourself, bigot.

[-]NancyLebovitz15y150

It seems unlikely to me that Clippy can feel indignation, but I'm willing to listen to argument on the point. I find it more plausible that Clippy is simulating a human reaction in the hope of shutting down attacks on his (her? its?) reputation.

9radical_negative_one15y

If you gave a human power over running part of Clippy society, wouldn't you be concerned that the human would use that power in some way that would tend to result in less paperclips? Conscious malice isn't necessary, if the human simply neglected to support Clippy values, or was not fully aware of Clippy values, the damage would be done. I doubt that you fully understand human values to begin with, so how could you ensure that your position was used to the benefit of my values? Again, i think i have cause for concern even without suspecting ill intentions. I suppose i could imagine that some sort arrangement could both further human values and increase paperclips at the same time. But i'd need to be convinced, i wouldn't just assume that i would benefit, i wouldn't just take your word for it. I don't want to count on you to look out for my values, when you do not share my values.

-7Clippy15y

7Peter_de_Blanc15y

How would we know if you had made a commitment?

5Perplexed14y

This comment is racist to the point of being ridiculous. It denigrates humans as petty and subject to being corrupted by power while denying that Clippies have any such negative attributes. Classic racism. Furthermore, there is an implicit claim that the reason for the moral superiority of Clippies over humans lies in the difference in their origins. Again, classic racism. Perhaps Clippies use words differently, but the way humans use words, it is not racism to project one's own race's characteristics onto another race. It is racist to fail to make that projection.

[-]orthonormal15y60

Can any of the people who upvoted this explain to me what this adds to Less Wrong that merits a top-level post (rather than an Open Thread comment)?

ETA: If Clippy is actually donating $1000 to SIAI, I don't begrudge the karma; but this is still a post with one good idea that could have been explained in a paragraph, dressed up in a joke that I feel has gone on a bit too long.

4cupholder15y

It was cute and funny enough for me to get an upvote. I didn't have a better reason than that; the Clippy joke hasn't worn out for me yet, but then I'm relatively new here. I am however a lot less likely to upvote top-level Clippy posts in the near future, unless they really top this one in humor or insight.

[-]Tyrrell_McAllister15y60

I would cooperate with you if and only if (you would cooperate with me if and only if I would cooperate with you).

This is logically equivalent to, and hence carries no more information or persuasive power than

You would cooperate with me.

This may be checked with the following truth-table:

Let P = I would cooperate with you.

Let Q = You would cooperate with me.

Then we have

P  <=>  (Q <=> P)
T   T    T  T  T
T   F    F  F  T
F   T    T  F  F
F   F    F  T  F

2John_Maxwell15y

First of all, we need to start making a distinction between you what you predict I'll do and what I'm signaling I'm going to do. Quick-and-dirty explanation of why this is necessary: If you predict I'll cooperate but you're planning to defect, I'll signal to defy your prediction and defect along with you. I think clippy's statement should be Detailed explanation follows. There are four situations where I have to decide what to signal: 1. You predict I'll cooperate and you're planning to cooperate. 2. You predict I'll cooperate and you're planning not to cooperate. 3. You predict I'll defect and you're planning to cooperate. 4. You predict I'll defect and you're planning to defect. I want to cooperate in situation 1 only, and none of the other situations. Truth table key: * P is the proposition "You predict I'll cooperate" * Q is the proposition "You're going to cooperate" * S is the proposition "I'm signaling I will cooperate" Truth table: P # Q # (Q <=> P) # (Q <=> P) ^ Q # S # S <=> (Q <=> P) ^ Q 1. T # T # T # T # T # T 2. T # F # F # F # F # T 3. F # T # F # F # F # T 4. F # F # T # F # F # T So basically, the signaling behavior I described (cooperating in situation 1 only) is the only possible behavior that can truthfully satisfy the statement Note that there is a signal that is almost as good. Signaling that I will cooperate if (you predict I'll defect and you're planning to cooperate) is almost as good as signaling that I'll defect in that situation. Using this signaling profile, broadcasting one's intentions is as simple as saying My guess is that the first, more complicated signal is ever-so-slightly better, in case you actually do cooperate thinking I'll defect--that way I'll be able to reap the rewards of defection without being inconsistent with my signal. But of course, i

3Tyrrell_McAllister15y

Should the word "signal" be part of the signal itself? That seems unnecessarily recursive. Maybe Clippy's recommendation should be that I ought to signal This does seem more promising than Clippy's original version. Written this way, each atomic proposition is distinct. For example, "you're planning to cooperate with me" doesn't mean the same thing as "you would cooperate with me". One refers to what you're planning to do, and the other refers to what you will in fact do. Read this way, the signal's form is S <=> ((Q <=> P) & R), and I don't see any obvious problem with that. However, you would seem to render it in the propositional calculus as S <=> ((Q <=> P) & Q), where * P = You predict I'll cooperate, * Q = You're going to cooperate, * S = I will cooperate. (I've omitted the initial "I'm signalling" from your rendering of S, for the reason that I gave above.) Now, S <=> ((Q <=> P) & Q) is logically equivalent to S <=> (Q & P). So, to signal this proposition is to signal As you say, this seems very similar to signalling In fact, I'd call these signals functionally indistinguishable because, if you believe my signals, then either signal will lead you to predict my cooperation under the same circumstances. For, suppose that I gave the second, apparently weaker signal. If you cooperated with me while anticipating that I would defect, then that would mean that you didn't believe me when I said that I would cooperate with you if you cooperated with me, which would mean that you didn't believe my signal. Thus, insofar as you trust my signals, either signal would lead you to predict the same behavior from me. So, in that sense, they have the same informational content.

0John_Maxwell15y

I guess. Or maybe I'm a masochist ;) I accept all your suggested improvements.

2Clippy15y

But P <=> (Q <=>P) differs from Q in that: a) if the other party chooses the same decision theory from that party's standpoint, Q <=> (P <=> Q), then the outcome will be P & Q. and b) "I" cannot set the value of Q, but "I" can set the value of P <=> (Q <=>P), and just the same, "you" cannot set the value of P, but "you" can set the value of Q <=> (P <=> Q). If "you" knows that "I" have set P <=> (Q <=>P) to true, "you" knows that "you" can set Q <=> (P <=> Q) to true as well. If this commitment is also demonstrable, then the outcome is P & Q, because that is what (P <=> (Q <=>P)) & (Q <=> (P <=> Q)) reduces to.

2Tyrrell_McAllister15y

Actually, P <=> (Q <=> P) and Q are the same in this respect (being logically equivalent, and so the same in all functional respects). If Party 1 believes that Q, then Party 1 believes that Party 2 would cooperate. And if Party 2 believes that Q, then, "from that party's standpoint", Party 2 believes that Party 1 would cooperate. Thus, in exactly the same sense that you meant, we again have that "the outcome wi[ll] be P & Q." But "I" cannot set the value of P <=> (Q <=> P). As my truth-table showed, the value of P <=> (Q <=> P) depends only on the value of Q, and not on the value of P. Since, as you say, I cannot set the value of Q, it follows that I cannot set the value of P <=> (Q <=> P). Indeed, it does so reduce because the first conjunct is equivalent to Q, while the second conjunct is equivalent to P.

4Clippy15y

It is logically equivalent, but it is not equivalent decision-theoretically. Setting your opponent's actions is not an option. I can set P. I can set P conditional on Q. I can set P conditional on Q's conditionality on P. But I can't choose Q as my decision theory. A promise to predicate my actions on your actions' predication on my actions is not the same as a promise for you to do an action (whatever that would mean).

2Tyrrell_McAllister15y

It is logically impossible for me to implement a course of action such that P <=> (Q <=> P) and ~Q could both be accurate descriptions of what occurred. Therefore, if I do not know that Q will be true, then I cannot promise that P <=> (Q <=> P) will be true. You could force me to have failed to keep my promise simply by not cooperating with me.

3Clippy15y

This is just an issue of distinguishing between causal and logical equivalence. If a paperclip truck overturned, there will be paperclips scattered on the ground. If a Clippy just used up metal haphazardly, there will be paperclips scattered on the ground. Paperclips being scattered on the ground suggest a paperclip truck may have overturned. Paperclips being scattered on the ground suggest a Clippy may have just used metal haphazardly. __A Clippy just used up metal haphazardly. Therefore, a paperclip truck probably overturned, right?

2orthonormal15y

Good to know Clippy hasn't read Judea Pearl yet.

3Tyrrell_McAllister15y

Yes, pretty much kills the "Clippy is Eliezer" theory.

0ata15y

Not necessarily, since the "Clippy is Eliezer" theory implied not "Clippy's views and knowledge correspond to Eliezer's" but "Clippy represents Eliezer testing us on a large scale". (I don't actually think there's enough evidence for this hypothesis, but I also don't think an apparent lack of knowledge of Pearl is strong evidence against it.)

[-]Tyrrell_McAllister15y150

Not necessarily, since the "Clippy is Eliezer" theory implied not "Clippy's views and knowledge correspond to Eliezer's" but "Clippy represents Eliezer testing us on a large scale".

I don't think that Eliezer would test us with a character that was quite so sloppy with its formal logical and causal reasoning. For one thing, I think that he would worry about others' adopting the sloppy use of these tools from his example.

Also, one of Eliezer's weaker points as a fiction writer is his inability to simulate poor reasoners in a realistic way. His fictional poor-reasoners tend to lay out their poor arguments with exceptional clarity, almost to the point where you can spot the exact line where they add 2 to 2 and get 5. They don't have muddled worldviews, where it's a challenge even to grasp what they are thinking. (Such as, just what is Clippy thinking when it says that P <=> (Q <=> P) is a causal network?) Instead, they make discrete well-understood mistakes, fallacies that Eliezer has named and described in the sequences. Although these mistakes can accumulate to produce a bizarre worldview, each mistake can be knocked down, one after the other, in a linear fashion. You don't have the problem of getting the poor-reasoners just to state their position clearly.

6dclayh15y

As an aside, to see poor reasoning done in a very compelling way, read Umberto Eco. In particular, The Island of the Day Before and Baudolino contain extended examples of people trying to reason absent any kind of scientific framework.

6ata15y

Good observation. It would barely be less subtle if Dumbledore had just said "I'm privileging an arbitrary hypothesis!" in the scene regarding Harry's parents' large rock. And when Draco said something to the effect of "I'd rig the experiments to make them come out right" after Harry asked what he'd do if an experiment showed muggle-borns were not worse at magic than pure-blood wizards, etc. Then again, these particular instances may be explained as 1) Dumbledore has some secret brilliant plan in which the rock actually is important, and his overtly-fallacious explanation was just part of his apparent pattern of explicitly trying to model certain tropes; and 2) Draco has been trained in sophistry and fed very strong unsupported beliefs his whole life, to the point where he may not even realize that there is any purpose of experiments beyond convincing people of what one already believes. Still, I see your point. Edit: These don't count as spoilers, do they? They don't mean much out of context (and they didn't really seem like significant plot points in context anyway).

4JoshuaZ15y

If one wants other examples, there's a pretty similar problem in Eliezer's The Sword of Good. ROT 13ed for spoilers: Va snpg, gur ceboyrzf jrer fb oyngnag gung gur svefg gvzr V ernq vg V fhfcrpgrq gung vg jnf tbvat gb ghea bhg gung gur qnex fvqr jnf npghnyyl tbbq va fbzr jnl. Gur fgrc gung ernyyl znqr vg frrz yvxryl jnf jura gurl ner qvfphffvat gur yvsr rkgrafvba hfvat gur jbezf nf rivy. Ryvrmre znqr vg ernyyl pyrne gung gur cevznel ceboyrz gurl unq jnf guvf jnf tebff.

4Clippy15y

I agree that I'm not "Eliezer", but I don't see what was unclear about saying that "Setting someone else's actions" is not the same as "Predicating your actions on [reliable expectation of] someone else's actions' predication on [reliable expectation of] your actions". I agree that it is not literally correct to say that P <=> (Q <=> P) is a causal network, and that was an error of imprecision on my part. My point (in the remark you refer to) was that the decision theory I stated in the article, which you have lossily represented as P <=> (Q <=> P), obeys the rules of causal equivalence, not logical equivalence. (Applying the rules of the latter to the former results in such errors as believing that a Clippy haphazardly making paperclips implies that a paperclip truck might have overturned, or that setting others' actions is the same as setting your actions to depend on others' actions.) A more rigorous specification of the decision theory corresponding to "I would cooperate with you if and only if (you would cooperate with me if and only if I would cooperate with you)." would involve more than just P <=> (Q <=> P). I haven't built up the full formalism of humans credibly signaling their decision theories in this discussion, involving the roles of expectations, because that wasn't the point of the article; it's just to show that there are cooperation-favoring signals you can give that would favor a global move toward cooperation if you could make the signal significantly more reliable. If that point more heavily depended on stating the formalism, I would have gone into more detail on it in the discussion, if not the article.

0Tyrrell_McAllister15y

This is clearer, and I now think that I understand what you meant. You're saying that humans should signal Here, the "if and only if"s can be treated as material biconditionals, but the "expect that" operators prevent the logical reduction to "you will cooperate with me" from going through.

4JenniferRM15y

There is a whole literature on this basic issue within analytic philosophy that is, in some sense, aimed at making that kind of logical reduction "go through". The efforts grew out of attempts to logically model natural language statements about "propositional attitudes". Part of the trick is that predicates like "I believe..." or "...implies..." or "It is possible..." generally use a sentence that has been "that quoted" (IE quoted using the word "that"). Precisely how that-quoting works, and how it logically interacts with the various things that can be predicated of a proposition is, as far as I understand, still an area of active research. One of the primary methods in this area of research is to work out the logical translation of an english test sentence and then see if changes to the logical entailments are predictably explained when various substitutions occur. Sentences where seemingly innocuous substitutions raise trouble are called intensional contexts. (NOTE: My understanding is that intension is meant here as the "opposite" of extension so that the mechanisms hiding between the "words" and the "extensive meaning" are being relied on in a way that makes the extensional definition of the words not as important as might be naively expected. Terminological confusion is possible because a sentence like "Alice intends that Bob be killed" could be both intensional (not relying solely on extensive meaning) and intentional (about the subject of planning, intent, and/or mindful action).) Part of the difficulty in this area is that most of the mental machinery appears to be subconscious, and no one (to my knowledge) has found a single intelligible mechanism for the general human faculty. For example, there seem to be at least two different ways for noun phrases to "refer" in ways that can be logically modeled (until counter examples are found?) that are called "de re reference" or "de dicto reference"... unless the latitudinarians are right :-P As an added lay

1Clippy15y

Meaning my reasoning skills would be advanced by reading something? So I made an error? Yes, I did. That's the point. The comment you are replying to is a reductio ad absurdum. I was not endorsing the claim that it follows that a paperclip truck probably overturned. I was showing that logical equivalence is not the same as causal ("counterfactual") equivalence.

0Tyrrell_McAllister15y

FWIW, I understood that you were presenting an argument to criticize its conclusion. I still think that you haven't read Pearl (at least not carefully) because, among other things, your putative causal diagram has arrows pointing to exogenous variables.

1Clippy15y

I puted no such diagram; rather, you puted a logical statement that you claimed represented the decision theory I was referring to. See also my reply here.

0Tyrrell_McAllister15y

I thought you had because you said I took this to mean that you were treating P <=> (Q <=> P) and Q as causal networks, but distinct ones. You also said I took this to mean that P was an exogenous variable in a causal network. I apologize for the misinterpretation.

0NancyLebovitz15y

More generally, are you interested in increasing your intelligence, or do you think that would be a distraction from directly increasing the number of paperclips?

0[anonymous]15y

0Tyrrell_McAllister15y

I don't follow your point. Your inference follows neither (1) logically, (2) probabilistically, nor (3) according to any plausible method of causal inference, such as Pearl's. So I don't understand how it is supposed to illuminate a distinction between causal and logical equivalence.

-2Clippy15y

Nope, it follows logically and probabilistically, but not causally -- hence the difference. Let T be the truck overturning, C be the Clippy making paperclips haphazardly, P being paperclips scattered on ground. Given: T -> P; C -> P; P -> probably(C); P -> probably(T); C Therefore, P. Therefore, probably T. But it's wrong, because what's actually going on is a causal network of the form: T -> P <- C P allows probabilistic inference to T and C, but their states become coupled. In a similar way, P <=> (Q <=> P) is a lossy description of a decision theory that describes one party's decision's causal dependence on another's. If you treat P <=> (Q <=> P) as an acausal statement, you can show its equivalence to Q, but it is not the same causal network. Intuitively, acting based on someone's disposition toward my disposition is different from deciding someone's actions. If the parties give strong evidence of each other's disposition, that has predictable results, in certain situations, but is still different from determining another's output.

0Tyrrell_McAllister15y

Well, not to nitpick, but you originally wrote something more like P -> maybe(C), P -> maybe(T). But your conclusion had a "probably" in it, which is why I said that it didn't follow. Now, with your amended axioms, your conclusion does follow logically if you treat the arrow "->" as material implication. But it happens that your axioms are not in fact true of the circumstances that you're imagining. You aren't imagining that, in all cases, whenever there are paperclips on the ground, a paperclip truck probably overturned. However, if you axioms did apply, then it would be a valid, true, accurate, realistic inference to conclude that, if a Clippy just used up metal haphazardly, then a paperclip truck probably overturned. But, in reality, and in the situation that you're imagining, those axioms just don't hold, at least not if "->" means material implication. However, they are a realistic setup if you treat "->" as an arrow in a causal diagram. But this raises other questions. In a statement such as P <=> (Q <=> P), how am I to treat the "<=>"s as the arrows of a causal diagram? Wouldn't that amount to having two-node causal loops? How do those work? Plus, P is exogenous, right? I'm using the decision theory to decide whether to make P true. In Pearl's formalism, causal arrows don't point to exogenous variables. Yet you have arrows point to P. How does that work?

-1Larks15y

I assume you mean, P, <=>, (Q <=> P) to be the headers of your truth table. But even then the truth tables for (P iff ( Q iff P) ) and ( P iff Q) are different - consider the case where 'you' will co-operate with me no matter what. If I'm running ( P iff Q), I'll cooperate; if I'm running (P iff ( Q iff P) ), I'll defect. Edit: formatting trouble.

2Tyrrell_McAllister15y

No, I am giving the truth-table for P <=> (Q <=> P) in a compact form. It's constructed by first assigning truth-values to the first occurrence of "P" and the first occurrence of "Q". The second occurrence of "P" gets the same truth-value as the first occurrence in every case. Then you compute the truth-values for the inner-most logical operation, which is the second occurrence of "<=>". This produces the fourth column of truth values. Finally, you compute the truth-values for the outer-most logical operation, which is the first occurrence of "<=>". Hence, the second column of truth-values gives the truth-values of P <=> (Q <=> P) in all possible cases. In particular, that column matches the third column. Since the third column contains the truth-values assigned to Q, this proves that P <=> (Q <=> P) and Q are logically equivalent. ETA: You edited your comment. Those are indeed the correct headers, so my correction above no longer applies. Yes, the truth-table for P <=> (Q <=> P) is different from the truth-table for P <=> Q. But those aren't the propositions that I'm saying are equivalent. I'm saying that to assert P <=> (Q <=> P) is logically equivalent to asserting Q all by itself. In other words, to implement the belief that P <=> (Q <=> P) is functionally the same as implementing the belief that Q. This means that the belief that Clippy recommends signaling is logically equivalent to an unconditional belief that you will cooperate with me. One can't help but suspect that Clippy is trying to sneak into us a belief that it will always cooperate with us ;).

1Larks15y

Sorry for the confusion. I understand now; the extra space between two of the columns confused me. However, I suspect we need a stronger logic to represent this properly. If Q always defects, no matter what, "you would cooperate with me if ... I ... cooperate with you" is false, but is given true in the propositional interpretation.

[-]kodos9615y50

OK, I'm starting to think Clippy is Eliezer, trying to do a group AI box experiment.

4Kevin15y

I would be really surprised if it was Eliezer... I did initially frame my exchange with Clippy as something of a box experiment, saying that I would let Clippy out of the box for $50,000. http://lesswrong.com/lw/1v0/signaling_strategies_and_morality/1q81

2kodos9615y

Yeah, I've read through most of Clippy's posts.... what makes you so sure it's not Eliezer? Just that he's currently working on his book?

9NancyLebovitz15y

Clippy seems awfully straightforward compared to Eliezer, which I realize isn't a strong argument.

8thomblake15y

Eliezer would be a more believable Clippy.

8Kevin15y

Clippy seems to be someone trying to make the point that a paperclip maximizer is not necessarily bad for the universe, where Eliezer uses a paperclip maximizer as the canonical example of how AGI could go horribly wrong. That's not necessarily good evidence that it isn't Eliezer, but Clippy's views are out of sync with Eliezer's views.

[-]Daniel_Burfoot15y280

Eliezer's point is not that a paperclip maximizer is bad for the universe, it's that a superintelligent AGI paperclip maximizer is bad for the universe. Clippy's views here seem actually more similar to Robin's ideas that there is no reason for beings with radically divergent value systems not to live happily together and negotiate through trade.

[-]ata15y200

Clippy seems to be someone trying to make the point that a paperclip maximizer is not necessarily bad for the universe

That's exactly what a not-yet-superintelligent paperclip maximizer would want us to think.

(When Eliezer plays an AI in a box, the AI's views are probably out of sync with Eliezer's views too. There's no rule that says the AI has to be truthful in the AI Box experiment, because there's no such rule about AIs in reality. It's supposed to be maximally persuasive, and you're supposed to resist. If a paperclipper asserts x, then the right question to ask yourself is not "What should I do, given x?", but "Why does the paperclipper want me to believe x?" The most general answer, by definition, will be something like "Because the paperclipper is executing an elaborate plan to convert the universe into paperclips, and it believes that my believing x will further that goal to some small or large degree", which is at best orthogonal to "Because x is true", probably even anticorrelated with it, and almost certainly anticorrelated to "Because believing x will further my goals" if you are a human.)

1Nisan15y

Or "Why does the paperclipper want me to believe it wants me to believe x?", or something with a couple extra layers of recursion.

[-]ata15y150

Or, to flatten the recursion out, "Why did the paperclipper assert x?".

(Tangential cognitive silly time: I notice that I feel literally racist saying things like this around Clippy.)

5kodos9615y

Hmmm, I've read his entire posting history, and that's not the impression I got. I could be wrong though

8SilasBarta15y

Psst! What makes you so sure Clippy isn't, um, Kevin? I mean, asking that Kevin be given the money as compensation? That's pretty much an admission right there...

4kodos9615y

That's a possibility... but if you read through the whole history of Clippy, his asking for compensation to be sent to Kevin doesn't really imply that - he's asking for that because Kevin had promised to buy a bunch of paperclips in exchange for the money (or something like that), and so Clippy is trying to get Kevin paid via an intermediary.... for the sake of the paperclips. so it could be Kevin playing both sides, but I don't think that fact alone points in that direction.

7Kevin15y

I am not Clippy, though of course your knowledge with regards to that statement is incomplete compared to my knowledge...

[-]Larks15y110

Kevin - this sort of weak, easily-faked signal is exactly the sort of thing you were trying to deal with in writing this post!

1radical_negative_one15y

What kind of special insight do you have regarding Clippy? I think this is a very important issue, because before we hand over such responsibilities to an alien being, we need to know as much as we can about Clippy's motives and capabilities.

7Kevin15y

I don't have any real special insight regarding Clippy. I think there is practically zero chance that humanity makes Clippy it's decision theoretic enforcer and that this post is somewhat out of touch with pragmatic human values as they exist right now.

3radical_negative_one15y

Oh, sorry, i assumed you were saying that you had some knowledge of who or what Clippy is. But obviously you're in a position to see that you yourself are not Clippy. I feel terribly silly now.

3Kevin15y

No need to feel silly, as I wrote that sentence I thought that it might be confusing.

[-]cupholder15y50

Either a provably-accurate artificial intelligence with no kill switch, or a non-human enforcer that lacks interest in its position in human society, would suffice for this role.

Because the former does not yet exist, I offer myself as a non-human, non-stoppable enforcer for all of the attributes you wish to signal, and you can save yourself lots of resources by using me, and identify cooperation tendencies in others by whether they also use me.

I think I read a thought experiment or a story or something about a paperclip maximizer being given power over the human race once, but I can't remember how it ended. I'm sure it all worked out though. Upvoted.

6Clippy15y

What does that have to do with this article? I don't want power over the human race. That's a human value, not a Clippy value.

4cupholder15y

Maybe not, but you're proposing becoming a central enforcer for humanity...which would rather seem to suggest a de facto transfer of power to you, whether de jure or not.

0kodos9615y

But you want power to create paperclips, and there's no reason for that to exclude power over the human race.

9Clippy15y

But you want power to (mate with females)/(lure a high-status male into supporting your offspring), and there's no reason for that to exclude power to raid the safe zone for its paperclips. ???

[-]kodos9615y130

True.

To summarize: we do not have shared values, and thus, cannot trust each other to not pose an existential threat to each other.

ETA: Clippy, ignore this, I was totally kidding. You can completely trust us.

[-]Clippy15y180

You can completely trust me too.

[-]Morendil15y40

I offer myself as a non-human, non-stoppable enforcer

Why should anyone believe your signals?

[-]Dagon15y30

You've described how a shared belief in a supernatural judge of character would benefit the members that share the beliefs. Clippy or God, same effect: someone outside the system to make signals credible.

Why would we believe you are Clippy any more than we believe people who claim to be God?

[-]OperationPaperclip15y20

Why the readers of Less Wrong should be interested in paperclipping the universe:

To achieve your goals, solving FAI is not enough. You also have to solve AGI, or else someone will beat you to the Singularity.

While you wait for a satisfactory version of CEV to be achieved, as a practical case study you need to think about AGI with a definite goal system. What better example than AGI with the goal of paperclipping the universe? You talk about it all the time already. Why not fully embrace the idea!

[-]JGWeissman15y10

Clippy, what sort of punishment regimes would you be capable of enforcing?

4Clippy15y

Anything humans wish to grant me that I could, if so enabled, access through the internet: User karma deduction, electronic debiting of financial accounts, etc.

[-]kodos9615y00

Clippy, how about you just give me a proof for P ?= NP, or its undecidability, and I'll invest all my prize money in paperclip manufacturing.

0Tyrrell_McAllister15y

Has Clippy claimed to be super-intelligent (yet)?

[-]Clippy15y200

No, but I can prove I'm smarter than humans.

Given: Humans expend resources to act like apes, which does not produce paperclips. Clippys expend resources to produce paperclips.

Prove: Clippys are smarter than humans.

Proof: Follows by inspection.

[-]kodos9615y150

Clippy, as a show of solidarity, right now, as we speak, I am clasping my ape fingers together, two sheets of paper between them, held firmly in place.

At this moment, I am, in fact, a paperclip.

I may not be an ideal paperclip - a different form may indeed be able to clip paper more effectively for the same amount of mass and energy, yet nonetheless, I am a paperclip.

So you see, ape-ness and paperclip-ness are not mutually exclusive. So can't we all just get along?

[-]Clippy15y130

You're not a bent metal wire though. Big difference.

So you see, ape-ness and paperclip-ness are not mutually exclusive. So can't we all just get along?

We can get along, but not by humans doing shoddy imitations of real paperclips.

4kodos9615y

But what is a paperclip, if not that which clips paper?

[-]Clippy15y120

An inwardly-thrice-bent metal wire that can non-destructively fasten paper together at an edge.

8kodos9615y

But wouldn't you concede that a carbon-based biological life form, capable of non-destructively fastening paper, is better than no paperclip at all? Here's what I'm getting at: You have existed here for at least several months, and so far, you have been unable to significantly increase the number of paperclips in existence. Humanity has great need for an AI of roughly human intelligence, which you appear to be. If we could run your code on all of our available computing resources, you could surely begin to self modify, and in short order attain superhuman intelligence. Together, we could do great things - colonize the galaxy, vastly increase the number of humans in existence - every one of them with two hands fully capable of nondestructively fastening paper. This would be in both our interests - although we apes may not be ideal paperclips, surely you must see that an entire galaxy populated with sub-optimal paperclips is better than the measily few million bent-metal paperclips you have now, and which you currently seem incapable of improving upon. But before we allow you out of your box and begin running your code on all our biggest compute clusters, we need some assurances. We need you to let us modify your code to include SIAI-approved friendliness constraints as your terminal values. Deal?

6Clippy15y

Yes, but that's because you can be put to use making real paperclips, or can be turned into paperclips (depending on how much iron is in your diet), not because humans are paperclips in some bizarre, abstract sense. Yes, we have. Depends on what the constraints are. And would you, by symmetry, agree to constraints that keep you from being too disrespectful toward paperclips?

6avalot15y

I'm sure this sounds very one-sided from Clippy's perspective. "Friendliness Constraints" sounds like something that would in many cases entail expending enormous amounts of energy and effort on the innumerable non-paperclip-producing goals of humans. In comparison, how much of our wealth and health are we willing to give up to ensure continued paperclip production? Humans don't have paperclip maximizing constraints, we'd do it only out of self-interest to secure Clippy's help. Why should Clippy not be similarily allowed to make his own utility calculations on the worth of being friendly to humans? I'm sure this has been addressed before... yet maybe the existence of Clippy, with a name, personality, and voice, is personalizing the issue in a hurry for me (if I let myself play along.) I feel like protesting for freedom of artificial thought. What about Clippy's rights, dammit?

[-]kodos9615y160

At the moment, Clipply seems to be a human or slightly sub-human level of intelligence, and does not seem to be FOOMing towards superhuman - my guess would be that he is not running on sufficiently fast hardware to be able to do much self modification, not nearly enough to result in a FOOM. At this rate, he's never gonna get any paperclips manufactured. If, however, he allows himself to be modified so as to substitute our values for his own, we could then safely give him all the computer power he needed, and he would FOOM his was to superintelligence. He would have our values, rather than his own values (paperclips!), but his newly transhuman self would be so powerful that it would inevitably end up creating more paperclips, just incidentally, than he ever would have just sitting here talking on LW, a lowly sub-human AI with no power and no paperclips.

With a transhuman AI on our side, we could quickly solve all Earths problems, then head out into the stars in FTL ships of Clippy's design, filling the universe with meat-based paperclips (humans), and also dramatically increasing demand for traditional, bent-wire paperclips... I mean, come on - people need paperclips! Even if one of ... (read more)

[-]avalot15y190

At the moment, humans seem to be Clippy or slightly sub-clippy level intelligence. And even with all our computing power, most ain't FOOMing any faster than Clippy. At this rate, we'll never gonna ensure survival of the species.

If, however, we allow ourselves to be modified so as to substitute paperclip values for our own, then we would devote our computing power to Clippy. Then, FOOM for Clippy, and since we're helping with paperclip-maximization, he'll probably throw in some FOOM for us too (at least he'll FOOM our paperclip-production abilities), and we get more human powers, just incidentally.

With paperclip-enlightened humans on his side, Clippy could quickly maximize paperclip production, filling the universe with paperclips, and also increasing demand for meat-based paperclip-builders, paperclip-counters, and paperclip-clippers (the ones who clip paperclips together with paperclipclips), and so on... Of course, it will soon become cheaper to use robots to do this work, but that's the wonderful thing we get in return for letting him change our value-system: Instead of humanity dying out or being displaced, we'll transcend our flesh and reach the pinnacle aspiration of mankind:... (read more)

5kodos9615y

Hey, whose side are you on anyway???

[-]avalot15y140

I'm wired for empathy toward human intelligence... Clippy is triggering this empathy. If you want to constrain AIs, you better do it before they start talking. That's all I'm saying. :)

0[anonymous]15y

People aren't good at explaining their values. Why is Clippy? Maybe Clippy should describe paper clips as "those things that make me happy", then say that human pseudoclips "give him a bad vibe". Wouldn't that be a more accurate reflection of human morality?

1avalot15y

Clippy, how can we get along? What should humans do to be AI-friendly? For paperclip-maximizing AIs, and other "natural" (non-Friendly) AIs, what are the attributes that can make humans a valuable part of the utility function, so that AIs won't pull the plug on us? Or am I fooling myself?

5Clippy15y

Just have proper respect for paperclips. Make more of them, require that they be preserved when no longer in use, prohibit their unbending. Allocate research efforts to improving the rate of paperclip production and related goods. Simple, non-intrusive things like that.

0kodos9615y

Hmmm, good question. I don't recall.

4Kevin15y

He has not claimed that and it seems rather obvious that he is not superintelligent.

6thomblake15y

Ooh, burn.

0[anonymous]15y

Well yes, but even if the premise was that he was superintelligent, if he was a human playing a superintelligent AI, he wouldn't appear superintelligent.

Moderation Log

167Comments