In my darker moments, I think that every human political tendency is just an instance of this very problem. Terry Pratchett (our most underrated explicator and critic of both Traditional Rationality and Bourgeois Morality) described it most pithily as "Wouldn't It Be Nice, If Everyone Was Nice". It's most obvious on the left, but can be seen on the right and libertarian tendencies as well.

[deleted]

Caledonian,

There was selection at the level of the group, according to the standard definitions. If you're going to make confrontational non-substantive comments on every post, please at least try to know what you're talking about.

"Sorry, you're not allowed to suggest ideas using that method" is not something you hear, under Traditional Rationality.

But it is a fact of life, ....

It is a fact of life that ....

I disagree. You list a whole collection of mistakes people make after they have a bad hypothesis that they're attached to. I say, the mistake should not be to use your prior experience when you come up with hypotheses. The mistakes are first to get too attached to one hypothesis, followed by the list of "facts of life" mistakes you then described.

People will ... (read more)

[deleted]

To get this whole line of reasoning off the ground, you need a decent way to rank phenomena in terms of how similar they are to us. Given this ranking, the warning is to beware of treating low ranked items like high ranked items. On AI, you need to give an argument why AI is a low ranked item, i.e., why AI is especially unlike us.

Now that you mention it, I'm actually not 100% sure that a paperclip maximizer wouldn't give humans some fraction of computing resources as some sort of very cheap game theory move.

Oh, and my ants say they're offended.

I guess "game theory move" doesn't make much sense; it should have read "given the possibility that it's being simulated".

Hardly any phenomena are like us, though. You can't hold a conversation with gentrification, or teach nitrogen narcosis to play piano.

It strikes me that if you want to rank phenomena as to how like us they are, you have a bunch of humans with gigantic numbers, and then chimps and chatbots rolling around at about .1, and then a bunch of numbers small enough you want scientific notation to express them.

@steven:

Do you devote a significant amount of your time and resources to making paperclips, given the possibility that you're being simulated? If not, why would a paperclip-maximizer devote time and resources to human life?

steven: your "not 100% sure" is a perfect example of the problem eliezer is trying to explain. "not 100% sure that X is false" is not a valid excuse to waste thought on X if the prior improbability of X is as incredibly tiny as it is for thoughts like "paperclip maximizers will find their own paperclip-related reasons not to murder everyone".

This posting, like many of Eliezer's, offers good advice for someone who is advancing the frontiers of human knowledge and breaking new ground. But most of us aren't in that situation. Do these considerations offer useful insights for the average person living his life? Or are they just abstract philosophy without practical import for most people?

I would say yes, the overcoming bias project is useful for laypeople--it's changed my life, at least. I don't read these posts as being about becoming a great scientist. I realize the irony of quoting "TMoLFAQ" in the comments to this post, but "[t]here's no such thing as science": rationality is about forming beliefs that are actually true, and plans that might actually work. Advancing the frontiers is a special case.

But it also seems too easy, as if you could take ten paper-clip maximizers competing to convert things into differently colored paperclips, and ended out with utopia.

No, let me guess, you'll be more productive if you're happy Deja vu.

Caledonian, HA has been discussing other superorganisms as possibly being conscious which Eliezer says evolution does not apply to (both could be right). What's your opinion on such matters?

No, let me guess, you'll be more productive if you're happy Deja vu.

Caledonian, HA has been discussing other superorganisms as possibly being conscious which Eliezer says evolution does not apply to (both could be right). What's your opinion on such matters?

No, let me guess, you'll be more productive if you're happy Deja vu.

Caledonian, HA has been discussing other superorganisms as possibly being conscious which Eliezer says evolution does not apply to (both could be right). What's your opinion on such matters?

A paperclip maximizer might keep humans around for a while (because, as of right now, we're the only beings around that make paperclips) but yeah, if it had enough power (magic nanotechnology, etc.), we'd most likely be gone.

It seems to me like the simplest way to solve friendliness is: "Ok AI, I'm friendly so do what I tell you to do and confirm with me before taking any action." It is much simpler to program a goal system that responds to direct commands than to somehow try to infuse 'friendliness' into the AI.

This has been addressed before. Basically, you'll get what you asked for, but it probably won't be what you really want.

Do you devote a significant amount of your time and resources to making paperclips, given the possibility that you're being simulated?

Keeping everyone alive would not take a significant amount of a paperclip maximizer's time and resources. (Though for utilitarians this probably means it doesn't count.) But the key difference is this: human-like goal systems seem like they will gain access to lots more simulation resources than paperclip maximizers or some other specific human-indifferent goal system (the set of all human-indifferent goal systems together is a different matter, but they're not a coherent bloc).

Caledonian, Those conditions weren't created in the laboratory, because the individual strategy dominated over the group; ergo, the conditions necessary for that to happen were not met.

I am not sure how to interpret this. When we remove whole groups from the gene pool because of some group characteristic (i.e. averaged over the population of that group), it sounds for me natural to call that a group selection. Do you have some different meaningful definition of group selection? What does it mean in general when you say that the individual strategy dominates?

It is much simpler to program a goal system that responds to direct commands than to somehow try to infuse 'friendliness' into the AI.

If the AI receives commands frequently it AI would be weak - and probably not very competitive. It would be like a child running to its mummy all the time. To make decisions fast, that sort of thing is not on the cards.

If the AI receives commands infrequently, that's more-or-less what is under discussion.

However, AIs can be expected to naturally defend their goals. It may be best not to provide a convenient interface fo... (read more)

I have a question for you Eliezer. When you were figuring out how powerful AIs made from silicon were likely to be, did you have a goal that you wanted? Do you want AI to be powerful so it can stop death?

What's your opinion on such matters?

[misrepresentation deleted] Evolution will continue. But as the substrate of human-memetic organisms is so flexible, this actually limits their ability to pass on data in a... (read more)

Eliezer: You've got various people insisting that an arbitrary mind, including an expected paperclip maximizer, would do various nice things or obey various comforting conditions: "Keep humans around, because diversity is important to creativity, and the humans will provide a different point of view." Now you might want to seriously ask if, even granting that premise, you'd be kept in a nice house with air conditioning; or kept in a tiny cell with life support tubes and regular electric shocks if you didn't generate enough interesting ideas th... (read more)

Saying that grey goo will spread, and then never change or create new forms, is as mistaken as saying that single-celled organisms should never have given rise to multi-cellular organisms because competition between individuals is so stringent.

One of the things that Eliezer doesn't grasp is that optimization is not something evolution has generally produced because optimization is often maladaptive. What would appear to be an ideal strategy in the short term fails in the long term because overall environmental conditions have a tendency to change. Biolog... (read more)

Yvain: It must have something to do with capitalism including a term for the human utility function in the form of demand

Why, yes, I do think that has something to do with why the market builds houses with air conditioning instead of tiny little cells.

Hal: Do these considerations offer useful insights for the average person living his life? Or are they just abstract philosophy without practical import for most people?

Well, this particular abstract philosophy could end up having a pretty large practical import for all people, if they end up reprocess... (read more)

[Deleted. Caledonian, whenever you say something poorly reasoned or that misrepresents others' arguments, I am going to treat it as deliberate trolling and excise it. I've seen you do better, just keep it consistent.]

I said: "The reason I got interested in UIVs to start with is that I didn't have a good way to decide what counted as a good outcome."

So I have realized that perhaps the best prophylactic against anthropic optimism bias is, in fact, to be genuinely unsure of what outcome you think is best. If you don't have a predetermined idea of what to argue in favor of, then you don't have a preferred outcome to argue in favor of. Admittedly this is not something that one can always do, but in the specific case of trying to work out what kind of goals to prog... (read more)

TGGP, please see ni.codem.us for a brief response to your question that Eliezer will not permit.

To return to the implications of things Eliezer has said recently, please consider this

"Stop! Stop right there! You defeated yourself the moment you knew what you wanted! You need to throw away your thoughts and start over with a neutral forward extrapolation, not seeking any particular outcome."

Eliezer, sharing causal parentage with us sounds like a plausible heuristic for ranking things in terms of similarity to us, but in many important senses an AI could share a great deal of causal parentage with us. So you still need a more detailed argument to rank AI low.

this particular abstract philosophy could end up having a pretty large practical import for all people

Eliezer:

Personally, I am not disputing the importance of friendliness. My question is, what do you think I should do about it?

If I were an AI expert, I would not be reading this blog since there is clearly very little technical content here.

My time would be simply too valuable to waste reading or writing popular futurism.

I certainly wouldn't post everyday, just to recapitulate the same material with minor variations (basically just killing time).

Of cour... (read more)

I second spindizzy, yet hope that something major is happening.

Eliezer, sharing causal parentage with us sounds like a plausible heuristic for ranking things in terms of similarity to us, but in many important senses an AI could share a great deal of causal parentage with us. So you still need a more detailed argument to rank AI low.

Which AI? A Friendly AI shares goal-systemic shape with us due to a direct causal link: humans successfully shaping the FAI. The law against deciding what you want, applies when you don't have control over the outcome - for intermediate cases, where you have partial control, you have t... (read more)

Perhaps we can use this defense of theory instinct as a simplified map of what we want the AI to do.

So create a paperclip maximizer that is (handwave) somehow restricted from doing anything that it's creator would try to convince people it would never do.

This is assuming that how we steer ourselves away from horrid ideas is simpler than how we decide we like something.

@Eliezer Yudkowsky said: Spindizzy and sophiesdad, I've spent quite a while ramming headlong into the problem of preventing the end of the world. Doing things the obvious way has a great deal to be said for it; but it's been slow going, and some help would be nice. Research help, in particular, seems to me to probably require someone to read all this stuff at the age of 15 and then study on their own for 7 years after that, so I figured I'd better get started on the writing now.

I have posted this before without answer, but I'll try again. You are working a... (read more)

James: See The Hidden Complexity of Wishes. You can't think of everything you'd need to ban.

sophiesdad: I don't see convincing anyone who matters in the government of the potential of AGI, let alone the need for Friendliness, as very likely. Tim Kyger (DOD employee) says "I don't know a soul in DoD or any of the services off the top of my head that has any inkling of the very existence of trans-H or of the various technical/scientific lanes of approach that are leading to a trans/post-human future of some sort. Zip. Zero. Nada." If you could get ... (read more)

Eliezer,

You should either: a) ban Caledonian; b) let him write whatever he wants.

Censoring his posts is kind of nasty, because it looks like he can only express opinions you think worth posting. Personally, I think you should choose (a), because his comments are boring, disruptive and useless, but if you don't wanna do it, then go for (b).

And as for this: "Research help, in particular, seems to me to probably require someone to read all this stuff at the age of 15 and then study on their own for 7 years after that, so I figured I'd better get started ... (read more)

It seems to me like the simplest way to solve friendliness is: "Ok AI, I'm friendly so do what I tell you to do and confirm with me before taking any action." It is much simpler to program a goal system that responds to direct commands than to somehow try to infuse 'friendliness' into the AI.

As was pointed out, this might not have the consequences one wants. However, even if that wasn't true, I'd still be leery of this option - this'd effectively be giving one human unlimited power. History has shown that people who are given unlimited power (or something close to it) tend to easily misuse it, even if they started out with good intentions.

I wanted to ban him, but other commenters requested that he be allowed to stay. So I haven't banned him, but I'm not going to let his trollings take over the comment threads either. Caledonian can write passable comments when he puts his mind to it; and if that's all that's allowed through, he has no motive to write anything else.

Whoever is censoring Caledonian: can it be done without adding the content-free nastiness (such as "bizarre objection", "illogic", and "gibberish")?

Pyramid Head:

There're plenty of smart guys out there, and if they have access to the proper literature, I'm sure you can find worthy contributors, instead of waiting 7 more years.

If you know of a good way to find such people beyond that what SIAI is already doing, do go ahead and tell us.

This idea may be contaminated by optimism, but to avoid the risk of destroying humanity with AI, would it not be sufficient to make the AI more or less impotent? If it were essentially a brain in a jar type of thing that showcased everything humanity could create in terms of intelligence without the disastrous options of writing its own code or having access to a factory for creating death-bots? I suppose this is also anthromorphizing the AI because if it were really that super-intelligent it could come up with a way to do its optimization beyond the constraints we think we are imposing. Surely building a tooth-less though possibly "un-Friendly" AI is a more attainable goal than building an unrestricted Friendly AI?

Constant: Done.

Boris: See That Alien Message.

As for the Manhattan Project, who do you think they're going to pick to lead it? Some young unknown with a mad brilliant idea? Or, say, Roger Schank? (I've got nothing against him personally, but he's pretty old-style.) Japan tried something like this with their Fifth Generation project. Didn't help them any.

When the basic theory is done, then I'll know if implementation requires a Manhattan Project or not. I don't think it will. AI done right is not about brute force.

Eliezer, I agree SF fiction writers find it far easier to just write AIs, and also true aliens, as just odd humans, as it is far more work to write a plausible intelligent non-human. I don't mean to be saying much more than the obvious point that an AI that was a "mind child" of human civilization would in that obvious sense share causal parentage with us. A whole brain emulation would have started out as a particular human brain, a hard coded AI would have started out as the sort of code a human would write, and an AI that evolved under pressu... (read more)

'It seems to me like the simplest way to solve friendliness is: "Ok AI, I'm friendly so do what I tell you to do and confirm with me before taking any action." It is much simpler to program a goal system that responds to direct commands than to somehow try to infuse 'friendliness' into the AI.'

As was pointed out, this might not have the consequences one wants. However, even if that wasn't true, I'd still be leery of this option - this'd effectively be giving one human unlimited power.

Would you expect all the AIs to work together under one person'... (read more)

Will Pearson: When you were figuring out how powerful AIs made from silicon were likely to be, did you have a goal that you wanted? Do you want AI to be powerful so it can stop death?

Eliezer: ..."Yes" on both counts....

I think you sidestepped the point as it related to your post. Are you're rationally taking into account the biasing effect your heartfelt hopes exert on the set of hypotheses raised to your concious attention as you conspire to save the world?

Are you're rationally taking into account the biasing effect your heartfelt hopes exert on the set of hypotheses raised to your concious attention as you conspire to save the world?

There's really no way to eliminate one's own biases without recourse to objective and impersonal tests, of course. You can't identify your own mistakes with armchair theorizing. But you have to give him credit for effort.

Are you're rationally taking into account the biasing effect your heartfelt hopes exert on the set of hypotheses raised to your conscious attention as you conspire to save the world?

Recovering, in instances like these, reversed stupidity is not intelligence; you cannot say, "I wish fast takeoff to be possible, therefore it is not".

Indeed. But you can, for example, say "I wish fast takeoff to be possible, so should be less impressed, all else equal, by the number of hypothesis I can think of that happen to support it".

Do you wish fast ... (read more)

RI, the point is not in consciously reevaluating the hypotheses to oppose the bias (=reverse stupidity), but in avoiding the wrong conclusion reaching the level of awareness, in repairing the outlook before it misguides intuition. Intuition is relatively blind, but it is the engine of human intelligence. Feed it with misinterpreted information, and it will turn out gibberish. You can't convert gibberish into sensible output, and you can't make the engine produce correct output by hacking it with a hammer. You need to feed it quality fuel.

@Kaj Sotala: I can't - I'm not smart enough :)

But seriously, do you really think that we ought to wait a decade before a brilliant researcher shows up? And it seems all the more suspicious because this brilliant researcher has to read Eliezer's material in a tender age, or else he won't be good enough.

Now don't get me wrong, I love Eliezer's posts here, and I've learned A LOT of stuff. And I also happen to think that he's onto something when he talks about Friendly AI (and AI in general). But I don't see how he can hope to save the world by writing blog posts...

@Pyramid Head: I don't see how he can hope to save the world by writing blog posts...

Ditto. Autodidactism may be a superior approach for the education of certain individuals, but it allows the individual to avoid one element crucial to production: discipline. Mr. Yudkowsky's approach, and his resistance to work with others, along with his views that it is his job to save the world, and no one else can do it, suggest an element of savantism. Hardly a quality one would want in a superhuman intelligence.

I, too, enjoy his writing, but the fact that he discove... (read more)

I disagree with the last 2 comments.

Eliezer's priority has gradually shifted over the last 5 years or so from increasing his own knowledge to transmitting what he knows to others, which is exactly the behavior I would expect from someone with his stated goals who knows what he is doing.

Yes, he has suggested or implied many times that he expects to implement the intelligence explosion more or less by himself (and I do not like that) but ever since the Summer of AI his actions (particularly all the effort he has put into blogging and his references to 15-to-... (read more)

Suppose we break the problem down into multiple parts.

1. Understand how the problem works, what currently happens.
2. Find a better way that things can work, that would not generate the same problems.
3. Find a way to get from here to there.
4. Do it.

Then part 1 might easily be aided by a guy on a blog. Maybe part 2. Possibly part 3.

A blog is better than a newsgroup because the threads don't scroll off, they're all sitting on the site's computer if anybody cares. Also, as old posts are replaced by new posts people stop responding to old posts. So there isn... (read more)

It would seem that the big development in our lifetimes has been the advent of the digital computer, the Turing Machine. Assuming all humans come into the world with no basic knowledge other than hard-wired reflexes, we must all gain our knowledge from those who have have preceded us, along with our own reflections about that knowledge and reflections on our environmental observations. The entire Library of Congress is available digitally. Using the concepts of trend analysis and Bayesian Probabilities (and others I don't know about), couldn't a properly ... (read more)

Well, is there really no one else in the world right now to work in this problem along with Eliezer (who, in my opinion, don't lack discipline)? I can't help but think that it's rather arrogant...

Well, that's one of the reasons I'm not a SIAI donor, though. Can't donate money to someone who write blogs instead of researching Friendly AI theory. And I'm not nearly smart enough to make any progress on my own, or even help someone else. So I guess mankind is screwed :)

Retired urologist: "properly programed"

This is the hard part.

@sophiesdad: Autodidactism may be a superior approach for the education of certain individuals, but it allows the individual to avoid one element crucial to production: discipline.

@Pyramid Head: Eliezer (who, in my opinion, don't lack discipline)

My comment about discipline was not meant to be inflammatory, nor even especially critical. Rather, it was meant to be descriptive of one aspect of autodidactism. In comparison, suppose that Mr. Yudkowsky was working toward his PhD at (say) University of Great Computer Scientists. His chosen topic for his dissertat... (read more)

Sophiesdad, you should be aware that I'm not likely to take your advice, or even take it seriously. You may as well stop wasting the effort.

@Eliezer: Sophiesdad, you should be aware that I'm not likely to take your advice, or even take it seriously. You may as well stop wasting the effort.

Noted. No more posts from me.

An unusually moderate and temperate exchange.

Eliezer- Have you written anything fictional or otherwise about how you envision an ideal post-fAI or post-singularity world? Care to share?

Oh... I should have read these comments to the end, somehow missed what you said to sophiesdad.

Eliezer... I am very disappointed. This is quite sad.

Well, heck. At least he's being honest. Maybe a little blunt, but definitely honest.

Ok- Eliezer- you are just a human and therefore prone to anger and reaction to said anger, but you, in particular, have a professional responsibility not to come across as excluding people who disagree with you from the discussion and presenting yourself as the final destination of the proverbial buck. We are all in this together. I have only met you in person once, have only had a handful of conversations about you with people who actually know you, and have only been reading this blog for a few months, and yet I get a distinct impression that you have ... (read more)

IMO, the last two posts, and especially this one, are some of the best on Less Wrong. It's a pity this post isn't included in any of the sequences.