Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Eliezer_Yudkowsky 09 September 2013 05:25:56PM 5 points [-]

XiXiDu wasn't attempting or requesting anonymity - his LW profile openly lists his true name - and Alexander Kruel is someone with known problems (and a blog openly run under his true name) whom RobbBB might not know offhand was the same person as "XiXiDu" although this is public knowledge, nor might RobbBB realize that XiXiDu had the same irredeemable status as Loosemore.

I would not randomly out an LW poster for purposes of intimidation - I don't think I've ever looked at a username's associated private email address. Ever. Actually I'm not even sure offhand if our registration process requires/verifies that or not, since I was created as a pre-existing user at the dawn of time.

I do consider RobbBB's work highly valuable and I don't want him to feel disheartened by mistakenly thinking that a couple of eternal and irredeemable semitrolls are representative samples. Due to Civilizational Inadequacy, I don't think it's possible to ever convince the field of AI or philosophy of anything even as basic as the Orthogonality Thesis, but even I am not cynical enough to think that Loosemore or Kruel are representative samples.

Comment author: Juno_Watt 22 January 2014 12:01:57PM *  -1 points [-]

Alexander Kruel is someone with known problems

Hmmm...

Comment author: Yvain 14 March 2009 12:34:00AM *  40 points [-]

A lot of dojos preserve to some degree the social standards of Eastern countries where the sensei's sensei came from. And in Eastern countries, it's much less acceptable to try to question your teacher, or change things, or rock the boat, or show any form of weakness. I taught school in Japan for a while, and the first thing I learned was that naively asking "Any questions?" or "Any opinions on this?" or even "Anyone not understand?" was a waste of time.

Western cultures are a lot better at this, but not ideal. There's still pressure not to be the one person who asks all the questions all the time, and there's pressure not to say anything controversial out of the blue because you lose more status if you're wrong than you gain if you're right. I think part of the problem is that there really are dumb or egotistical people who, if given the chance will protest that they know a much better way to do everything and will waste the time of everyone else, and our society's decided to .make a devil's bargain to keep them under control.

The best solution to this is to found a new culture, live isolated from the rest of the world for a century developing different cultural norms, and then start the rationality dojo there. Of possible second-best solutions:

  • My Favorite Liar. Tell people that you're going to make X deliberately incorrect statements every training session and they've got to catch them.

  • Clickers. One of my lecturers uses these devices sort of like remote controls. You can input information into them and it gets sent wirelessly and anonymously to the lecturer's laptop. The theory is that if he says "Raise your hand if you don't understand this" or even "...if you disagree with this", no one will, but if he says "Enter whether or not you understand this into your clicker" he may get three or four "don't understand" responses. Anonymous suggestion boxes are a low-tech form of the same principle.

  • I always found the concept of Crocker's Rules very interesting. I also remember hearing of a community (wish I could remember which) in which it was absolutely forbidden to give negative feedback under certain circumstances, and the odd social dynamics that created. In a dojo-like setting, there might be situations when either of these two rules could be ritually enacted - for example, a special Crocker Hat, such that anyone wearing that hat was known to be under Crocker's Rules, and a special No Negative Feedback Hat (but with a flashier name, like White Crane Hat of Social Invincibility), which someone could wear when questioning the master or something and be absolutely immune to any criticism.

Comment author: Juno_Watt 28 September 2013 01:09:16PM 0 points [-]

My Favorite Liar. Tell people that you're going to make X deliberately incorrect statements every training session and they've got to catch them.

I can think of only one example of someone who actually did this, and that was someone generally classed a a mystic.

Comment author: RobbBB 12 September 2013 05:36:28PM *  1 point [-]

That problem has got to be solved somehow at some stage, because something that couldn't pass a Turing Test is no AGI.

Not so! An AGI need not think like a human, need not know much of anything about humans, and need not, for that matter, be as intelligent as a human.

To see this, imagine we encountered an alien race of roughly human-level intelligence. Would a human be able to pass as an alien, or an alien as a human? Probably not anytime soon. Possibly not ever.

(Also, passing a Turing Test does not require you to possess a particularly deep understanding of human morality! A simple list of some random things humans consider right or wrong would generally suffice.)

Why is that a problem? Is anyone suggesting AGI can be had for free?

The problem I'm pointing to here is that a lot of people treat 'what I mean' as a magical category. 'Meaning' and 'language' and 'semantics' are single words in English, which masks the complexity of 'just tell the AI to do what I mean'.

Ok. NL is hard. Everyone knows that. But its got to be solved anyway.

Nope!

Yeah. But it wouldn't be an AGI or an SI if it couldn't pass a TT.

It could certainly be an AGI! It couldn't be an SI -- provided it wants to pass a Turing Test, of course -- but that's not a problem we have to solve. It's one the SI can solve for itself.

A problem which has been solved over and over by humans.

No human being has ever created anything -- no system of laws, no government or organization, no human, no artifact -- that, if it were more powerful, would qualify as Friendly. In that sense, everything that currently exists in the universe is non-Friendly, if not outright Unfriendly.

Humans don't need to be loaded apriori with what makes other humans happy, they only need to know general indicators, like smiles and statements of approval.

All or nearly all humans, if they were more powerful, would qualify as Unfriendly.

Moreover, by default, relying on a miscellaneous heap of vaguely moral-sounding machine learning criteria will lead to the end of life on earth. 'Smiles' and 'statements of approval' are not adequate roadmarks, because those are stimuli the SI can seize control of in unhumanistic ways to pump its reward buttons.

"Intelligence on its own does not imply Friendliness."

That is an open question.

No, it isn't. And this is a non sequitur. Nothing else in your post calls orthogonality into question.

Comment author: Juno_Watt 13 September 2013 08:57:36AM *  1 point [-]

Not so! An AGI need not think like a human, need not know much of anything about humans, and need not, for that matter, be as intelligent as a human.

Is that a fact? No, it's a matter of definition. It's scarecely credible you are unaware that a lot of people think the TT is critical to AGI.

The problem I'm pointing to here is that a lot of people treat 'what I mean' as a magical category.

I can't see any evidence of anyone invlolved in these discussions doing that. It looks like a straw man to me.

Ok. NL is hard. Everyone knows that. But its got to be solved anyway.

Nope!

An AI you can't talk to has pretty limited usefulness, and it has pretty limited safety too, since you don;t even have the option of telling it to stop, or expaling to it why you don;t like what it is doing. Oh, and isn't EY assumign that an AGi will have NLP? After all, it is supposed to be able to talk its way out of the box.

It's one the SI can solve for itself.

It can figure out semantics for itslef. Values are a subsert of semantics...

No human being has ever created anything -- no system of laws, no government or organization, no human, no artifact -- that, if it were more powerful, would qualify as Friendly. I

Wherer do you get this stuff from? Modern societies, with their complex legal and security systems are much less violent than ancient socieites. To take ut one example.

All or nearly all humans, if they were more powerful, would qualify as Unfriendly.

Gee. Then I guess they don't have an architecutre with a basic drive to be friendly.

'Smiles' and 'statements of approval' are not adequate roadmarks, because those are stimuli the SI can seize control of in unhumanistic ways to pump its reward buttons.

Why don't humans do that?

No, it isn't.

Uh-huh. MIRI has settled that centuries-aold quesiton for once and all has it?

And this is a non sequitur.

It can't be a non-sequitur, since it is not an arguemnt but a statement of fact.

Nothing else in your post calls orthogonality into question.

So? It wasn't relevant anywhere else.

Comment author: JGWeissman 16 May 2012 06:31:24PM 4 points [-]

One major element of philosophical reasoning seems to be a distaste for and tendency to avoid arbitrariness.

If an agent has goal G1 and sufficient introspective access to know its own goal, how would avoiding arbirtrariness in its goals help it achieve goal G1 better than keeping goal G1 as its goal?

I suspect we humans are driven to philosophize about what our goals ought to be by our lack of introspective access, and that searching for some universal goal, rather than what we ourselves want, is a failure mode of this philosophical inquiry.

Comment author: Juno_Watt 12 September 2013 04:30:16PM -1 points [-]

If an agent has goal G1 and sufficient introspective access to know its own goal, how would avoiding arbirtrariness in its goals help it achieve goal G1 better than keeping goal G1 as its goal?

Avoiding arbitrariness is useful to epistemic rationality and therefore to instrumental rationality. If an AI has rationality as a goal it will avoid arbitrariness, whether or not that assists with G1.

Comment author: RobbBB 05 September 2013 04:23:35PM *  13 points [-]

But how could a seed AI be able to make itself superhuman powerful if it did not care about avoiding mistakes such as autocoreccting "meditating" to "masturbating"?

Those are only 'mistakes' if you value human intentions. A grammatical error is only an error because we value the specific rules of grammar we do; it's not the same sort of thing as a false belief (though it may stem from, or result in, false beliefs).

A machine programmed to terminally value the outputs of a modern-day autocorrect will never self-modify to improve on that algorithm or its outputs (because that would violate its terminal values). The fact that this seems silly to a human doesn't provide any causal mechanism for the AI to change its core preferences. Have we successfully coded the AI not to do things that humans find silly, and to prize un-silliness before all other things? If not, then where will that value come from?

A belief can be factually wrong. A non-representational behavior (or dynamic) is never factually right or wrong, only normatively right or wrong. (And that normative wrongness only constrains what actually occurs to the extent the norm is one a sufficiently powerful agent in the vicinity actually holds.)

Maybe that distinction is the one that's missing. You're assuming that an AI will be capable of optimizing for true beliefs if and only if it is also optimizing for possessing human norms. But, by the is/ought distinction, there is no true beliefs about the physical world that will spontaneously force a being that believes it to become more virtuous, if it didn't already have a relevant seed of virtue within itself.

Comment author: Juno_Watt 12 September 2013 04:06:29PM -2 points [-]

Those are only 'mistakes' if you value human intentions. A grammatical error is only an error because we value the specific rules of grammar we do; it's not the same sort of thing as a false belief (though it may stem from, or result in, false beliefs).

You will see a grammatical error as a mistake if you value grammar in general, or if you value being right in general.

A self-improving AI needs a goal. A goal of self-improvement alone would work. A goal of getting things right in general would work too, and be much safer, as it would include getting our intentions right as a sub-goal.

Comment author: nshepperd 05 September 2013 12:17:29PM 15 points [-]

Present day software is a series of increasing powerful narrow tools and abstractions. None of them encode anything remotely resembling the values of their users. Indeed, present-day software that tries to "do what you mean" is in my experience incredibly annoying and difficult to use, compared to software that simply presents a simple interface to a system with comprehensible mechanics.

Put simply, no software today cares about what you want. Furthermore, your general reasoning process here—define some vague measure of "software doing what you want", observe an increasing trend line and extrapolate to a future situation—is exactly the kind of reasoning I always try to avoid, because it is usually misleading and heuristic.

Look at the actual mechanics of the situation. A program that literally wants to do what you mean is a complicated thing. No realistic progression of updates to Google Maps, say, gets anywhere close to building an accurate world-model describing its human users, plus having a built-in goal system that happens to specifically identify humans in its model and deduce their extrapolated goals. As EY has said, there is no ghost in the machine that checks your code to make sure it doesn't make any "mistakes" like doing something the programmer didn't intend. If it's not programmed to care about what the programmer wanted, it won't.

Comment author: Juno_Watt 12 September 2013 03:11:47PM -1 points [-]

Present day software may not have got far with regard to the evaluative side of doing what you want, but the XiXiDu's point seems to be that it is getting better at the semantic side. Who was it who said the value problem is part of the semantic problem?

Comment author: Juno_Watt 12 September 2013 02:14:48PM *  0 points [-]

A. Solve the Problem of Meaning-in-General in advance, and program it to follow our instructions' real meaning. Then just instruct it 'Satisfy my preferences', and wait for it to become smart enough to figure out my preferences.

That problem has got to be solved somehow at some stage, because something that couldn't pass a Turing Test is no AGI.

But there are a host of problems with treating the mere revelation that A is an option as a solution to the Friendliness problem. 1. You have to actually code the seed AI to understand what we mean. Y

Why is that a problem? Is anyone suggesting AGI can be had for free?

  1. The Problem of Meaning-in-General may really be ten thousand heterogeneous problems, especially if 'semantic value' isn't a natural kind. There may not be a single simple algorithm that inputs any old brain-state and outputs what, if anything, it 'means'; it may instead be that different types of content are encoded very differently.

Ok. NL is hard. Everyone knows that. But its got to be solved anyway.

3... On the face of it, programming an AI to fully understand 'Be Friendly!' seems at least as difficult as just programming Friendliness into it, but with an added layer of indirection.

Yeah, but it's got to be done anyway.

[more of the same snipped]

It's clear that building stable preferences out of B or C would create a Friendly AI.

Yeah. But it wouldn't be an AGI or an SI if it couldn't pass a TT.

The genie — if it bothers to even consider the question — should be able to understand what you mean by 'I wish for my values to be fulfilled.' Indeed, it should understand your meaning better than you do. But superintelligence only implies that the genie's map can compass your true values. Superintelligence doesn't imply that the genie's utility function has terminal values pinned to your True Values, or to the True Meaning of your commands.

The issue of whether the SI's UF contains a set of human values is irrelevant. In a Loosemore architecture, an AI needs to understand and follow the directive "be friendly to humans", and those are all the goals it needs-- to understand, and to follow;

When you write the seed's utility function, you, the programmer, don't understand everything about the nature of human value or meaning. That imperfect understanding remains the causal basis of the fully-grown superintelligence's actions, long after it's become smart enough to fully understand our values.

The UF only needs to contain "understand English, and obey this directive". You don't have to code semantics into the UF. You do of course, have to code it in somewhere,

Instead, we have to give it criteria we think are good indicators of Friendliness, so it'll know what to self-modify toward

A problem which has been solved over and over by humans. Humans don't need to be loaded apriori with what makes other humans happy, they only need to know general indicators, like smiles and statements of approval.

Yes, the UFAI will be able to solve Friendliness Theory. But if we haven't already solved it on our own power, we can't pinpoint Friendliness in advance, out of the space of utility functions. And if we can't pinpoint it with enough detail to draw a road map to it and it alone, we can't program the AI to care about conforming itself with that particular idiosyncratic algorithm.

Why would that be necessary? In the Loosemore architecture, the AGI has the goals of understanding English and obeying the Be Friendly directive. It eventually gets a detailed, extensional, understanding of Friendliness from pursuing those goals, Why would it need to be preloaded with a detailed, extensional unpacking of friendliness? It could fail in understanding English, of course. But there is no reason to think it is unlikely to fail at understanding "friendliness" specifically, and its competence can be tested as you go along.

And if we can't pinpoint it with enough detail to draw a road map to it and it alone, we can't program the AI to care about conforming itself with that particular idiosyncratic algorithm.

I don't see the problem. In the Loosemore architecture, the AGI will care about obeying "be friendly", and it will arrive at the detailed expansion, the idiosyncracies, of "friendly" as part of its other goal to understand English. It cares about being friendly, and it knows the detailed expansion of friendliness, so where's the problem?

Yes, the UFAI will be able to self-modify to become Friendly, if it so wishes. But if there is no seed of Friendliness already at the heart of the AI's decision criteria, no argument or discovery will spontaneously change its heart.

Says who? It has the high level directive, and another directive to understand the directive. It's been Friendly in principle all along, it just needs to fill in the details.

Unless we ourselves figure out how to program the AI to terminally value its programmers' True Intentions,

Then we do need to figure out how to program the AI to terminally value its programmers' True Intentions. That is hardly a fatal objection. Did you think the Loosemore architecture was one that bootstraps itself without any basic goals?

And if we do discover the specific lines of code that will get an AI to perfectly care about its programmer's True Intentions, such that it reliably self-modifies to better fit them — well, then that will just mean that we've solved Friendliness Theory.

No. The goal to understand English is not the same as a goal to be friendly in every way, it is more constrained.

Solving Friendliness, in the MIRI sense, means preloading a detailed expansion of "friendly". That is not what is happening in the Loosemore architecture. So it is not equivalent to solving the same problem.

The clever hack that makes further Friendliness research unnecessary is Friendliness.

Nope.

Intelligence on its own does not imply Friendliness.

That is an open question.

It's true that a sufficiently advanced superintelligence should be able to acquire both abilities. But we don't have them both, and a pre-FOOM self-improving AGI ('seed') need not have both. Being able to program good programmers is all that's required for an intelligence explosion; but being a good programmer doesn't imply that one is a superlative moral psychologist or moral philosopher.

Then hurrah for the Loosemore architecture, which doesn't require humans to"solve" friendliness in the MIRI sense.

Comment author: linkhyrule5 10 September 2013 11:04:30PM 1 point [-]

You need non-cyclical reasoning. Which would generally be something where you aren't the one having to explain people that the achievement in question is profound.

This bit confuses me.

That aside:

You think Yudkowsky is not a crank, so you think the folks that play that silly game with him are intelligent and rational

Non sequitur. From the posts they make, everyone on this site seems to me to be sufficiently intelligent as to make "selling snake oil" impossible, in a cut-and-dry case like the AI box. Yudowsky's own credibility doesn't enter into it.

Comment author: Juno_Watt 12 September 2013 06:53:17AM 0 points [-]

Some folks on this site have accidentally bought unintentional snake oil in The Big Hoo Hah That Shall not Be Mentioned. Only an intelligent person could have bought that particular puppy,

Comment author: mwengler 28 August 2013 06:46:09PM 1 point [-]

The argument against p-zombies is that there is no physical difference that could explain the difference in consciousness. That does not extend to silicon WBEs or AIs

Two things. 1) that the same electronic functioning produces consciousness if implemented on biological goo but does not if implemented on silicon seems unlikely, what probability would you assign that this is the meaningful difference? 2) if it is biological goo we need to have consciousness, why not build an AI out of biological goo? Why not synthesize neurons and stack and connect them in the appropriate ways, and have understood the whole process well enough that either you assemble it working or you know how to start it? It would still be artificial, but made from materials that can produce consciousness when functioning.

Comment author: Juno_Watt 08 September 2013 10:20:08AM 0 points [-]

1) What seems (un)likely to an individual depends on their assumptions. If you regard consc. as a form of information processing, thern there is very little inferrential gap to a conclusion of functionalism or computationalism. But there is a Hard Problem of consc, precisely because some aspects --subjective experince, qualia -- don't have any theoretical or practical basis in functionalism of computer technology: we can build memory chips and write storage routines, but we can't even get a start on building emotion chips or writing seeRed().

2) It's not practical at the monent, and wouldn't answer the theoretical questions.

Comment author: Ghatanathoah 29 August 2013 04:22:59AM 1 point [-]

Mary isn't a normal human.

If this is the case then, as I said before, my intuition that she would not understand qualia disappears.

Comment author: Juno_Watt 31 August 2013 08:45:40AM -1 points [-]

my intuition that [Mary] would not understand qualia disappears.

For any value of abnormal? SHe is only quantitatively superior: she does not have brain-rewiring abilities.

View more: Next