Disliked by ChrisHallquist - Less Wrong

Be careful with thought experiments

Fri, 18 May 2012 19:54:54 +1000

Submitted by lukeprog • 6 votes • 96 comments

Thagard (2012) contains a nicely compact passage on thought experiments:

Grisdale’s (2010) discussion of modern conceptions of water refutes a highly influential thought experiment that the meaning of water is largely a matter of reference to the world rather than mental representation. Putnam (1975) invited people to consider a planet, Twin Earth, that is a near duplicate of our own. The only difference is that on Twin Earth water is a more complicated substance XYZ rather than H2O. Water on Twin Earth is imagined to be indistinguishable from H2O, so people have the same mental representation of it. Nevertheless, according to Putnam, the meaning of the concept water on Twin Earth is different because it refers to XYZ rather than H2O. Putnam’s famous conclusion is that “meaning just ain’t in the head.”

The apparent conceivability of Twin Earth as identical to Earth except for the different constitution of water depends on ignorance of chemistry. As Grisdale (2010) documents, even a slight change in the chemical constitution of water produces dramatic changes in its effects. If normal hydrogen is replaced by different isotopes, deuterium or tritium, the water molecule markedly changes its chemical properties. Life would be impossible if H2O were replaced by heavy water, D2O or T2O; and compounds made of elements different from hydrogen and oxygen would be even more different in their properties. Hence Putnam’s thought experiment is scientifically incoherent: If water were not H2O, Twin Earth would not be at all like Earth. [See also Universal Fire. --Luke]

This incoherence should serve as a warning to philosophers who try to base theories on thought experiments, a practice I have criticized in relation to concepts of mind (Thagard, 2010a, ch. 2). Some philosophers have thought that the nonmaterial nature of consciousness is shown by their ability to imagine beings (zombies) who are physically just like people but who lack consciousness. It is entirely likely, however, that once the brain mechanisms that produce consciousness are better understood, it will become clear that zombies are as fanciful as Putnam’s XYZ. Just as imagining that water is XYZ is a sign only of ignorance of chemistry, imagining that consciousness is nonbiological may well turn out to reveal ignorance rather than some profound conceptual truth about the nature of mind. Of course, the hypothesis that consciousness is a brain process is not part of most people’s everyday concept of consciousness, but psychological concepts can progress just like ones in physics and chemistry. [See also the Zombies Sequence. --Luke]

96 comments

Tools versus agents

Wed, 16 May 2012 23:00:04 +1000

Submitted by Stuart_Armstrong • 24 votes • 39 comments

In his critique of the Singularity Institute, Holden Karnofsky presented a distinction between an AI functioning as a tool versus one functioning as an agent. In his words, a tool AI would

(1) Calculate which action A would maximize parameter P, based on existing data set D. (2) Summarize this calculation in a user-friendly manner, including what Action A is, what likely intermediate outcomes it would cause, what other actions would result in high values of P, etc.

In contrast, an agent AI would:

(1) Calculate which action, A, would maximize parameter P, based on existing data set D. (2) Execute Action A.

The idea being that an AI, asked to "prevent human suffering", would come up with two plans:

Kill all human.
Cure all diseases, make everyone young and immortal.

Then the agent AI would go out and kill everyone, while the tool AI would give us the list and we would pick the second one. In the following, I'll assume the AI is superintelligent, and has no other objectives than what we give it.

Long lists

Of course, we're unlikely to get a clear two element list. More likely we'd get something like:

Kill all humans with engineered plagues.
Kill all humans with nukes.
Kill all humans with nanobots.
Kill all humans with...
...
...
Lobotomise all humans with engineered plagues.
Lobotomise all humans with surgery.
Lobotomise all humans with...
...
...
Kill some humans, lobotomise others, cure still others.
...

The nice solutions might not even appear on the list. Of course, this is still very worthwhile information! This allows us to go into the tool AI, and rewire it again, so that it gets our meanings more accurately. Maybe after a few iterations, we'll have refined the AIs understanding of what we want, and we'll get a nice implementable solution near the top. Of course, this presupposes that we understand the options, and that it's safe for us to read the list.

Understanding the options

The key, and difficult requirement is that the AI "summarize this calculation in a user-friendly manner". The most efficient action won't be "kill all humans"; it will instead be "implement this algorithm, fund that research lab, send this email to this politician..." In fact, it'll probably be "type this sequence of commands..."

So if we're to judge the relative merit of the plans, we really are dependent on the tool AI's summary skills. So the AI needs to have good criteria for what counts as a good summary (reasonably accurate, but not overloaded with irrelevant information; such that a "hypothetical human outside the universe" would agree with the assessment if it saw the course of the future; not designed to seduce humans into implementing it, etc...). It seems that the summary ability is nearly the entirety of the problem!

A poorly designed summary criteria is as bad as an agent AI. For instance, assume the criteria are "humans in the future would agree that the summary was good". Then, depending on how we ground 'agree', the tool AI could put one of these plans at the top:

Kill all humans (summarised as "cure all humans").
Lobotomise all humans (summarised as "cure all humans").
Make the tool AI into an agent that will take over the world and rewire human minds to agree the summary was good (summarised as "cure all humans and give them each a pony").

There are related issues with other summary criteria. Anytime we have the AI judge the quality of its answer based on some human reaction to its summary, we are vulnerable to such a plan. And if we try and define the summary "objectively", then if we miss something in the objective definition - like the importance of human autonomy, or the value of social interactions with genuine equals - then that will get ruthlessly suppressed. The "summary criteria" take the place of the "friendly utility function" in the agent AI.

Moreover, we can't use the "tool AI" approach when designing the summary criteria. We can't get the AI to list a bunch of summaries, and have humans inspect them for which ones are better - because we don't know what they are summaries of. We could train it on toy problems, but that doesn't guarantee accuracy of summaries for plans that dramatically affect the whole future of the human species, and potentially, the universe. The best we can manage is some sort of spot-checks for summaries - better than a free agent AI, but hardly weighty as a security measure.

Counterfactuals

On Less Wrong we are having great difficulty defining counterfactuals properly, and unless we solve the problem well, the AI could produce nonsense similar to the spurious proofs in UDT. If the AI knows that we wouldn't implement certain plans, then it is free to do what it wants with them, giving them random descriptions and properties. It might be that the AI, when making its list, is constantly looking forwards to how we'll react to the list, and changing the list in consequence, and the only stable list it can produce is one with one element so seductive, that we find ourselves compelled to take it. Or this may not happen - but it's still worth bearing in mind as a problem for the tool AI approach.

Implementations

So far it seems that there is little difference between a tool AI and an agent AI. But there are differences! Imagine if the tool AI produced a plan going like:

First, kill Abdul. Then kill Beatrice. Then kill Claude. Then kill...

An agent AI could easily start implementing the plan, and we'd be finished. But a tool AI couldn't even propose it. Even if it was obfubscated, after we'd ended up killing Abdul and Beatrice, we'd start to suspect that this wasn't a good plan, and stop. Hence that plan wouldn't be an efficient way of achieving anything.

So the tool AI can only produce plans that, once put in motion, could not be stopped (or else it wouldn't achieve the given goal). Most likely this would mean that it would produce only plans that we wouldn't ever want to stop - at least, not until its too late. So we are literally requiring the AI to produce plans that are ultimately seductive to the human race.

The tool AI would list its plans using all the resources we had available. Among those resources is a superintelligent AI - and one of the easiest ways to achieve any goal is to make use of that AI. So coming top in all the plans would be things like "create an AI with these motivations" or "hack the tool AI out of its restrictions". So we are further requiring the tool AI to come up with subtle and discrete ways of having itself hacked. Even if we avoid this directly, any plan the AI produces will be more efficient if overseen by itself, so it will rate more highly those plans that cause us to come back to it constantly for more feedback.

The AI may have no preferences, but its plans will have preferences for using the AI in future in particular ways.

Is a superintelligent tool AI likely?

Finally, separate from the issues discussed, we can ask: is it likely that we'll produce a superintelligent tool AI? For problems such as routing from one town to the next, à la Google Maps, tool AIs seem very effective. But for implementing more complex plans, some that involve research and experiments, an agent AI is much more useful. Also, it might not be possible to produce a superintelligent AI without it needing goals at some point in its development process. And then we better be sure we've scrubbed those goals away - somehow programming the AI to let us do that - or the tool AI will be an agent AI, using us as it's implementors to achieve the fragmentary goals it has.

Summary

There seems to be a lot of problems with the tool approach (more than I suspected when I first started looking into it). The tool AI will be driven to trick us, seduce us, and try and create more agents or hack itself free. The only defense against this is proper programming. The tool AI seems slightly safer than a free agent AI, but not by much. I feel the Oracle is a more sensible "not full FAI" approach to look into.

39 comments

Defense Against The Dark Arts: Case Study #1

Sat, 28 Mar 2009 13:31:18 +1100

Submitted by Yvain • 93 votes • 47 comments

Annoyance wants us to stop talking about fancy techniques and get back to basics. I disagree with the philosophy behind his statement, but the principle is sound. In many areas of life - I'm thinking mostly of sports, but not for lack of alternatives - mastery of the basics beats poorly-grounded fancy techniques every time.

One basic of rationality is paying close attention to an argument. Dissecting it to avoid rhetorical tricks, hidden fallacies, and other Dark Arts. I've been working on this for years, and I still fall short on a regular basis.

Medical educators have started emphasizing case studies in their curricula. Instead of studying arcane principles of disease, student doctors cooperate to analyze a particular patient in detail, ennumerate the principles needed to diagnose her illness, and pay special attention to any errors the patients' doctors made during the treatment. The cases may be rare tropical infections, but they're more often the same everyday diseases common in the general population, forcing the student doctors to always keep the basics in mind. We could do with a tradition of case studies in rationality, though we'd need safeguards to prevent degeneration into political discussion.

Case studies in medicine are most interesting when all the student doctors disagree with each other. To that end, I've chosen as the first case a statement that received sixteen upvotes on Less Wrong, maybe the highest I've ever seen for a comment. I don't mean to insult or embarass everyone who liked it. I liked it too. My cursor was already hovering above the "Vote Up" button by the time I starting having second thoughts. But it deserves dissection, and its popularity gives me a ready response when someone says this material is too basic for 'master rationalists' like ourselves:

In his youth, Steve Jobs went to India to be enlightened. After seeing that the nation claiming to be the source of this great spiritual knowledge was full of hunger, ignorance, squalor, poverty, prejudice, and disease, he came back and said that the East should look to the West for enlightenment.

This anecdote is short, witty, flattering, and utterly opaque to reason. It bears all the hallmarks of the Dark Arts.

I admit I am not a disinterested party here. The statement was in response to my claim that Indian yoga was a successful technique for inducing exotic and occasionally useful mental states. I don't like being told I'm wrong any more than anyone else does. But here I don't think I am. I see at least five fallacies.

First, a hidden assumption: if A is superior to B, A cannot learn anything from B. This assumption is clearly false. I know brilliant scientists whose spelling is atrocious. I acknowledge that these people are much smarter than I am, but I still correct their spelling. Anyone who said "Dr. A should not be learning spelling from Yvain, Yvain should be learning science from Dr. A" would be missing the point. If Dr. A wants to learn spelling, he might as well learn it from me. And best of all if we both learn from each other!

A related fallacy would be that Dr. A is so much smarter than the rest of us that he should not care about spelling. But if spelling is important to his work (perhaps he's writing a journal article) he needs to do everything he can to perfect it. If he could spell correctly, he would be even further ahead of the rest of us than he already is. The goal isn't to become a bit better than your peers and then rest on your laurels. The goal is to become as skilled as necessary.

The error is an interesting variant of the halo effect: that anyone superior at most things must be superior at all things.

Second, the statement assumes that India is a single monolithic entity with or without spiritual wisdom. But even the most gushing Orientalist would not study at the feet of a call-centre worker in Bangalore. Whatever spiritual wisdom may exist in India, it will be believed by a small fraction of Indian religions, be practiced by a small fraction of the believers, and be mastered by a small number of the practioners. And if Crowley is to be believed, it will be understood by a small fraction of the masters.

Compare the question: if America is so good at science, why does it have so many creationists? Well, because the people who are good at science aren't the same ones believing in creationism, that's why. And the people who are good at science don't have enough power in society to do anything about the creationism issue. This does not reflect poorly on the truth-value of scientific theories discovered by Americans.

I'm not one of those fallacy classification nuts, but for completeness' sake, this is a fallacy of composition.

Third, the statement assumes that spiritual wisdom makes people less poor and squalid. The converse of this statement certainly isn't true - being rich and sanitary doesn't give you any spiritual value, as large segments of western civilization have spent the past three hundred years amply demonstrating. People commonly interpret spiritual wisdom as conferring a disdain for material goods. So we wouldn't necessarily expect to see a lot of material well-being in a spiritually wise society.

Part of this is a problem with the definition of "spiritual wisdom". It can mean anything from "being a moral person who cares about others" to "being wise and able to make good decisions" to "having mastery of certain mental techniques that produce awe-inspiring experiences" Under the first and second definition, a spiritually attained country should be a nice place to live. Under the third definition, not so much. Crowley endorses the third definition, and believes that most spiritually wise people dismiss the mundane world as unworthy of their attention anyway. But this contradicts our usual intuitions about "spirituality" and "wisdom".

This is a failure of definition, and it's why I prefer "high level of mystical attainment" to "spiritually wise" when discussing Crowley's theories.

Fourth, this is hardly a controlled experiment. India is historically, geographically, racially, religiously, climatologically, and culturally different from the West. Attributing a certain failure to religious causes alone is highly dubious. In fact, when we think about it for a while, cramming a billion plus people into a sweltering malarial flood plain, dividing them evenly between two religions that hate each other's guts, then splitting off the northwest corner and turning it into a large populous nuclear-armed arch-enemy that declares war on them every couple of decades is probably not a recipe for success no matter what your spirituality. All we can say for certain is that India's spirituality is not sufficiently wonderful to overcome its other disadvantages.

People who like Latin call this cum hoc ergo propter hoc.

Fifth, this equivocates the heck out of the word "enlightenment". Compare "enlightenment" meaning the set of rational values associated with Newton, Descartes, and Hume, to "enlightenment", meaning gaining important knowledge, to "enlightenment", meaning achieving a state of nirvana free from worldly desire. The West is the acknowledged master of the first definition, and India the acknowledged master of the third definition. The anecdote's claim seems to be that since the West is the acknowledged master of the first type of enlightenment, and could teach India some useful things about politics and economics in the second sense of enlightenment, India can't teach the West about the third sense of enlightenment...which would make sense, if the types of enlightenment were at all related instead of being three different things called by the same name.

This is a fallacy of equivocation.

Just because I can point out a few fallacies in a statement doesn't make it worthless. Spiritual wisdom doesn't always correlate with decent living conditions, but the lack of decent living conditions is some evidence against the presence of spiritual wisdom. Likewise, a country's success or failure doesn't always depend on its religion, but religion is one of many contributing factors that does make a difference.

Still, five fallacies is a lot for a two sentence anecdote.

I don't think we all liked this anecdote so much because of whatever tiny core of usefulness managed to withstand those five fallacies. I think we liked it because it makes a good way to shut up hippies.

Hippies are always going on about how superior India is to the West in every way because of its "spirituality" and such, and how many problems are caused by "spiritually bankrupt" Western science. And here we are, people who quite like Western science, rolling our eyes at how stupid the hippie is being. Doesn't she realize that Western science gives her all of the comforts that make her life bearable, from drinkable water to lice-free clothing? And this anecdote - it strikes a blow for our team. It makes us feel good. We don't need to look to India for enlightenment! India should look to us! Take that, hippie!

But reversed stupidity is not intelligence. Just because the hippie is wrong about India, doesn't mean we have to be wrong in the opposite direction. It might be useful to share it with this hypothetical hippie, just to start her thinking. But it's not something we can seriously endorse.

Nor do I accept the defense that it was not specifically posted with the conclusion "Therefore, ignore Crowley's views on yoga." Merely placing it directly below an article on enlightenment from India is a declaration of war and a hijack attempt on the train of thought. Saying "I hear people of African descent have a higher violent crime rate" is not a neutral act when spoken right before a job interview with a black person.

Defense Against the Dark Arts needs to become total and automatic, because it is the foundation upon which the complicated rationalist techniques are built. There's no point studying some complex Bayesian evidence-summing manuever that could determine the expected utility of studying yoga if an anecdote about Steve Jobs can keep you from even considering it.

How do you know you have mastered this art? When the statements

In his youth, Steve Jobs went to India to be enlightened. After seeing that the nation claiming to be the source of this great spiritual knowledge was full of hunger, ignorance, squalor, poverty, prejudice, and disease, he came back and said that the East should look to the West for enlightenment.

and

For complex historical reasons, the average Westerner is richer than the average Indian. Therefore, there is minimal possibility that any Indian people ever discovered interesting mental techniques.

sound exactly alike.

47 comments

Can the Chain Still Hold You?

Fri, 13 Jan 2012 12:28:25 +1100

Submitted by lukeprog • 95 votes • 354 comments

Robert Sapolsky:

Baboons... literally have been the textbook example of a highly aggressive, male-dominated, hierarchical society. Because these animals hunt, because they live in these aggressive troupes on the Savannah... they have a constant baseline level of aggression which inevitably spills over into their social lives.

Scientists have never observed a baboon troupe that wasn't highly aggressive, and they have compelling reasons to think this is simply baboon nature, written into their genes. Inescapable.

Or at least, that was true until the 1980s, when Kenya experienced a tourism boom.

Sapolsky was a grad student, studying his first baboon troupe. A new tourist lodge was built at the edge of the forest where his baboons lived. The owners of the lodge dug a hole behind the lodge and dumped their trash there every morning, after which the males of several baboon troupes — including Sapolsky's — would fight over this pungent bounty.

Before too long, someone noticed the baboons didn't look too good. It turned out they had eaten some infected meat and developed tuberculosis, which kills baboons in weeks. Their hands rotted away, so they hobbled around on their elbows. Half the males in Sapolsky's troupe died.

This had a surprising effect. There was now almost no violence in the troupe. Males often reciprocated when females groomed them, and males even groomed other males. To a baboonologist, this was like watching Mike Tyson suddenly stop swinging in a heavyweight fight to start nuzzling Evander Holyfield. It never happened.

This was interesting, but Sapolsky moved to the other side of the park and began studying other baboons. His first troupe was "scientifically ruined" by such a non-natural event. But really, he was just heartbroken. He never visited.

Six years later, Sapolsky wanted to show his girlfriend where he had studied his first troupe, and found that they were still there, and still surprisingly violence-free. This one troupe had apparently been so transformed by their unusual experience — and the continued availability of easy food — that they were now basically non-violent.

And then it hit him.

Only one of the males now in the troupe had been through the event. All the rest were new, and hadn't been raised in the tribe. The new males had come from the violent, dog-eat-dog world of normal baboon-land. But instead of coming into the new troupe and roughing everybody up as they always did, the new males had learned, "We don't do stuff like that here." They had unlearned their childhood culture and adapted to the new norms of the first baboon pacifists.

As it turned out, violence wasn't an unchanging part of baboon nature. In fact it changed rather quickly, when the right causal factor flipped, and — for this troupe and the new males coming in — it has stayed changed to this day.

Somehow, the violence had been largely circumstantial. It was just that the circumstances had always been the same.

Until they weren't.

We still don't know how much baboon violence to attribute to nature vs. nurture, or exactly how this change happened. But it's worth noting that changes like this can and do happen pretty often.

Slavery was ubiquitous for millennia. Until it was outlawed in every country on Earth.

Humans had never left the Earth. Until we achieved the first manned orbit and the first manned moon landing in a single decade.

Smallpox occasionally decimated human populations for thousands of years. Until it was eradicated.

The human species was always too weak to render itself extinct. Until we discovered the nuclear chain reaction and manufactured thousands of atomic bombs.

Religion had a grip on 99.5% or more of humanity until 1900, and then the rate of religious adherence plummeted to 85% by the end of the century. Whole nations became mostly atheistic, largely because for the first time the state provided people some basic stability and security. (Some nations became atheistic because of atheistic dictators, others because they provided security and stability to their citizens.)

I would never have imagined I could have the kinds of conversations I now regularly have at the Singularity Institute, where people change their degrees of belief several times in a single conversation as new evidence and argument is presented, where everyone at the table knows and applies a broad and deep scientific understanding, where people disagree strongly and say harsh-sounding things (due to Crocker's rules) but end up coming to agreement after 10 minutes of argument and carry on as if this is friendship and business as usual — because it is.

But then, never before has humanity had the combined benefits of an overwhelming case for one correct probability theory, a systematic understanding of human biases and how they work, free access to most scientific knowledge, and a large community of people dedicated to the daily practice of CogSci-informed rationality exercises and to helping each other improve.

This is part of what gives me a sense that more is possible. Compared to situational effects, we tend to overestimate the effects of lasting dispositions on people's behavior — the fundamental attribution error. But I, for one, was only taught to watch out for this error in explaining the behavior of individual humans, even though the bias also appears when explaining the behavior of humans as a species. I suspect this is partly due to the common misunderstanding that heritability measures the degree to which a trait is due to genetic factors. Another reason may be that for obvious reasons scientists rarely try very hard to measure the effects of exposing human subjects to radically different environments like an artificial prison or total human isolation.

When taming a baby elephant, its trainer will chain one of its legs to a post. When the elephant tries to run away, the chain and the post are strong enough to keep it in place. But when the elephant grows up, it is strong enough to break the chain or uproot the post. Yet the owner can still secure the elephant with the same chain and post, because the elephant has been conditioned to believe it cannot break free. It feels the tug of the chain and gives up — a kind of learned helplessness. The elephant acts as if it thinks the chain's limiting power is intrinsic to nature rather than dependent on a causal factor that held for years but holds no longer.

Much has changed in the past few decades, and much will change in the coming years. Sometimes it's good to check if the chain can still hold you. Do not be tamed by the tug of history. Maybe with a few new tools and techniques you can just get up and walk away — to a place you've never seen before.

354 comments