Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: gworley 16 March 2017 07:45:58PM 2 points [-]

I had the same thoughts after listening to the same talk. I think the advantage of utility functions, though, is that they are well-defined mathematical constructs we can reason about and showcase the corner cases that may pop up in other models but would also be easier to miss. AGI, just like all existing intelligences, may not be implemented with a utility function, but the utility function provides a powerful abstraction for reasoning about what we might call more loosely its "preference relation" that, by admitted contradictions, may risk us missing cases where the contradictions do not exist and the preference relation becomes a utility function.

The point being, for the purpose of alignment, studying utility functions makes more sense because your control method can't possibly work on a preference relation if it can't even work on the simpler utility function. That real preference relations contain things that prevent the challenges of aligning utility functions in existing intelligences instead provides evidence of how the problem might be solved (at least for some bounded cases).

Comment author: chaosmage 16 March 2017 09:14:34PM 4 points [-]

That makes sense. But it isn't what Eliezer says in that talk:

There’s a whole set of different ways we could look at agents, but as long as the agents are sufficiently advanced that we have pumped most of the qualitatively bad behavior out of them, they will behave as if they have coherent probability distributions and consistent utility functions.

Do you disagree with him on that?

Could utility functions be for narrow AI only, and downright antithetical to AGI?

5 chaosmage 16 March 2017 06:24PM

Could utility functions be for narrow AI only, and downright antithetical to AGI? That's a quite fundamental question and I'm kind of afraid there's an obvious answer that I'm just too uninformed to know about. But I did give this some thought and I can't find the fault in the following argument, so maybe you can?

Eliezer Yudkowsky says that when AGI exists, it will have a utility function. For a long time I didn't understand why, but he gives an explanation in AI Alignment: Why It's Hard, and Where to Start. You can look it up there, but the gist of the argument I got from it is:

  1. (explicit) If an agent's decisions are incoherent, the agent is behaving foolishly.
    1. Example 1: If an agent's preferences aren't ordered, the agent prefers A to B, B to C but also C to A, it behaves foolishly.
    2. Example 2: If an agent allocates resources incoherently, it behaves foolishly.
    3. Example 3: If an agent's preferences depend on the probability of the choice even having to be made, it behaves foolishly.
  2. (implicit) An AGI shouldn't behave foolishly, so its decisions have to be coherent.
  3. (explicit) Making coherent decisions is the same thing as having a utility function.

I accept that if all of these were true, AGI should have a utility function. I also accept points 1 and 3. I doubt point 2.

Before I get to why, I should state my suspicion why discussions of AGI really focus on utility functions so much. Utility functions are fundamental to many problems of narrow AI. If you're trying to win a game, or to provide a service using scarce computational resources, a well-designed utility function is exactly what you need. Utility functions are essential in narrow AI, so it seems reasonable to assume they should be essential in AGI because... we don't know what AGI will look like but it sounds similar to narrow AI, right?

So that's my motivation. I hope to point out that maybe we're confused about AGI because we took a wrong turn way back when we decided it should have a utility function. But I'm aware it is more likely I'm just too dumb to see the wisdom of that decision.

The reasons for my doubt are the following.

  1. Humans don't have a utility function and make very incoherent decisions. Humans are also the most intelligent organisms on the planet. In fact, it seems to me that the less intelligent an organism is, the easier its behavior can be approximated with model that has a utility function!
    1. Apes behave more coherently than humans. They have a far smaller range of behaviors. They switch between them relatively predictably. They do have culture - one troop of chimps will fish for termites using a twig, while another will do something like a rain dance - but their cultural specifics number in the dozens, while those of humans are innumerable.
    2. Cats behave more coherently than apes. There are shy cats and bold ones, playful ones and lazy ones, but once you know a cat, you can predict fairly precisely what kind of thing it is going to do on a random day.
    3. Earthworms behave more coherently than cats. There aren't playful earthworms and lazy ones, they basically all follow the nutrients that they sense around them and occasionally mate.
    4. And single-celled organisms are so coherent we think we can even model them them entirely on standard computing hardware. Which, if it succeeds, means we actually know e.coli's utility function to the last decimal point.
  2. The randomness of human decisions seems essential to human success (on top of other essentials such as speech and cooking). Humans seem to have a knack for sacrificing precious lifetime for fool's errands that very occasionally create benefit for the entire species.

    A few occasions where such fool's errands happen to work out will later look like the most intelligent things people ever did - after hindsight bias kicks in. Before Einstein revolutionized physics, he was not obviously more sane than those contemporaries of his who spent their lives doing earnest work in phrenology and theology.

    And many people trying many different things, most of them forgotten and a few seeming really smart in hindsight - that isn't a special case that is only really true for Einstein, it is the typical way humans have randomly stumbled into the innovations that accumulate into our technological superiority. You don't get to epistemology without a bunch of people deciding to spend decades of their lives thinking about why a stick looks bent when it goes through a water surface. You don't settle every little island in the Pacific without a lot of people deciding to go beyond the horizon in a canoe, and most of them dying like the fools that they are. You don't invent rocketry without a mad obsession with finding new ways to kill each other.
  3. An AI whose behavior is determined by a utility function has a couple of problems that human (or squid or dolphin) intelligence doesn't have, and they seem to be fairly intrinsic to having a utility function in the first place. Namely, the vast majority of possible utility functions lead directly into conflict with all other agents.

    To define a utility function is to define a (direction towards a) goal. So a discussion of an AI with one, single, unchanging utility function is a discussion of an AI with one, single, unchanging goal. That isn't just unlike the intelligent organisms we know, it isn't even a failure mode of intelligent organisms we know. The nearest approximations we have are the least intelligent members of our species.
  4. Two agents with identical utility functions are arguably functionally identical to a single agent that exists in two instances. Two agents with utility functions that are not identical are at best irrelevant to each other and at worst implacable enemies.

    This enormously limits the interactions between agents and is again very different from the intelligent organisms we know, which frequently display intelligent behavior in exactly those instances where they interact with each other. We know communicating groups (or "hive minds") are smarter than their members, that's why we have institutions. AIs with utility functions as imagined by e.g. Yudkowsky cannot form these.

    They can presumably create copies of themselves instead, which might be as good or even better, but we don't know that, because we don't really understand whatever it is exactly that makes institutions more intelligent than their members. It doesn't seem to be purely multiplied brainpower, because a person thinking for ten hours often doesn't find solutions that ten persons thinking together find in an hour. So if an AGI can multiply its own brainpower, that doesn't necessarily achieve the same result as thinking with others.

Now I'm not proposing an AGI should have nothing like a utility function, or that it couldn't temporarily adopt one. Utility functions are great for evaluating progress towards particular goals. Within well-defined areas of activity (such as playing Chess), even humans can temporarily behave as if they had utility functions, and I don't see why AGI shouldn't.

I'm also not saying that something like a paperclip maximizer couldn't be built, or that it could be stopped once underway. The AI alignment problem remains real.

I do contend that the paperclip maximizer wouldn't be an AGI, it would be narrow AI. It would have a goal, it would work towards it, but it would lack what we look for when we look for AGI. And whatever that is, I propose we don't find it within the space of things that can be described with (single, unchanging) utility functions.

And there are other places we could look. Maybe some of it is in whatever it is exactly that makes institutions more intelligent than their members. Maybe some of it is in why organisms (especially learning ones) play - playfulness and intelligence seem correlated, and playfulness has that incoherence that may be protective against paperclip-maximizer-like failure modes. I don't know.

Comment author: chaosmage 16 March 2017 10:34:06AM 1 point [-]

I would have benefitted from about two detailed real-life examples, because while you seem to make sense, I find your reasoning to be slightly too abstract for my puny mind. But cool article. Thanks.

Comment author: gjm 22 February 2017 03:46:58PM 1 point [-]

This is a useful general prescription against irrationality: if a belief is supported by reason and evidence then you should be able to say what evidence would make you revise it. But it's worth noting that sometimes a belief may be reasonable but really hard to imagine remotely plausible evidence that would change your mind about it. What would Donald Trump have to do that would make you think he's a progressive internationalist who favours open borders and free trade? What would ISIS have to do to convince you that they are primarily an organization dedicated to fostering peace and cooperation among people of different religions?

Clearly it's not actually unreasonable to think that Donald Trump isn't keen on open borders and free trade, or that ISIS aren't particularly into peace and cooperation. But the question you should be able to answer to justify a claim that you believe those things rationally is, I suggest, not so much "what evidence would change your mind?" but "what different evidence would have led to a different conclusion?". If Donald Trump had campaigned on promises to lower tariffs and offer amnesties to illegal immigrants, or if ISIS gave out pamphlets about peace and love and charity instead of blowing things up, I'd have different opinions about them. (Though I'd probably still mistrust both.)

So if someone can't tell you what Donald Trump could do to convince them he's a pathological liar, rather than writing them off you might instead ask them "well, then what could he have done differently that would have led you to think of him that way?".

(Where you think they're not only wrong but obviously wrong, you might reasonably take the view that anyone who thinks the evidence is so one-sided that it would take an impossible amount of future evidence to change their mind is ipso facto probably nuts. So you might write them off after all, if you think no remotely reasonable person could think the evidence overwhelmingly favours Trump not being a pathological liar.)

Comment author: chaosmage 23 February 2017 10:36:31AM 0 points [-]

you might reasonably take the view that anyone who thinks the evidence is so one-sided that it would take an impossible amount of future evidence to change their mind is ipso facto probably nuts. So you might write them off after all

I agree that would be the sensible response, but I'm curious for ways to engage with people who see the world radically differently.

An ability to build particularly long bridges of consensus across particularly wide chasms of preconceptions could do the world a lot of good, if it is a learnable and teachable skill.

Comment author: chaosmage 22 February 2017 11:56:52AM 5 points [-]

Here's a nifty little trick Sam Harris shared in his recent AMA.

When someone says they don't believe something that you believe and consider to be obvious...

(Examples: "I don't think Trump is a pathological liar" - "I don't think ISIS is motivated by Islam")

...ask them for the counterfactual that would convince them the proposition was true.

(Examples: "What would Trump have to do that would make you think he is a pathological liar?" - "What would ISIS have to do to convince you they are motivated by Islam?")

Comment author: Good_Burning_Plastic 16 February 2017 12:18:44AM 1 point [-]

Is there a deadline for registration?

Also, is there a timetable? I mean, by what time would I have to be there on September 1?

Comment author: chaosmage 21 February 2017 08:08:51AM 0 points [-]

The last two times, there was an excellent welcome dinner at noon, but the actual programme started only at like 3pm. I definitely recommend being there by noon.

Comment author: siIver 12 February 2017 07:27:24PM *  6 points [-]

L. : While obviously being rational is good, LW as a community seems to be promoting elitism and entitlement.

s: Rationality can be scary that way. But it is about seeking truth, and the community does happen to consist of smart people. Denying that is false humility. Similarly, a lot of ideas many people support just happen to be false. It's not our fault that our society got it wrong on so many issues. We're just after truth.

L. : How does it serve truth to label people which aren't smart as mentally ill?

s: That's terrible, of course. But that's not a flaw of rationality, nothing about rationality dictates "you have to be cruel to other people". In fact if you think about this really hard you'll see that rationality usually dictates being nice.

L: Then how come this post on LessWrong is the most upvoted thing of the last 20 submissions?

s: ...

s: I can't defend that.

L. : Oh, okay. So I'm right and Yudkowsky's site does promote entitlement and sexism.

s: wait, sexism?

L. : Yeah. The last thing I saw from LW was two men talking about what a woman needs to do to fit the role they want her to have in society.

s: Okay, but that's not Yudkowsky's fault! He is not responsible for everyone on LW! The sequences don't promote sexism-

L. : I heard HPMoR is sexist, too.

s: That's not remotely true. It actually promotes feminism. Hermione is-

L. : I'm sorry, but I think I value the thoughts of other people who are more knowledgeable about sexism over yours. At least you condemn this article, but you still hang out on this site.


Scott Alexander has said that it's society's biggest mistake to turn away from intelligence (can't find the article). Even minor increases of intelligence correlate meaningfully with all sorts of things (a negative correlation with crime being one of them afaik). Intelligence is the most powerful force in the universe. A few intelligence points on the people working on Friendly AI right now could determine the fate of our entire species. I want to make it extra clear that I think intelligence is ultra-important and almost universally good.

None of this excuses this article. None of it suggests that it's somehow okay to label stupid people as mentally ill. Rationality is about winning, and this article is losing in every sense of the word. It won't be good for the reputation of LW, it won't be good for our agenda, and it won't be good for the pursuit of truth. The only expected positive effect is making people who read it feel good. It essentially says "being intelligent is good. Being stupid is bad. Other people are stupid. They are the problem. We are better than them." Which is largely true, but as helpful as making an IQ test, and emailing a friend saying "look here I am verifiable smarter than you and being smart is the most important thing in our society!"

Okay, but that's not a content critique. I just said I think this is bad and went from there. If the article was actually making a strong case, well then it could still be bad for having an unnecessarily insulting and harmful framing that is bad for our cause, but it might be defend-able on other grounds. Maybe. We want to do both; to win and to pursue truth, and those aren't the same thing. But I strongly feel the article doesn't succeed on that front, either. Let's take a look.


It's great to make people more aware of bad mental habits and encourage better ones, as many people have done on LessWrong.

sure.

The way we deal with weak thinking is, however, like how people dealt with depression before the development of effective anti-depressants:

seems to be true.

"Stupidity," like "depression," is a sloppy "common-sense" word that we apply to different conditions, which may be caused by genetics (for instance, mutations in the M1 or M3 pathways, or two copies of Thr92Ala), deep subconscious conditioning (e.g., religion), general health issues (like not getting enough sleep), environment (ignorance, lack of reward for intelligent behavior), or bad habits of thought.

There is an implicit assumption here that being stupid requires some kind of explanation, but nothing at all in the article provides a reason of why this would be the case. Stupidity is not depression. The reason why it makes sense to label depression as a mental illness is (I assume) that it corresponds to an anomaly in the territory. Suppose we had a function, depressedness(human, time) which displayed how depressed each person on earth has been for, say, the past 10 years. I would expect to see weird behavior of that function, strange peaks over intervals of time on various people, many of whom don't have unusually high values most of the time. This would suggest that it is something to be treated.

If you did the same for intelligence, I'd expect relatively low change on the time axis (aside from an increase at young age and a decrease in the case of actual mental illnesses) and some kind of mathematically typical distribution among the person axis ranging from 60 to dunno 170 or something. I feel really strange about having to make this argument, but this is really the crux of the problem here. The article doesn't argue "here and here are stats suggesting that there are anomalies with this function, therefore there is a thing which we could sensibly describe as a mental illness" it just says "some people are dumb, here are some dumb things they do, let's label that mental illness." To sum the fallacy committed here up in one sentence, it talks about a thing without explaining why that thing should exist.

It is implied that people being ashamed of admitting to depression is a problem, and I infer that the intention is to make being stupid feel less bad by labeling their condition a "mental illness." But it clearly fails in this regard, and is almost certainly more likely to do the opposite.. It's sort of a Lose-Lose dynamic: it implies that there is some specific thing influencing a natural distribution of intelligence, some special condition that covers "stupid "people which explains why they are stupid – which likely isn't the case, in that way having low IQ is probably worse than the article was meant to imply, since there is no special condition, you just got the lower end of the stick – while also being framed in such a way that it will make unintelligent people feel worse than before, not better.

And where is the reverse causation of believing in religion causing stupidity coming from? Postulating an idea like this ought to require evidence.

The article goes on to say that we should do something to make people smarter. I totally, completely, whole-heartedly agree. But saying high IQ is better than low IQ is something that can and has been done without all of the other stuff attached to it. And research in that direction is being done already. If you wanted to make a case for why we should have more of that, then you could do that so much more effectively without all the negativity attached to it.

Here are the accusations I am making. I accuse this article of not making a good case for anything that is both true and non-obvious, on top of being offensive and harmful for our reputation, and consequently our agenda. (Even if it is correct and there is an irregularity in the intelligence function, it doesn't make a good case.) I believe that if arguments of the same quality were brought forth on any other topic, the article would be treated the same way most articles with weak content are treated: with indifference, few upvotes, and perhaps one or two comments pointing out some flaws in it (if Omega appeared before me, I would bet a lot of money on that theory with a pretty poor ratio). I'll go as far as to accuse upvoting this as a failure of rationality. I agree with Pimgd on everything they said, but I feel like it is important to point out how awful this article is, rather than treating it as a simple point of disagreement. The fact that this has 12 upvotes is really, really really bad, and a symptom of a much larger problem.

This is not how you are being nice. This is not how you promote rationality. This is not how you win.

Comment author: chaosmage 14 February 2017 11:37:34AM 0 points [-]

I agree it isn't nice. I upvoted it anyway, because it is a very original idea that merits a discussion with not entirely predictable outcomes.

This isn't just the most-upvoted submission in a while, it is also the most-discussed in an even longer while.

Comment author: username2 13 February 2017 04:22:46PM 2 points [-]

Are there interesting youtubers lesswrong is subscribed to ? I never really used youtube and after watching history of japan I get the feeling I'm missing out on some stuff.

Comment author: chaosmage 14 February 2017 11:27:43AM 0 points [-]

Computerphile has a lot of interesting material. But my favorite is interviews with famous scientists and Elon Musk.

Comment author: chaosmage 14 February 2017 11:24:06AM 8 points [-]

I like this. I wonder what would happen if you post this on some LOTR fanfiction subreddit or something, with a link to a discussion of AI risk at the end.

Comment author: ChristianKl 14 February 2017 07:53:10AM *  1 point [-]

I thought we were arguing about actual clinical practice. Testosterone tests do exist but from what I read they are seldomly done in actual clinical practice.

Freudian psychoanalysis is still a large part of actual clinical practice.

Obviously. What's your point?

I'm not that certain that St John's Wort really has much worse side effects than many of the regular drugs. It might have more drug-drug interactions than various drugs because it has more active components.

There's much money invested into proving that existing drugs do better than something like St John's Wort and we know that this money skrews study results.

Comment author: chaosmage 14 February 2017 11:17:51AM 1 point [-]

Testosterone tests are common in the group that tends to need them (men over 40).

Freudian psychoanalysis continues to be paid for by health insurers in Germany for historical reasons and there's an aging cohort of psychoanalysts making their living with it in private practice, but clinics overwhelmingly do CBT instead, even in Germany.

What would convince you that St John's Wort is inferior to modern antidepressants?

View more: Next