Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Extending the stated objectives

6 Stuart_Armstrong 13 January 2016 04:20PM

A putative new idea for AI control; index here.

A system that is optimizing a function of n variables, where the objective depends on a subset of size k<n, will often set the remaining unconstrained variables to extreme values; if one of those unconstrained variables is actually something we care about, the solution found may be highly undesirable.

Stuart Russell

Think of an AI directing a car, given the instructions to get someone to the airport as fast as possible (optimised variables include "negative of time taken to airport") with some key variables left out - such as a maximum speed, maximum acceleration, respect for traffic rules, and survival of the passengers and other humans.

Call these other variables "unstated objectives" (UO), as contrasted with the "stated objectives" (SO) such as the time to the airport. In the normal environments in which we operate and design our AIs, the UOs are either correlated with the SOs (consider the SO "their heart is beating" and the UO "they're alive and healthy") or don't change much at all (the car-directing AI could have been trained on many examples of driving-to-the-airport, none of which included the driver killing their passengers).

Typically, SOs are easy to define, and the UOs are the more important objectives, left undefined either because they are complex, or because they didn't occur to us in this context (just as we don't often say "driver, get me to the airport as fast a possible, but alive and not permanently harmed, if you please. Also, please obey the following regulations and restrictions: 1.a.i.α: Non-destruction of the Earth....").

The control problem, in a nutshell, is that optimising SOs will typically set other variables to extreme values, including the UOs. The more extreme the optimisation, and the furthest from the typical environment, the more likely this is to happen.

continue reading »

Don't teach people how to reach the top of a hill

30 PhilGoetz 04 March 2014 09:38PM

When is it faster to rediscover something on your own than to learn it from someone who already knows it?

Sometimes it's faster to re-derive a proof or algorithm than to look it up. Keith Lynch re-invented the fast Fourier transform because he was too lazy to walk all the way to the library to get a book on it, although that's an extreme example. But if you have a complicated proof already laid out before you, and you are not Marc Drexler, it's generally faster to read it than to derive a new one. Yet I found a knowledge-intensive task where it would have been much faster to tell someone nothing at all than to tell them how to do it.

continue reading »

Playing the student: attitudes to learning as social roles

9 Swimmer963 23 November 2012 02:56AM

This is a post about something I noticed myself doing this year, although I expect I’ve been doing it all along. It’s unlikely to be something that everyone does, so don’t be surprised if you don’t find this applies to you. It's also an exercise in introspection, i.e. likely to be inaccurate. 


If I add up all the years that I’ve been in school, it amounts to about 75% of my life so far–and at any one time, school has probably been the single activity that I spend the most hours on. I would still guess that 50% or less of my general academic knowledge was actually acquired in a school setting, but school has tests, and grades at the end of the year, and so has provided most of the positive/negative reinforcement related to learning. The ‘attitudes to learning’ that I’m talking about apply in a school setting, not when I’m learning stuff for fun.

Role #1: Overachiever

Up until seventh grade, I didn’t really socialize at school–but once I started talking to people, it felt like I needed a persona, so that I could just act ‘in character’ instead of having to think of things to say from scratch. Being a stereotypical overachiever provided me with easy material for small talk–I could talk about schoolwork to other people who were also overachievers.

Years later, after acquiring actual social skills in the less stereotyped environments of part-time work and university, I play the overachiever more as a way of reducing my anxiety in class. (School was easy for me up until my second year of nursing school, when we started having to do scary things like clinical placements and practical exams, instead of nice safe things like written exams.) If I can talk myself into always being curious and finding everything exciting and interesting and cool I want to do that!!!, I can’t find everything scary–or, at the very least, to other people it looks like I’m not scared.


Role #2: Too Cool for School

This isn’t one I’ve played too much, aside from my tendency to put studying for exams as maybe my fourth priority–after work, exercise, and sleep–and still having an A average. (I will still skip class to work a shift at the ER any day, but that doesn’t count–working there is almost more educational than class, in my mind.) As one of my LW Ottawa friends pointed out, there’s a sort of counter-signalling involved in being a ‘lazy’ student–if you can still pull off good grades without doing any work, you must be smart, so people notice this and respect it.

My brother is the prime example of this. He spent grades 9 through 11 alternately sleeping and playing on his iPhone in class, and maintained an average well over 80%. In grade 12 he started paying attention in class and occasionally doing homework, and graduated with, I believe, an average over 95%. He had a reputation throughout the whole school–as someone who was very smart, but also cool.

Role #3: Just Don’t Fail Me!

Weirdly enough, it wasn’t at school that I originally learned this role. As a teenager, I did competitive swimming. The combination of not having outstanding talent for athletics, plus the anxiety that came from my own performance depending on how fast the other swimmers were, made this about 100 times more terrifying than school. At some point I developed a weird sort of underconfidence, the opposite of using ‘Overachiever’ to deal with anxiety. My mind has now created, and made automatic, the following subroutine: “when an adult takes you aside to talk to you about anything related to ‘living up to your potential’, start crying.” I’m not sure what the original logic behind this was: get the adult to stop and pay attention to me? Get them to take me more seriously? Get them to take me less seriously? Or just the fact that I couldn’t stomach the fact of being ordinarily below average at something–I had to be in some way differently below average. Who knows if there was much logic behind it at all?  

Having this learned role comes back to bite me now, sometimes–the subroutine gets triggered in any situation that feels too much like my swim coach’s one-on-one pre-competition pep talks. Taekwondo triggers it once in a while. Weirdly enough, being evaluated in clinicals triggers it too–this didn’t originally make much sense, since it’s not competitive in the sense of ‘she wins, I lose.’ I think the associative chain there is through lifeguarding courses–the hands-on evaluation aspect used to be fairly terrifying for my younger self, and my monkey brain puts clinicals and lab evaluations into that category, as opposed to the nice safe category of written exams, where I can safely be Too Cool for School and still get good grades.  

The inconvenience of thinking about school this way really jumped out at me this fall. I started my semester of clinicals with a prof who was a) spectacularly non-intimidating compared to some others I’ve had, and b) who liked me from the very start, basically because I raised my hand a lot and answered questions intelligently during our more classroom-y initial orientation. I was all set up for a semester of playing ‘Overachiever’, until, quite near the beginning of the semester, I was suddenly expected to do something that I found scary, and I was tired and scared of looking confident but being wrong, and I fell back on ‘Just Don’t Fail Me!’ My prof was, understandably, shocked and confused as to why I was suddenly reacting to her as ‘the scary adult who has the power to pass or fail me and will definitely fail me unless I’m absolutely perfect, so I had better grovel.’ I think she actually felt guilty about whatever she had done to intimidate me–which was nothing.

Since then I’ve been doing fine, progressing at the same rate as all the other students (maybe it says something about me that this isn’t very satisfying, and even kind of feels like failure in itself...I would like to be progressing faster). That is, until I’m alone with my prof and she tries to give me a pep talk about how I’m obviously very smart and doing fine, so I just need to improve my confidence. Then I start crying. At this point, I’m pretty sure she thinks I should be on anti-depressants–which is problematic in itself, but could be more problematic if she was the kind of prof who might fail me in my clinical for a lack of confidence. There’s no objective reason why I can’t hop back into Overachiever mode, since I managed both my clinicals last spring entirely in that mode. But part of my brain protests: ‘she’s seen you being insecure! She wouldn’t believe you as an overachiever, it would be too out of character!’ It starts to make sense once I stop seeing this behaviour as 'my learning style' and recognize it as a social role that I, at some point, probably subconsciously, decided I ought to play.



The main problem seems to be that my original mental models for social interaction–with adults, mostly–are overly simplistic and don’t cut reality at the joints. That’s not a huge problem in itself–I have better models now and most people I meet now say I have good communication skills, although I sometimes still come across as ‘odd’. The problem is that every once in a while, a situation happens, pattern recognition jumps into play, and whoa, I’m playing ‘Just Don’t Fail Me’. (It’s happened with the other two roles too, but they’re is less problematic.) Then I can’t get out of that role easily, because my social monkey brain is telling me it would be out of character and the other person would think it was weird. This is despite the fact that I no longer consciously care if I come across as weird, as long as people think I’m competent and trustworthy and nice, etc.

Just noticing this has helped a little–I catch my monkey brain and remind it ‘hey, this situation looks similar to Situation X that you created a stereotyped response for, but it’s not Situation X, so how about we just behave like a human being as usual’. Reminding myself that the world doesn’t break down into ‘adults’ and ‘children’–or, if it did once, I’m now on the other side of the divide–also helps. Failing that, I can consciously try to make sure I get into the 'right’ role–Overachiever or Too Cool For School, depending on the situation–and make that my default. 

Has anyone else noticed themselves doing something similar? I’m wondering if there are other roles that I play, maybe more subtly, at work or with friends. 


LW Biology 101 Introduction: Constraining Anticipation

50 virtualAdept 25 May 2011 12:32AM

Since the responses to my recent inquiry were positive, I've rolled up my sleeves and gotten started.  Special thanks to badger for eir comment in that thread, as it inspired the framework used here.  

My intent in the upcoming posts is to offer a practical overview of biological topics of both broad-scale importance and particular interest to the Less Wrong community.  This will by no means be exhaustive (else I’d be writing a textbook instead, or more likely, you’d be reading one); instead I am going to attempt to sketch what amounts to a map of several parts of the discipline – where they stand in relation to other fields, where we are in the progress of their development, and their boundaries and frontiers.  I’d like this to be a continually improving project as well, so I would very much welcome input on content relevance and clarity for any and all posts. 

I will list relevant/useful references for more in-depth reading at the end of each post.  The majority of in-text links will be used to provide a quick explanation of terms that may not be familiar or phenomena that may not be obvious.  If the terms are familiar to you, you probably do not need to worry about those links.  A significant minority of in-text links may or may not be purely for amusement.

continue reading »

Go Try Things

6 atucker 03 March 2011 01:51AM

This is the first in what will hopefully be a series of posts about why you should try things, and with strategies against common reasons for not doing so.


You’ve probably read about how to properly turn information into beliefs, and how to squeeze every last bit from your data. There's been less attention on the importance of going and getting data.

This article is about how personal experience is an incredibly useful form of data, and in particular how in many activities going out and trying something is more marginally useful than doing more exploratory research into it. In particular, I'll examine how personal experience is useful because it makes information more tangible and easier to learn, is good practice, and exposes common circumstances that you didn't build your models to handle.

For precise and well-defined fields and problems, clear thinking and reasoning will get you really far. Mathematics departments don’t use that much equipment, and they’ve been going on pretty well for hundreds of years.

Rationality is about how to turn data into maps. But this still requires data. I think that in a wide variety of not particularly theoretical subjects (like sports, social interactions, negotiation, cooking, etc.) rationality needs to be augmented by personal experience. Instrumental Rationality turns models into high-utility actions, but before you can do that you need a model.

continue reading »

Back to the Basics of Rationality

80 lukeprog 11 January 2011 07:05AM

My deconversion from Christianity had a large positive impact on my life. I suspect it had a small positive impact on the world, too. (For example, I no longer condemn gays or waste time and money on a relationship with an imaginary friend.) And my deconversion did not happen because I came to understand the Bayesian concept of evidence or Kolmogorov complexity or Solomonoff induction. I deconverted because I encountered some very basic arguments for non-belief, for example those in Dan Barker's Losing Faith in Faith.

Less Wrong has at least two goals. One goal is to raise the sanity waterline. If most people understood just the basics Occam's razor, what constitutes evidence and why, general trends of science, reductionism, and cognitive biases, the world would be greatly improved. Yudkowsky's upcoming books are aimed at this first goal of raising the sanity waterline. So are most of the sequences. So are learning-friendly posts like References & Resources for LessWrong.

A second goal is to attract some of the best human brains on the planet and make progress on issues related to the Friendly AI problem, the problem with the greatest leverage in the universe. I have suggested that Less Wrong would make faster progress toward this goal if it worked more directly with the community of scholars already tackling the exact same problems. I don't personally work toward this goal because I'm not mathematically sophisticated enough to do so, but I'm glad others are!

Still, I think the first goal could be more explicitly pursued. There are many people like myself and jwhendy who can be massively impacted for the better not by coming to a realization about algorithmic learning theory, but by coming to understand the basics of rationality like probability and the proper role of belief and reductionism.

continue reading »

Self-fulfilling correlations

103 PhilGoetz 26 August 2010 09:07PM

Correlation does not imply causation.  Sometimes corr(X,Y) means X=>Y; sometimes it means Y=>X; sometimes it means W=>X, W=>Y.  And sometimes it's an artifact of people's beliefs about corr(X, Y).  With intelligent agents, perceived causation causes correlation.

Volvos are believed by many people to be safe.  Volvo has an excellent record of being concerned with safety; they introduced 3-point seat belts, crumple zones, laminated windshields, and safety cages, among other things.  But how would you evaluate the claim that Volvos are safer than other cars?

Presumably, you'd look at the accident rate for Volvos compared to the accident rate for similar cars driven by a similar demographic, as reflected, for instance in insurance rates.  (My google-fu did not find accident rates posted on the internet, but insurance rates don't come out especially pro-Volvo.)  But suppose the results showed that Volvos had only 3/4 as many accidents as similar cars driven by similar people.  Would that prove Volvos are safer?

continue reading »

Crunchcourse - a tool for combating learning akrasia

11 whpearson 14 March 2010 10:53PM

Crunchcourse is a free website that might be of use to people trying to learn things outside the normal classroom setting. It aims to get together groups of people interested in the same topic and use our social instincts to motivate us to do the work.

It is in its early stages. If it proves useful, it might be useful to standardize on it as the place to learn the various prerequisites that lesswrong has.