It's funny, I wrote a blog post arguing against humility not too long ago. I had a somewhat different picture of humility than you:

People internalize norms in very different ways and to very different degrees. There are people out there who don’t seem to internalize the norms of humility at all. We usually call these people “arrogant jerks”. And there are people – probably the vast majority of people – who internalize them in reasonable, healthy ways. We usually call these people “normal”.

But then there are also people who internalize the norms of humility in highly unhealthy ways. Humility taken to its most extreme limit is not a pretty thing – you don’t end up with with wise, virtuous, Gandalf-style modesty. You end up with self-loathing, pathological guilt, and scrupulosity. There are people out there – and they are usually exceptionally good, kind, and selfless people, although that shouldn’t matter – who are convinced that they are utterly worthless as human beings. For such people, showing even a modicum of kindness or charity towards themselves would be unthinkable. Anti-charity is much more common – whatever interpretation of a situation puts themselves in the worst light, that’s the one they’ll settle on. And why? Because it’s been drilled into their heads, over and over again, that to think highly of yourself – even to the tiniest, most minute degree – is wrong. It’s something that bad, awful, arrogant people do, and if they do it then they’ll be bad, awful, arrogant people too. So they take refuge in the opposite extreme: they refuse to think even the mildest of nice thoughts about themselves, and they never show themselves even the slightest bit of kindness.

Or take insecurity (please). All of us experience insecurity to one degree or another, of course. But again, there’s a pathological, unhealthy form it can take on that’s rooted in how we internalize the norms of humility. When you tell people that external validation is the only means by which they can feel good about themselves…well, surprisingly enough, some people take a liking to external validation. But in the worst cases it goes beyond a mere desire for validation, and becomes a need – an addiction, even. You wind up with extreme people-pleasers, people who center every aspect of their lives around seeking out praise and avoiding criticism.

But I actually don't think we disagree all that much, we're just using the same word to describe different things. I think the thing I called humility - the kind of draconian, overbearing anti-self-charity that scrupulous people experience - that is a bad thing. And I think the thing you called humility - acceptance of your flaws, self-compassion - that is a very good thing. In fact, I ended the essay with a call for more self-charity from (what I called) humble people. And I've been trying to practice self-compassion since writing that essay, and it's been a boon for my mental health.

(By far the most useful technique, for what it's worth, has been "stepping outside of myself", i.e. trying to see myself as just another person. I find when I do something embarrassing it's the worst thing to have ever happened, and obviously all my friends are thinking about how stupid I am and have lowered their opinion of me accordingly...whereas when a friend does something embarrassing, it maybe warrants a laugh, but then it seems totally irrelevant and has absolutely no bearing on what I think of them as a person. I now try as much as possible to look at myself with that second mindset.)

Anyway, language quibbles aside, I agree with this post.

I find that negative visualization in conjunction with Mark Williams' guided meditation "Exploring Difficulties" is useful for getting me in that stoic mindset of being more okay with a worst-case scenario. (Or at least, I hope so - I guess I'll see how well it worked if the worst-case scenario ever comes to pass.)

Hi Victoria,

I am highly interested in AI safety research. Unfortunately, I do not have a strong math background and I live in an area distant from AI research. After spending some time thinking about my future I have come to the decision to go for a math intensive PhD in some area not far from MIRI or FLI. I have only the bachelor degree in Engineering with major in Computer Science and Software Engineering. Currently, I spend most of my time working full time as a software developer, preparing for a GRE general exam and thinking about PhD and FAI.

Andrew Critch from MIRI and Berkeley is very enthusiastic about pursuing the PhD. He suggested the Statistics. I would be glad to know your opinions about PhD/AI & FAI research. Here is a list of some questions, which are bothering me.

• What do you think would be more relevant for AI safety research - CS, Statistics or something else?
• What areas of research are the most promising for AI safety, in your opinion?
• Is it better to pick the research area close to what MIRI working on, or a more general AI research one (such as a ML).
• Is it possible to increase the chances of successful admission by gaining some research experience before the admissions in this year? Or is it better to spend the time in some other way?
• Does the Math GRE subject test increase the chance of admission?

The above-mentioned researchers are skeptical in different ways. Andrew Ng thinks that human-level AI is ridiculously far away, and that trying to predict the future more than 5 years out is useless. Yann LeCun and Yoshua Bengio believe that advanced AI is far from imminent, but approve of people thinking about long-term AI safety.

Okay, but surely it’s still important to think now about the eventual consequences of AI. - Absolutely. We ought to be talking about these things.

Upvoted to encouraging people to get hands-on. Learning is good. Trying to go for a higehr level of understanding in whatever you do is a core rationality skill.

Sadly you stopped there though. For the sake of discussion, I've heard Artificial Intelligence: A Modern Approach is a good book on the subject. Hopefully a discussion could start here; perhaps there's something flawed, or perhaps the book is outdated. If anyone here, and I'm looking at you, the AI, AGI, FAI, IDK and other acronym-users whom I can't keep up with can provide some more directions for the potentially aspiring AI researchers lurking around, it would be very appreciated.

Janos and I are at NIPS!

I know there are many papers that show that neural nets learn features that can in some regimes be given nice interpretations. However in all cases of which I am aware where these representations have been thoroughly analyzed, they seem to fail obvious tests of naturality, which would include things like:

(1) Good performance on different data sets in the same domain.

(2) Good transference to novel domains.

(3) Robustness to visually imperceptible perturbations to the input image.

Moreover, ANNs almost fundamentally cannot learn natural representations because they fail what I would call the "canonicality" test:

(4) Replacing the learned features with a random invertible linear transformation of the learned features should degrade performance.

Note that the reason for (4) is that if you want to interpret an individual hidden unit in an ANN as being meaningful, then it can't be the case that a random linear combination of lots of units is equally meaningful (since a random linear combination of e.g. cats and dogs and 100 other things is not going to have much meaning).

That was a bit long-winded, but my question is whether the linked paper or any other papers provide representations that you think don't fail any of (1)-(4).

Thanks for writing this; a couple quick thoughts:

For example, it turns out that a learning algorithm tasked with some relatively simple tasks, such as determining whether or not English sentences are valid, will automatically build up an internal representation of the world which captures many of the regularities of the world – as a pure side effect of carrying out its task.

I think I've yet to see a paper that convincingly supports the claim that neural nets are learning natural representations of the world. For some papers that refute this claim, see e.g.

http://arxiv.org/abs/1312.6199 http://arxiv.org/abs/1412.6572

I think the Degrees of Freedom thesis is a good statement of one of the potential problems. Since it's essentially making a claim about whether a certain very complex statistical problem is identifiable, I think it's very hard to know whether it's true or not without either some serious technical analysis or some serious empirical research --- which is a reason to do that research, because if the thesis is true then that has some worrisome implications about AI safety.

I think there'd be value in just listing graduate programs in philosophy, economics, etc., by how relevant the research already being done there is to x-risk, AI safety, or rationality. Or by whether or not they contain faculty interested in those topics.

For example, if I were looking to enter a philosophy graduate program it might take me quite some time to realize that Carnegie Melon probably has the best program for people interested in LW-style reasoning about something like epistemology.

I have some questions about step 1 (find a flexible program):

My understanding is that there are two sources of inflexibility for PhD programs: A. Requirements for your funding source (e.g. TA-ing) and B. Vague requirements of the program (e.g. publish X papers). I'm excluding Quals, since you just have to pass a test and then you're done.

Elsewhere in the comments, someone wrote:

"Grad school is free. At most good PhD programs in the US, if you get in then they will offer you funding which covers tuition and pays you a stipend on the order of $25K per year. In return, you may have to do some work as a TA or in a professor's lab."

So, there are two types of jobs you can have to fund your PhD (TA-ing and being a RA/Research Assistant to a professor). How time-consuming is TA-ing generally? I imagine it varies based on the school/class. How do you find a TA-ing gig that isn't time consuming? Can you generally TA during your entire PhD? I think I vaguely recall a university that only would let you TA for so many semesters.

You could also fund your PhD by getting a fellowship. Philip Guo has written about applying for the NSF, NDSEG, Hertz fellowships. I'm poorly calibrated about how hard it is to get one of these fellowships. I've also heard that certain schools will offer fellowships to some of their students. How hard are these to get relative to the fellowships mentioned above? There are ~33K science PhDs awarded each year. I wonder what distinguishes the ~4% who get fellowships from the median science PhD student.

Let's say that you were really frugal and/or financially independent already. My impression is that many schools would still require you to TA in order to have your tuition waved.

Let’s assume you have the financial aspect of your PhD taken care of (e.g. You have an easy/enjoyable TA job). What other requirements are there other than passing Quals? Could I read interesting books indefinitely until I find something interesting to publish?

I'd like to believe that achieving step 1 is possible, but as eli_sennesh pointed out, this is hard.