I know there are many papers that show that neural nets learn features that can in some regimes be given nice interpretations. However in all cases of which I am aware where these representations have been thoroughly analyzed, they seem to fail obvious tests of naturality, which would include things like:

(1) Good performance on different data sets in the same domain.

(2) Good transference to novel domains.

(3) Robustness to visually imperceptible perturbations to the input image.

Moreover, ANNs almost fundamentally cannot learn natural representations because they fail what I would call the "canonicality" test:

(4) Replacing the learned features with a random invertible linear transformation of the learned features should degrade performance.

Note that the reason for (4) is that if you want to interpret an individual hidden unit in an ANN as being meaningful, then it can't be the case that a random linear combination of lots of units is equally meaningful (since a random linear combination of e.g. cats and dogs and 100 other things is not going to have much meaning).

That was a bit long-winded, but my question is whether the linked paper or any other papers provide representations that you think don't fail any of (1)-(4).

Thanks for writing this; a couple quick thoughts:

For example, it turns out that a learning algorithm tasked with some relatively simple tasks, such as determining whether or not English sentences are valid, will automatically build up an internal representation of the world which captures many of the regularities of the world – as a pure side effect of carrying out its task.

I think I've yet to see a paper that convincingly supports the claim that neural nets are learning natural representations of the world. For some papers that refute this claim, see e.g.

http://arxiv.org/abs/1312.6199 http://arxiv.org/abs/1412.6572

I think the Degrees of Freedom thesis is a good statement of one of the potential problems. Since it's essentially making a claim about whether a certain very complex statistical problem is identifiable, I think it's very hard to know whether it's true or not without either some serious technical analysis or some serious empirical research --- which is a reason to do that research, because if the thesis is true then that has some worrisome implications about AI safety.

I think there'd be value in just listing graduate programs in philosophy, economics, etc., by how relevant the research already being done there is to x-risk, AI safety, or rationality. Or by whether or not they contain faculty interested in those topics.

For example, if I were looking to enter a philosophy graduate program it might take me quite some time to realize that Carnegie Melon probably has the best program for people interested in LW-style reasoning about something like epistemology.

I have some questions about step 1 (find a flexible program):

My understanding is that there are two sources of inflexibility for PhD programs: A. Requirements for your funding source (e.g. TA-ing) and B. Vague requirements of the program (e.g. publish X papers). I'm excluding Quals, since you just have to pass a test and then you're done.

Elsewhere in the comments, someone wrote:

"Grad school is free. At most good PhD programs in the US, if you get in then they will offer you funding which covers tuition and pays you a stipend on the order of $25K per year. In return, you may have to do some work as a TA or in a professor's lab."

So, there are two types of jobs you can have to fund your PhD (TA-ing and being a RA/Research Assistant to a professor). How time-consuming is TA-ing generally? I imagine it varies based on the school/class. How do you find a TA-ing gig that isn't time consuming? Can you generally TA during your entire PhD? I think I vaguely recall a university that only would let you TA for so many semesters.

You could also fund your PhD by getting a fellowship. Philip Guo has written about applying for the NSF, NDSEG, Hertz fellowships. I'm poorly calibrated about how hard it is to get one of these fellowships. I've also heard that certain schools will offer fellowships to some of their students. How hard are these to get relative to the fellowships mentioned above? There are ~33K science PhDs awarded each year. I wonder what distinguishes the ~4% who get fellowships from the median science PhD student.

Let's say that you were really frugal and/or financially independent already. My impression is that many schools would still require you to TA in order to have your tuition waved.

Let’s assume you have the financial aspect of your PhD taken care of (e.g. You have an easy/enjoyable TA job). What other requirements are there other than passing Quals? Could I read interesting books indefinitely until I find something interesting to publish?

I'd like to believe that achieving step 1 is possible, but as eli_sennesh pointed out, this is hard.

Best way to find out is to ask the LWer Vika, who I'm pretty sure was the driving force (Max Tegmark probably had something to do with it too). I think their niche is to be a more celebrity-centered face of existential risk reduction (compared to FHI), but they've also made some moves to try to be a host of discussions, and this grant really means that now they have to play funding agency.

Two data points: I basically did this in two very difficult life situations and in both cases it worked very well.

1) During a relationship crisis I imagined the worst thing that could happen and what would follow from that. That allowed me to acting instead of staying passive-depressive because of lack of perceived options. Actually the options that sprang to live together with an altered view on the relationship led to sudden surge of high I was quite surprised by.

2) During a loss of employment period I also imagined what the worst thing could happen and realized that I could live with that. Gave me some calm back (but acting from that wasn't needed as an earlier promised job actually materialized.

Is there, or will there be, an RSS feed for this? I didn't see one anywhere.