All of electroswing's Comments + Replies

I wrote a what I believe to be simpler explanation of this post here. Things I tried to do differently: 

1. More clearly explaining what Nash equilibrium means for infinitely repeated games -- it's a little subtle, and if you go into it just with intuition, it's not clear why the "everyone puts 99" situation can be a Nash equilibrium 
2. Noting that just because something is a Nash equilibrium doesn't mean it's what the game is going to converge to 
3. Less emphasis on minimax stuff (it's just boilerplate, not really the main point of folk theorems) 

The strategy profile I describe is where each person has the following strategy (call it "Strategy A"): 

• If empty history, play 99 
• If history consists only of 99s from all other people, play 99 
• If any other player's history contains a choice which is not 99, play 100 

The strategy profile you are describing is the following (call it "Strategy B"): 

• If empty history, play 99
• If history consists only of 99s from all other people, play 99 
• If any other player's history contains a choice which is not 99, play 30 

I agree Strategy B... (read more)

I'm not sure what the author intended, but my best guess is they wanted to say "punishment is bad because there exist really bad equilibria which use punishment, by folk theorems". Some evidence from the post (emphasis mine): 

Rowan: "If we succeed in making aligned AGI, we should punish those who committed cosmic crimes that decreased the chance of an positive singularity sufficiently."

Neal: "Punishment seems like a bad idea. It's pessimizing another agent's utility function. You could get a pretty bad equilibrium if you're saying agents sho

I have 2 separate claims:

1. Any researcher, inside or outside of academia, might consider emulating attributes successful professors have in order to boost personal research productivity. 
2. AI safety researchers outside of academia should try harder to make their legible to academics, as a cheap way to get more good researchers thinking about AI safety. 

What I'm questioning is the implicit assumption in your post that AI safety research will inevitably take place in an academic environment [...]

This assumption is not implicit, you're putting together ... (read more)

There are lots of ways a researcher can choose to adopt new productivity habits. They include:

1. Inside view, reasoning from first principles 
2. Outside view, copying what successful researchers do

The purpose of this post is to, from an outside view perspective, list what a class of researchers (professors) does, which happens to operate very differently from AI safety.

Once again, I am not claiming to have an inside view argument in favor of the adoption of each of these attributes. I do not have empirics. I am not claiming to have an airtight causal model.... (read more)

Overall, it seems like your argument is that AI safety researchers should behave more like traditional academia for a bunch reasons that have mostly to do with social prestige.

That is not what I am saying. I am saying that successful professors are highly successful researchers, that they share many qualities (most of which by the way have nothing to do with social prestige), and that AI safety researchers might consider emulating these qualities. 

Furthermore, I would note that traditional academia has been moving away from these practices, to a

However, I'd still like to know where you're drawing these observations from? Is it personal observation?

Yes, personal observation, across quite a few US institutions. 

And if so, how have you determined whether a professor is successful or not?

One crude way of doing it is saying that a professor is successful if they are a professor at a top 10-ish university. Academia is hypercompetitive so this is a good filter. Additionally my personal observations are skewed toward people who I think do good research, so additionally "successful" here means ... (read more)

I think the obvious answer here is AutoPay -- this should hedge against situations you are describing. 

The costs of making a mistake are certainly high, since it's a permanent hit to your credit report. I am not super knowledgeable of how late payments affect credit score (other than that it has a negative sign), this is an interesting question.

Hmmm...the orthogonality thesis is pretty simple to state, so I don't think necessarily that it has been grossly misunderstood. The bad reasoning in Fallacy 4 seems to come from a more general phenomenon with classic AI Safety arguments, where they do hold up, but only with some caveats and/or more precise phrasing. So I guess "bad coverage" could apply to the extent that popular sources don't go in depth enough. 

I do think the author presented good summaries of Bostrom's and Russell's viewpoints. But then they immediately jump to a "special sauce" ty... (read more)

I mean...sure...but again, this does not affect the validity of my counterargument. Like I said, I'm using as strong as possible of a counterargument by saying that even if the non-brain parts of the body were to add 2-100x computing power, this would not restrict our ability to scale up NNs to get human-level cognition. Obviously this still holds if we replace "2-100x" with "1x". 

The advantage of "2-100x" is that it is extraordinarily charitable to the "embodied cognition" theory—if (and I consider this to be extremely low probability) embodied cogni... (read more)

This claim is false.  (as in, the probability that it is true is vanishingly close to zero, unless the human brain uses supernatural elements).  All of the motor drivers except for the most primitive reflexes (certain spinal reflexes) are in the brain.  You can say that for all practical purposes, 100% of the computational power the brain has is in the brain.

I agree with your intuition here, but this doesn't really affect the validity of my counterargument. I should have stated more clearly that I was computing a rough upper bound. So saying... (read more)

What do you do to keep up with AI Safety / ML / theoretical CS research, to the extent that you do? And how much time do you spend on this? For example, do you browse arXiv, Twitter, ...? 

A broader question I'd also be interested in (if you're willing to share) is how you allocate your working hours in general. 

What's your take on "AI Ethics", as it appears in large tech companies such as Google or Facebook? Is it helping or hurting the general AI safety movement? 

You've appeared on the 80,000 Hours podcast two times. To the extent that you remember what you said in 2018-19, are there any views you communicated then which you no longer hold now? Another way of asking this question is—do you still consider those episodes to be accurate reflections of your views? 

According to your internal model of the problem of AI safety, what are the main axes of disagreement researchers have?