Per AlexMennen, I'd rename CarefulBot-as-formulated to something like BoldBot, and apply the name CarefulBot to "swerve unless it's proven the other bot swerves."

I think (I'm an amateur here, full disclosure; could be mistakes) we can capture some of the behavior you're interested in by considering bots with different levels of proof system. So e.g. CBn swerves unless it can prove, using PA+n, that the opponent swerves.

Then we see that for n > m, CBn(CBm) = D. Proof: Suppose CBn doesn't swerve. In PA+n, it is an axiom that provability in PA+m implies truth. Therefore PA+m cannot prove that CBn swerves. In a contest between CarefulBots, the stronger proof system wins.

Now consider BoldBot. We can see that BBn does not swerve against BBn, because by symmetry both agents prove the same thing; and if they both proved their opponent does not swerve, they would both swerve, and both have proven a falsehood.

Analyzing BBn vs. BBm is proving difficult, but I'm going to try a bit more.

Per AlexMennen, I'd rename CarefulBot-as-formulated to something like BoldBot, and apply the name CarefulBot to "swerve unless it's proven the other bot swerves."

I think (I'm an amateur here, full disclosure; could be mistakes) we can capture some of the behavior you're interested in by considering bots with different levels of proof system. So e.g. CBn swerves unless it can prove, using PA+n, that the opponent swerves.

Then we see that for n > m, CBn(CBm) = D. Proof: Suppose CBn doesn't swerve. In PA+n, it is an axiom that provability in PA+m implies truth. Therefore PA+m cannot prove that CBn swerves. In a contest between CarefulBots, the stronger proof system wins.

Now consider BoldBot. We can see that BBn does not swerve against BBn, because by symmetry both agents prove the same thing; and if they both proved their opponent does not swerve, they would both swerve, and both have proven a falsehood.

Analyzing BBn vs. BBm is proving difficult, but I'm going to try a bit more.

Given that we observe the same thing no matter how we model the rock, I'm not sure what that proves.

stupidity can be an advantage. A literal rock can defeat any intelligent opponent at chicken, if it's resting on the gas pedal

The advantage of a rock is not that it is stupid. The advantage of a rock is that it has nothing to lose -- it is indifferent between all outcomes.

Yes, stupidity can be an advantage. A literal rock can defeat any intelligent opponent at chicken, if it's resting on the gas pedal (the swerve-to-avoid-collision version, rather than the brake-closer-to-the-cliff-edge version).

The catch is that you making yourself dumber to harvest this advantage has the same issues as other ways of trying to precommit.

From their website, it looks like they'll be doing a lot of deep learning research and making the results freely available, which doesn't sound like it would accelerate Friendly AI relative to AI as a whole. I hope they've thought this through.

Edit: It continues to look like their strategy might be counterproductive. [Edited again in response to this.]

In the original context, the alleged desirable ambiguity was the ability to concisely omit information--that is, to say "people" instead of "men and women". Tabooing 'ambiguity', I'd frame this as a matter of having words for large sets rather than requiring speakers to construct them out of smaller sets, and say that this is a good thing if those sets are commonly referred to.

On a similar note, there can be intensions whose extensions are not agreed upon--"good" and "right" spring to mind. At first I thought it would be necessary to have words for these, but upon reflection I'm not sure. Could we replace them with more specific words like "right according to classical utilitarianism" or "right according the ethics of the person this word relates to"?

The term you will want to use in your Google search is "Bayesian cognitive science". It's a huge field. But the short answer is, yes, the people in that field do assume that the brain does something that can be modelled as keeping and updating a probability distribution according to Bayes' rule. Much of it is computational-level modelling, i.e. rather removed from questions of implementation in the brain. A quick Google search did, however, find some papers on how to implement Bayesian inference in neural networks - though not necessarily linked to the brain. I'm sure some people do the latter sort of thing as well, though.

Thanks for writing this; a couple quick thoughts:

For example, it turns out that a learning algorithm tasked with some relatively simple tasks, such as determining whether or not English sentences are valid, will automatically build up an internal representation of the world which captures many of the regularities of the world – as a pure side effect of carrying out its task.

I think I've yet to see a paper that convincingly supports the claim that neural nets are learning natural representations of the world. For some papers that refute this claim, see e.g.

http://arxiv.org/abs/1312.6199 http://arxiv.org/abs/1412.6572

I think the Degrees of Freedom thesis is a good statement of one of the potential problems. Since it's essentially making a claim about whether a certain very complex statistical problem is identifiable, I think it's very hard to know whether it's true or not without either some serious technical analysis or some serious empirical research --- which is a reason to do that research, because if the thesis is true then that has some worrisome implications about AI safety.

I was wondering about the state of the deterrence in place against nuclear weapons usage, having always assumed it to be massive, and I can't tell if there's actually any formal international treaty about the use of nuclear weapons in war.

https://en.wikipedia.org/wiki/List_of_weapons_of_mass_destruction_treaties has arms-reduction, non-proliferation, and test ban treaties, but apparently nothing about who you actually nuke. I think Geneva says you can't target civilians with any weapon, but does anything prohibit nuking your enemy's army?