Ah, interesting. Do you know how that timeline interacts with the growth of the rationalist community in Berkeley?

I have no special insider knowledge. My impression, which I will gladly have corrected by those who know more, is that

• MIRI (formerly SIAI) was founded in Berkeley because that's where Eliezer was.
• Most of the rationalists in Berkeley are not MIRI employees.
• Most of the rationalists in Berkeley did not move to Berkeley because of MIRI or Eliezer or other rationalists.

But, again, this is vague impressions and guesswork and assumptions rather than actual knowledge. So let's assume for a moment that I'm entirely wrong and the Berkeley rationalist community is a consequence of MIRI. MIRI was founded about 16 years ago, and I think it's only in the last few years that the Berkeley rationalist community has been a big thing. That would suggest that the "build a rationalist community by starting an institution there" strategy takes 10 years or so to work.

If so, then good places to consider might be places that already have kinda-MIRI-like institutions. Perhaps Oxford (home of the Future of Humanity Institute, and also of Giving What We Can if you're the EA sort of rationalist) and to a lesser extent Cambridge (home of the Centre for the Study of Existential Risk). I think the FHI and the CSER are the nearest non-MIRI things to MIRI.

By the way if OpenAI were suggested before Musk, it would likely be regarded as such shaky idea.

I think that too much investment could result in more noise in the field. First of all because it will result in large number of published materials, which could exceed capacity of other researchers to read it. In result really interesting works will be not read. It will also attract in the field more people than actually clever and dedicated people exist. If we have 100 trained ai safety reserchers, which is overestimation , and we hire 1000 people, than real reasesrchers will be dissolved. In some fields like nanotech overinvestment result even in expel of original reaserchers because they prevent less educated ones to spent money as they want. But most dangerous thing is creating of many incomparable theories of friendliness, and even AIs based on them which would result in AI wars and extinction.

Source?

Trying posting here since I don't see how to post to https://agentfoundations.org/.

People who haven't been given full membership in the forum can post links to things they have written elsewhere, but cannot make posts on the forum itself.

Is this kind of reasoning covered by already known desiderata for logical uncertainty?

It sounds similar to the Gaifman condition. Say you have a Pi_1 sentence, meaning a sentence of the form "for all x: phi(x)", where phi(x) is computable. If you've checked all values of x up to some large number, and phi(x) has always been true so far, you might think that this means that phi(x) is probably true for all the other values of x too. The Gaifman condition says that the probability that you assign to "for all x: phi(x)" should go to 1 as the range of values of x you've checked goes to infinity.

But it turns out that any computable way of handling logical uncertainty that satisfies the Gaifman condition also must give probabilities that go to 0 for some other true sentences. https://intelligence.org/files/Pi1Pi2Problem.pdf This may sound alarming, but I don't think it is too surprising; after all, the theory of the natural numbers is not computable, so any logical uncertainty engine will not be able to rule out different theories even in the limit.

That's a funny comment. It does exactly the same thing twice: Please tell us where we didn't do too well, oh, and you are COMPLETELY WRONG because we did everything very well.

Of course you are right, but it would just be a linear transformation of the whole diagram, so it doesn't change anything in the result. I've built the diagram starting from a square, so I can't change this easily... just imagine the whole thing scaling on the X axis, OK?

Edit: since two people asked for this, I remade the diagram and now you can put in any values of P(E|H) and P(E|~H)

Humans are already biased towards thinking that various positive characteristics are correlated with each other. Keeping track of an explicit "goodness" variable would make that even worse. So while I don't see anything wrong with comparing specific characteristics between people or groups of people, I endorse the norm that it is not acceptable to make statements of the form "Person A is better than Person B" or "Group A is better than Group B". "Quality of character" is nowhere near specific enough.

I suspect that constraining a superintelligence from creating subagents will be much harder than designing AI control methods that leave no incentive to subvert them through creation of subagents.