As far as I can tell, Paul's current proposal might still suffer from blackmail, like his earlier proposal which I commented on

What problem do you have in mind here?

I thought that the previous problem was mostly psychological, i.e. that if humans were rational agents then this AI would be roughly as vulnerable to blackmail as its designers. So I thought the issue was the psychological strangeness (and great length) of the weird hypothetical.

Here we have no such hypothetical, and the system's behavior only depends on the predicted behavior of humans in the real world. That seems to address the narrow version of your concern.

I can see two analogous problems:

1. The learning system may believe itself to be in a simulation, as discussed here. But that seems to cash out to a straightforward technical problem: we want to ensure that as long as there is some important scenario where the AI behaves strangely, we provide feedback on one such scenario. And this technical problem both (1) looks like it should be soluble in principle, (2) looks pretty tractable right now, and (3) is needed to resolve a whole host of other problems (a bunch of exotic failures, but also a bunch of more prosaic difficulties).
2. The behavior of the system depends on what a human would do across many cases. We can sample from these cases at random, but maybe a small fraction of them are sufficiently surprising that they would lead a human to believe they are in a simulation. We would like to sample these situations with disproportionate probability as well so that the human can won't panic when they find themselves in one, but I haven't talked in any detail about how to do that and it's not obvious whether it is possible. (I do think it's possible.)

Did you have in mind 1, 2, or something else?

Compared to its competition in the AGI race, MIRI was always going to be disadvantaged by both lack of resources and the need to choose an AI design that can predictably be made Friendly as opposed to optimizing mainly for capability. For this reason, I was against MIRI (or rather the Singularity Institute as it was known back then) going into AI research at all, as opposed to pursuing some other way of pushing for a positive Singularity.

In any case, what other approaches to Friendliness would you like MIRI to consider? The only other approach that I'm aware of that's somewhat developed is Paul Christiano's current approach (see for example https://medium.com/ai-control/alba-an-explicit-proposal-for-aligned-ai-17a55f60bbcf), which I understand is meant to be largely agnostic about the underlying AI technology. Personally I'm pretty skeptical but then I may be overly skeptical about everything. What are your thoughts? I don't recall seeing you having commented on them much.

Are you aware of any other ideas that MIRI should be considering?

Lee Sedol has just resigned the second game.

EY was influenced by E.T. Jaynes, who was really against neural networks, in favor of bayesian networks. He thought NNs were unprincipled and not mathematically elegant, and bayes nets were. I see the same opinions in some of EY's writings, like the one you link. And the general attitude that "non-elegant = bad" is basically MIRI's mission statement.

I don't agree with this at all. I wrote a thing here about how NNs can be elegant, and derived from first principles. But more generally, AI should use whatever works. If that happens to be "scruffy" methods, then so be it.

Google bought DeepMind for, reportedly, more than $500 million. Other than possibly Eliezer, MIRI probably doesn't have the capacity to employ people that the market places such a high value on.

I think that MIRI did a mistake than decided not be evolved in actual AI research, but only in AI safety research. In retrospect the nature of this mistake is clear: MIRI was not recognised inside AI community, and its safety recommendations are not connected with actual AI development paths.

It is like a person would decide not to study nuclear physics but only nuclear safety. It even may work until some point, as safety laws are similar in many systems. But he will not be the first who will learn about surprises in new technology.

When I started hearing about the latest wave of results from neural networks, I thought to myself that Eliezer was probably wrong to bet against them. Should MIRI rethink its approach to friendliness?

• Thorsby's Transdimensional Brain Chip (Another Thorsby webcomic has finished. You know what you're getting: clever high-concept plot which keeps slowly building with occasional comedy of errors, semi-awkward writing, and MS Paint art that never improves. If you liked Accidental Space Spy, you'll like this. If not, not.)

• Broforce is a 2D pixel-art scrolling 1-hit-death run-and-gun action-shooter in the vein of Metal Slug with a War on Terror/'Murica'/1980s-action-movie theme; it adds an almost-fully destructible environment and emphasizes vertical movement, so it's the campy offspring of Metal Slug & Minecraft. The homages to MS are particularly noticeable in the vehicles you can use to fight in and how the terrorist enemies give way to alien enemies with occasional three-way battles. I loved MS as a kid for its beautiful sprites, touches of humor (like sneaking up on Nazis while they chatted), and perfectly-balanced action gameplay, so when I saw BF come up on Steam during the 2016 Lunar sale for \$7.49, I bought it. I figured even as a single-player game, it looked fun.

  Overall, I enjoyed it a lot. The controls are slick & reactive, with the default mapping of the up d-key to 'jump' quickly coming to feel natural; the action is almost instantly addictive, especially as you start to figure out how to work around the limit that you can usually only shoot horizontally and you are often outranged so if you approach enemies the straightforward way, you will typically die immediately. There's a lot of fun in figuring out how to best dig beneath enemies and attack them from behind, panic or shoot the suicide bombers into exploding amid a group of enemies, shoot out the ground underneath a tricky opponent, or set up chain reactions of explosions; since you do not choose which character & weapons you use, you also must learn how to work with individual characters in different situations (the Man in Black has a powerful shot, but the recoil means it can be tricky to use without knocking yourself into a pit or dropping a boulder onto your head; the Terminator's minigun is fantastic for tunneling but also has steady recoil and takes a fraction of a second to spin up; MacGyver's bomb throwing is more useful than it initially seems because you can destroy parts of the environment far from you and create impromptu suicide bombers). It borders on a physics puzzle game at times. While somewhat repetitious, there's enough variety in level design to keep one interested: many levels are straightforward run-and-gun, but when encountering a mecha, it may be best to tunnel underneath it to kill it instead of hijacking it, and in a level with constant bombardment, tunneling is a necessity. The destructible environment may sound like a recipe for cheap deaths and the possibility of making a level impossible to complete when you've destroyed too much ground to progress, but while I had to make some tricky jumps and excavations after some particularly heated firefights, I don't think I ever totally cut myself off from the end of the level.

  All of which made it even more frustrating when I ran into parts of BF which aren't quite polished yet: either not yet balanced appropriately, or an editor needs to tell the BF developers to kill their darlings. BF is proud of its large assortment of characters, but some of them plain suck and the game punishes you every time you rescue a prisoner and a random character swap is forced onto you and you get one of the sucky characters (random character swaps can be fun when it forces you out of your comfort zone, but not in those cases); I never figured out how to use melee characters like Neo or Blade successfully, and one character with a machine-gun leg (which only fires downwards) is so utterly useless that by the end of the game I was seriously considering just jumping into a pit anytime I got them and sparing myself the frustration of such a gimped character. BF would be much better if the worst 5 or 10 characters were simply deleted. (It would also be good if special attacks transferred. Nothing like saving up for a mecha only to lose it all when you free a prisoner... Fun like a kick in the teeth.) Another intense source of frustration was the lack of even the most vital hints. In the second covert mission, the start has a huge pit which must be jumped across and can only be jumped across if you are 'sprinting' by having double-tapped-and-held the directional key; nowhere are you told that 'sprinting' even exists and there is no reason you would have tried this double-tap or noticed the sprinting since, as mentioned before, flatout running is a guaranteed way to die. I died 20 or 30 times before I finally gave up and googled "Broforce impossible level"; there were several different levels which were the topic of discussion, and I finally learned about sprinting and beat the level a few deaths later. Another covert mission is still worse: the level begins with an enormous explosion, which you have been trained by dozens of previous enormous explosions to wait out; when you do so, the mission is unbeatable because half the level is gone. It turns out that unknown and unknowable to you, there was an long catwalk at the top of the level, and the solution to the level is to immediately begin running, get a powerup while running, invoke the time-slowing while running, climb a several-screen-high ladder at top speed while the rest of the level is slowed, and then run across the catwalk which is already exploding to the point where you are jumping from falling crate to crate. So just to know that that catwalk exists, much less what you have to do to finish the level you have to have started the level running and time-slow immediately to climb the ladder and get to see that there was a catwalk there! There is no hint whatsoever of any of this by BF. So if anyone was able to figure this level out on their own without checking the Internet & watching YouTube, I doff my chapeau to them. All it would have taken in these two levels is a short hint: 'double-tap the d-key to sprint'; 'reach the catwalk before the explosion!' I am not asking for those levels to be made easier, just that the player has some idea what they're supposed to do! This lack of hints extends to the characters themselves and their special attacks; what did Neo's special do? I had no idea - it made a little glowy circle which didn't seem to do anything. What did Robocop's special do? I had no idea - it created a green grid on the screen with a targeting reticule you could move, but pressing fire/special did nothing, and moving the reticule over enemies did nothing. How did Braveheart, the Professional etc? Towards the end I looked them up on the Broforce wikia, but I should not have had to; would it have killed BF to include a one-line summary of the special like "reflects most bullets" or "select each enemy to automatically attack"? A few of the levels trade too heavily on trial-and-error: one alien's level is unbeatable unless you simply memorize each location it attacks throughout the whole level, because it is invulnerable, and moves too fast to react to. The cave levels feature way too many boulders and rocks so you will constantly die of falling rocks no matter how careful you are. Two levels are so dark that you cannot see where your character is and will die many times from jumping into a wall to avoid an attack. On the giant-helicopter level, you can die and lose even after you've won during the animation of the helicopter exploding, which happened to me 3 times because the end of the level is so full of explosives. 'Fun'. The alien levels are substantially less enjoyable because the various acid explosions extend insta-death across the screen and the fast-moving aggressive melee aliens make certain characters almost useless (good luck using either bomb-thrower when you're busy being swarmed!). Perhaps partly due to the randomized character selection and total absence of documentation, there seems to be extreme variation in difficulty: I might die 5 times clearing a level, then 30 the next level, then 1 the following level! Certainly some variation in difficulty from level to level is normal and desirable, but that much variation suggests that some levels need some tweaking... Hopefully the developers will fix some of these issues in the future, and in the meantime, to those who play BF, I advise being less proud than myself and to look online for hints.

95% of all statistics are made up. It's very easy to make up data or confuse people with bad statistical treatment, but most science reporters / news media don't bother, they just honestly misunderstand the source data instead. If you can't check the statistical technique in detail yourself, and you don't very highly trust the source to do so (hint: news media are almost never trustworthy), you should ignore any statistical claims as being basically uncorrelated with reality.

I think, but am not certain, that you're missing the point, by examining Bob's incredulity rather than the problem as stated. Let's say your probability that the universe is being simulated is 2^x.

Alice flips a coin (x+1) times. You watch her flip the coins, and she carefully marks down the result of each flip.

No matter what sequence you watch, and she records - that sequence has less likelihood of having occurred naturally than that the universe is simulated, according to your priors. If it helps, imagine that a coin you know to be fair turns up Heads each time. (A sequence of all heads seems particularly unlikely - but every other sequence is equally unlikely.)