Yes, but also remember that Turing's English, shy, and from King's College, home of a certain archness and dry wit. I think he's taking the piss, but the very ambiguity of it was why it appealed as a rationality quote. He's facing the evidence squarely, declaring his biases, taking the objection seriously, and yet there's still a profound feeling that he's defying the data. Or maybe not. Maybe I just read it that way because I don't buy telepathy.

The mathematician and Fields medalist Vladimir Voevodsky on using automated proof assistants in mathematics:

[Following the discovery of some errors in his earlier work:] I think it was at this moment that I largely stopped doing what is called “curiosity driven research” and started to think seriously about the future.

[...]

A technical argument by a trusted author, which is hard to check and looks similar to arguments known to be correct, is hardly ever checked in detail.

[...]

It soon became clear that the only real long-term solution to the problems that I encountered is to start using computers in the verification of mathematical reasoning.

[...]

Among mathematicians computer proof verification was almost a forbidden subject. A conversation started about the need for computer proof assistants would invariably drift to the Goedel Incompleteness Theorem (which has nothing to do with the actual problem) or to one or two cases of verification of already existing proofs, which were used only to demonstrate how impractical the whole idea was.

[...]

I now do my mathematics with a proof assistant and do not have to worry all the time about mistakes in my arguments or about how to convince others that my arguments are correct.

From a March 26, 2014 talk. Slides available here.

And whence the blasphemy?

1265 people are in group A. 947 are in group B, which is completely contained in A. Of all the people in group A, 450 satisfy property C, whereas this is true for 602 people in group B, all of whom are also in group A. 602 is larger than 450, so something has gone wrong.

On average, effective altruists (n = 412) donated $2503 to charity, and other people (n = 853) donated $523 - obviously a significant result.

There could be some measurement bias here. I was on the fence about whether I should identify myself as an effective altruist, but I had just been reminded of the fact that I hadn't donated any money to charity in the last year, and decided that I probably shouldn't be identifying as an effective altruist myself despite having philosophical agreements with the movement.

1265 people told us how much they give to charity; of those, 450 gave nothing. ... In order to calculate percent donated I divided charity donations by income in the 947 people helpful enough to give me both numbers. Of those 947, 602 donated nothing to charity, and so had a percent donated of 0.

This is blasphemy against Saint Boole.

It's worth mentioning that anyone with a strong argument against cryonics is likely to believe that you will be persuaded by it (due to low base-rates for these kinds of conversions). Thus the financial incentive is not as influential as you would like it to be.

Added: Relevant prediction

Also observe that 1900s ≠ 19th century, so they weren't that silly.

I guess gwern meant the construction was planned to take place in the 1900s.

If we assume a scenario without AGI and without a Hansonian upload economy, it seems quite likely that there are large currently unexpected obstacles for both AGI and uploading. Computing power seems to be just about sufficient right now (if we look at supercomputers), so it probably isn't the problem. So it will probably be a conceptual limitation for AGI and a scanning or conceptual limitation for uploads.

Conceptual limitation for uploads seems unlikely, because were just taking a system cutting it up into smaller pieces and and solving differential equations on a computer. Lots of small problems to solve, but no major conceptual ones. We could run into problems related to measuring quantum systems when doing the scanning (I believe Scott Aaronson wrote something about this suspicion lately). Note that this also puts a bound on the level of nano-technology we could have achieve, if we have neuron-sized scanning robots, we would be able to scan a brain and start the Hansonian scenario. Note that this does not preclude slightly larger scale manufacturing technologies, which would probably come from successive miniaturisations of 3d-printers.

Conceptual difficulties creating AGI are more or less expected by everyone around here, but in the case AGI is delayed by over a century we should get quite worried about other existential risks on our way there. Major contenders are global conflict and terrorism, especially involving nuclear, nano-technological or biological weapons. Even if nano-technology will not reach the level described in Sci-Fi, the bounds given above still allow for sufficient development to make advanced weapons be a question of blueprints and materials. Low probability huge impact risks from global warming are also worth mentioning, if only to note that there are a lot of other people working on them.

What does this tell us about analysing long-term risks like the slithy toves? Well I don't know anything about slithy toves, but let's look at the eugenics stuff discussed earlier and consider how it would influence the probability of major global conflicts, the question is not whether it would increase the risk of global conflict, but how much it would increase the risk of global conflict. On the other hand if AI-safety is already taken care of, it becomes a priority to develop AGI as soon as humanly possible. And then it would be really good if humanly possible was a sigma or so better than today. Still it wouldn't be great, since most of the risks we would be facing at this point would be quite small for each year (as it seems today we could of course get other info on our way there). It's really quite hard to say what would be the proper balance between more intelligent people and more time available at this point, we could say that if we've already had a century to solve the problem more time can't be that useful, on the other hand we could say that if we still haven't solved the problem in a century there are loads of sequential steps to get right we need all the time we can buy.

tldr: No AGI & No Uploads => most X-risk from different types of conflict => eugenics or any kind of superhumans increases X-risk due to risk of war between enhanced and old-school humans

I'm amused that when I was reading this, it didn't even occur to me that this might be about global warming - I just assumed it was about eugenics.

But fundamentally, I do think that the basic observation is right: our planning horizons should be fairly short, because we just don't know enough about future technology and developments to spend large amounts of resource on things with low option value. There are countless past crises that did not materialize or were averted by other developments; to give an imperfect list off the top of my head: horse shit in the streets of cities, the looming ice age, the degradation of the environment with industrialization, Kessler catastrophe, Y2K, and Hannu Kari's Internet apocalypse.

I am reminded of a story Kelly tells in The Clock of the Long Now about a Scandinavian country which set aside an island for growing big trees for making wooden warships in the 1900s, which was completely wrong since by that point, warships had switched to metal, and so the island became a nature preserve; this is a cute story of how their grossly mistaken forecasts had an unanticipated benefit, but being mistaken is not usually a good way of going about life, and the story would be a lot less cute if the action had involved something more serious like taxation or military drafts or criminal justice or economy-wide regulation.

and the dark arts that I use to maintain productivity.

Yes! Please tell us more about these!

I don't see the relevance of either of these links.