Thanks for taking the time to explain your reasoning, Mark. I'm sorry to hear you won't be continuing the discussion group! Is anyone else here interested in leading that project, out of curiosity? I was getting a lot out of seeing people's reactions.

I think John Maxwell's response to your core argument is a good one. Since we're talking about the Sequences, I'll note that this dilemma is the topic of the Science and Rationality sequence:

In any case, right now you've got people dismissing cryonics out of hand as "not scientific", like it was some kind of pharmaceutical you could easily administer to 1000 patients and see what happened. "Call me when cryonicists actually revive someone," they say; which, as Mike Li observes, is like saying "I refuse to get into this ambulance; call me when it's actually at the hospital". Maybe Martin Gardner warned them against believing in strange things without experimental evidence. So they wait for the definite unmistakable verdict of Science, while their family and friends and 150,000 people per day are dying right now, and might or might not be savable—

—a calculated bet you could only make rationally [i.e., using your own inference skills, without just echoing data from an experimental study, and without just echoing established, expert-verified scientific conclusions].

The drive of Science is to obtain a mountain of evidence so huge that not even fallible human scientists can misread it. But even that sometimes goes wrong, when people become confused about which theory predicts what, or bake extremely-hard-to-test components into an early version of their theory. And sometimes you just can't get clear experimental evidence at all.

Either way, you have to try to do the thing that Science doesn't trust anyone to do—think rationally, and figure out the answer before you get clubbed over the head with it.

(Oh, and sometimes a disconfirming experimental result looks like: "Your entire species has just been wiped out! You are now scientifically required to relinquish your theory. If you publicly recant, good for you! Remember, it takes a strong mind to give up strongly held beliefs. Feel free to try another hypothesis next time!")

This is why there's a lot of emphasis on hard-to-test ("philosophical") questions in the Sequences, even though people are notorious for getting those wrong more often than scientific questions -- because sometimes (e.g., in the case of cryonics and existential risk) the answer matters a lot for our decision-making, long before we have a definitive scientific answer. That doesn't mean we should despair of empirically investigating these questions, but it does mean that our decision-making needs to be high-quality even during periods where we're still in a state of high uncertainty.

The Sequences talk about the Many Worlds Interpretation precisely because it's an unusually-difficult-to-test topic. The idea isn't that this is a completely typical example, or that it's a good idea to disregard evidence when it is available; the idea, rather, is that we sometimes do need to predicate our decisions on our best guess in the absence of perfect tests.

Its placement in Rationality: From AI to Zombies immediately after the 'zombies' sequence (which, incidentally, is an example of how and why we should reject philosophical thought experiments, no matter how intuitively compelling they are, when they don't accord with established scientific theories and data) is deliberate. Rather than reading either sequence as an attempt to defend a specific fleshed-out theory of consciousness or of physical law, they should primarily be read as attempts to show that extreme uncertainty about a domain doesn't always bleed over into 'we don't know anything about this topic' or 'we can't rule out any of the candidate solutions'.

We can effectively rule out epiphenomenalism as a candidate solution to the hard problem of consciousness even if we don't know the answer to the hard problem (which we don't), and we can effectively rule out 'consciousness causes collapse' and 'there is no objective reality' as candidate solutions to the measurement problem in QM even if we don't know the answer to the measurement problem (which, again, we don't). Just advocating 'physicalism' or 'many worlds' is a promissory note, not a solution.

In discussions of EA and x-risk, we likewise need to be able to prioritize more promising hypotheses over less promising ones long before we've answered all the questions we'd like answered. Even deciding what studies to fund presupposes that we've 'philosophized', in the sense of mentally aggregating, heuristically analyzing, and drawing tentative conclusions from giant complicated accumulated-over-a-lifetime data sets.

You wrote:

The human capacity for abstract reasoning over mathematical models is in principle a fully general intelligent behaviour, as the scientific revolution has shown: there is no aspect of the natural world which has remained beyond the reach of human understanding, once a sufficient amount of evidence is available.

That's true, and it's one of the basic assumptions behind MIRI research: that understanding agents smarter than us isn't obviously hopeless, because our human capacity for abstract reasoning makes it possible for us to model systems even when they're extremely complex and dynamic. MIRI's research is intended to make this likelier to happen.

It's not the default that we're always able to predict what our inventions will do before we run them to see what happens; and there are some basic limits on our ability to do so when the system we're predicting is smarter than the predictor. But with enough intellectual progress we may become able to model abstract safety-relevant features of AGI behavior, even though we can't predict in detail the exact decisions the AGI will make. (If we could predict the exact decisions of the AGI, we'd have to be at least as smart as the AGI.)

If it isn't possible to learn a variety of generalizations about smarter autonomous systems, then, interestingly, that also undermines the case for intelligence explosion. Both 'humans trying to make superintelligent AI safe' and 'AI undergoing a series of recursive self-improvements' are cases where less intelligent agents are trying to reliably generate agents that meet various abstract criteria (including superior intelligence). The orthogonality thesis, likewise, simultaneously supports the claim 'many possible AI systems won't have humane goals' and 'it is possible for an AI system to have human goals'. This is why Bostrom/Yudkowsky-type arguments don't uniformly inspire pessimism.

Are you familiar with MIRI's technical agenda? You may also want to check out the AI Impacts project, if you think we should be prioritizing forecasting work at this point rather than object-level mathematical research.