Good points. However: (1) Most of the cataclysms we see are either fairly explicable (supernovae) or seem to occur only at remote points in spacetime, early in the evolution of the universe, when the emergence of intelligent life would have been very unlikely. Quasars and gamma ray bursts cannot plausibly be industrial accidents in my opinion, and supernovae need not be industrial accidents.

(2)Possible, but I can still imagine large civilizations of people whose utility function is weighted such that "99.9999% death plus 0.0001% superman" is inferior to "continued mortal existence."

(3)Again possible, but there will be a selection effect over time. Eventually, the remaining people (who, you will notice, live in a universe where people who try to ascend to godhood always die) will no longer think ascending to godhood is a good idea. Maybe the ancients were right and there really is a small chance that the ascent process works and doesn't kill you, but you have never seen it work, and you have seen your civilization nearly exterminated by the power-hungry fools who tried it the last ten times.

At what point do you decide that it's more likely that the ancients did the math wrong and the procedure just flat out does not work?

(4)The minority might have no problems with risks that do not have a track record of killing everybody. However, you have a point: a rational civilization that expects the galaxy to be heavily populated might be well advised to hide.

To our perspective, this is from (2): all advanced civilizations die off in massive industrial accidents; God alone knows what they thought they were trying to accomplish.

Also, wouldn't there still be people who chose to stay behind? Unless we're talking about something that blows up entire solar systems, it would remain possible for members of the advanced civilization to opt out of this very tempting choice. And I feel confident that for at least some civilizations, there will be people who refuse to bite and say "OK, you guys go inhabit a tiny subset of all universes as gods; we will stay behind and occupy all remaining universes as mortals."

If this process keeps going on for a while, you end up with a residual civilization composed overwhelmingly of people who harbor strong memes against taking extremely low-probability, high-payoff risks, even if the probability arithmetic indicates doing so.

For your proposal to work, it has to be an all-or-nothing thing that affects every member of the species, or affects a broad enough area that the people who aren't interested have no choice but to play along because there's no escape from the blast radius of the "might make you God, probably kills you" machine. The former is unlikely because it requires technomagic; the latter strikes me as possible only if it triggers events we could detect at long range.

This is my hypothesis (3c), with an implicit overlay of (3a).

Here goes:

Alternate explanations for rarity of intelligence:

3a) Interstellar travel is prohibitively difficult. The fact that the galaxy isn't obviously awash in intelligence is a sign that FTL travel is impossible or extremely unfeasible.

Barring technology indistinguishable from magic, building any kind of STL colonizer would involve a great investment of resources for a questionable return; intelligent beings might just look at the numbers and decide not to bother. At most, the typical modern civilization might send probes out to the nearest stellar neighbors. If the cost of sending a ton of cargo to Alpha Centauri is say, 0.0001% of your civilization's annual GDP, you're not likely to see anyone sending million-ton colony ships to Alpha Centauri. In which case intelligent life might be relatively common in the galaxy without any of it coming here; even the more ambitious cultures that actually did bother to make the trip to the nearest stars would tend to peter out over time rather than going through exponential expansion.

3b) Interstellar colonization is prohibitively difficult. If sending an STL colony expedition to another star is hard, sending one with a large enough logistics base to terraform a planet will be exponentially harder.

There are something on the order of 1000 stars within 50 to 60 light years of us. Assuming more or less uniform stellar densities, if the probability of a habitable planet appearing around any given star is much less than 0.1%, it's likely that such planets will remain permanently out of reach for a sublight colony ship. In that case, spreading one's civilization throughout the galaxy depends on being able to terraform planets across interstellar distances before setting up a large population on those worlds. Even if travel across short (~10 ly) interstellar distances is not prohibitively difficult, there might still be little or no incentive to colonize the available worlds beyond one's own star system. After all, if you're going to live in a climate-controlled bunker on an uninhabitable rock where you can't step outside without being freeze-dried or boiled alive, you might as well do it somewhere closer to home.

NOTE: This amounts to "super-difficult life," but it does not require that there are few intelligent species in the galaxy. If the emergence of life is (for lack of a better term) super-duper-difficult, or if most planets are inhospitable enough to make it impossible, then we could have many thousands of intelligent species in the galaxy without any of them being likely to reach each other.

3c) Interstellar colonization might be "psychologically" difficult. For instance, what if the next logical step in the evolution of modern civilization is an AI singularity, possibly coupled with some kind of uploading of consciousness into machines? Either way, our descendants of 200 years from now might well be, to our eyes, a civilization of robots. To a society of strong AIs, interstellar colonization is liable to look a little different. Traveling to even the nearest stars, you will be cut off from the rest of your civilization by a transmission gap on the order of 10^20 cycles just because of the lightspeed limit.*

That might sound like an even worse idea to them than spending a long lifetime in cryogenic storage and having a twenty year round trip communication cycle with Earth does to us. In which case they're likely to stay at home and come up with elaborate social activities or simulations to spend their time, because interstellar colonization is just too unpleasant to bear considering.

*Assuming roughly 1 THz computing, for relatively near stellar neighbors. This estimate is probably too low, but I need some numbers and I am nowhere near an expert on artificial intelligence or the probable limits of computer technology.

I dunno. I mean, a lot of horror stories that are famous for being good talk about stuff that can never be and should never be, but that nonetheless (in-story) is. I think it's that sense of a comforting belief about the world being violated that makes a good horror story, even if the prior probability of that belief being wrong is low.

I think you're misreading the story. It's not an argument in favor of irrationality, it's a horror story. The catch is that it's a good horror story, directed at the rationalist community. Like most good horror stories, it plays off a specific fear of its audience.

You may be immune to the lingering dread created by looking at all those foolish happy people around you and wondering if maybe you are the one doing something wrong. Or the fear that even if you act as rationally as you can, you could still box yourself into a trap you won't be able to think your way back out of. But quite a few of your peers are not so immune. I know I'm not, and that story managed to scare me pretty effectively.

The protagonist isn't an ideal rationalist, and the story isn't trying to assert that this is what the ideal rationalist does. Instead, the protagonist is an adolescent proto-rationalist, of a type many of us are familiar with, with her social instincts sucking her into a trap that a lot of us can understand well enough to dread.

And so there's a reason she thinks and acts like a Hollywood stereotype of an intelligent person is that, especially when they're just barely at the age of being able to really think at all. Where do you think Hollywood got the idea for the stereotype in the first place?

I submit that the reason so many of the average people think intelligent people act that way is because they lose social contact with the geniuses in high school, which is when they do think and act like that.

For a lot of the smartest people, being socially functional is a learned skill that comes late and not easily.

Countercounterevidence for 3: what are the assumptions made by those models of interstellar colonization?

Do they assume fusion power? We don't know if industrial fusion power works economically enough to power starships. Likewise for nanotech-type von Neumann machines and other tools of space colonization.

The adjustable parameters in any model for interstellar colonization are defined by the limits of capability for a technological civilization. And we don't actually know the limits, because we haven't gotten close enough to those limits to probe them yet. If the future looks like the more optimistic hard science fiction authors suggest, then the galaxy should be full of intelligence and we should be able to spot the drive flares of Orion-powered ships flitting around, or the construction of Dyson spheres by the more ambitious species. We should be able to see something, at any rate.

But if the future doesn't look like that, if there's no way to build cost-effective fusion reactors and the only really worthwhile sustainable power source is solar, if there are hard limits on what nanotech is capable of that limit its industrial applications, and so on... the barrier to entry for a planetary civilization hoping to go galactic may be so high that even with thousands of intelligent species to make the attempt, none of them make it.

This ties back into the hypotheses I left out of my post for the sake of brevity; I'm now considering throwing them in to explain my reasoning a little better. But I'm still not sure I should do it without invitation, because they are on the long side.