All of magic9mushroom's Comments + Replies

You're encouraged to write a self-review, exploring how you think about the post today. Do you still endorse it? Have you learned anything new that adds more depth? How might you improve the post? What further work do you think should be done exploring the ideas here?

Still endorse. Learning about SIA/SSA from the comments was interesting. Timeless but not directly useful, testable or actionable.

There is no war in the run-up to AGI that would derail the project, e.g. by necessitating that most resources be used for capabilities instead of safety research.

Assuming short timelines, I think it’s likely impossible to reach my desired levels of safety culture.

I feel obliged to note that a nuclear war, by dint of EMPs wiping out the power grid, would likely remove private AI companies as a thing for a while, thus deleting their current culture. It would also lengthen timelines.

Certainly not ideal in its own right, though.

There are a couple of things that are making me really nervous about the idea of donating:

1. "AI safety" is TTBOMK a broad term and encompasses prosaic alignment as well as governance. I am of the strong opinion that prosaic alignment is a blind alley that's mostly either wasted effort or actively harmful due to producing fake alignment that makes people not abandon neural nets. ~97% of my P(not doom) routes through Butlerian Jihad against neural nets (with or without a nuclear war buying us more time) that lasts long enough to build GOFAI. And frankly, I don

>Second, I imagine that such a near-miss would make Demis Hassabis etc. less likely to build and use AGIs in an aggressive pivotal-act-type way. Instead, I think there would be very strong internal and external pressures (employees, government scrutiny, public scrutiny) preventing him and others from doing much of anything with AGIs at all.

I feel I should note that while this does indeed form part of a debunk of the "good guy with an AGI" idea, it is in and of itself a possible reason for hope. After all, if nobody anywhere dares to make AGI, well,... (read more)

I've got to admit, I look at most of these and say "you're treating the social discomfort as something immutable to be routed around, rather than something to be fixed by establishing different norms". Forgive me, but it strikes me (especially in this kind of community with high aspie proportion) that it's probably easier to tutor the... insufficiently-assertive... in how to stand up for themselves in Ask Culture than it is to tutor the aspies in how to not set everything on fire in Guess Culture.

Amusingly, "rare earths" are actually concentrated in the crust compared to universal abundance and thus would make awful candidates for asteroid mining, while "tellurium", literally named after the Earth, is an atmophile/siderophile element with extreme depletion in the crust and one of the best candidates.

It strikes me that I'm not sure whether I'd prefer to lose $20,000 or have my jaw broken. I'm pretty sure I'd prefer to have my jaw broken than to lose $200,000, though. So, especially in the case that the money cannot actually be extracted back from the thief, I would tend to think the $200,000 theft should be punished more harshly than the jaw-breaking. And, sure, you've said that the $20,000 would be punished more harshly than the jaw-breaker, but that's plausibly just because 2 days is too long for a $100 theft to begin with.

I mean, most moral theories do either give the answers of "zero", "as large as can be fed", or "a bit less than as large as can be fed". Given the potential to scale feeding in the future, the latter two round off to "infinity".

I think the basic assumed argument here (though I'm not sure where or even if I've seen it explicitly laid out) goes essentially like this:

• Using neural nets is more like the immune system's "generate everything and filter out what doesn't work" than it is like normal coding or construction. And there are limits on how much you can tamper with this, because the whole point of neural nets is that humans don't know how to write code as good as neural nets - if we knew how to write such code deliberately, we wouldn't need to use neural nets in the first place.

The earliness of life appearing on Earth isn't amazingly-consistent with life's appearance on Earth being a filter-break. It suggests either abiogenesis is relatively-easy or that panspermia is easy (as I noted, in the latter case abiogenesis could be as hard as you like but that doesn't explain the Great Silence).

Frankly, it's premature to be certain it's "abiogenesis rare, no panspermia" before we've even got a close look at Earthlike exoplanets.

I'll note that most of the theorised catastrophes in that vein look like either "planet gets ice-nined", "local star goes nova", or "blast wave propagates at lightspeed forever". The first two of those are relatively-easy to work around for an intelligent singleton, and the last doesn't explain the Fermi observation since any instance of that in our past lightcone would have destroyed Earth.

I've read most of that paper (I think I've seen it before, although there could be something else near-identical to it; I know I've read multiple^[1] papers that claim to solve the Fermi Paradox and do not live up to their hype). TBH, I feel like it can be summed up as "well, there might be a Great Filter somewhere along the line, therefore no paradox". I mean, no shit there's probably a Great Filter somewhere, that's the generally-proposed resolution that's been going for decades now. The question is "what is the Filter?". And saying "a Filter exists"... (read more)

There is also the possibility of the parties competing over it to avoid looking "soft on AI", which is of course the ideal.

To the extent that AI X-risk has the potential to become partisan, my general impression is that the more likely split is Yuddite-right vs. technophile-left. Note that it was a Fox News reporter who put the question to the White House Press Secretary following Eliezer's TIME article, and a Republican (John Kennedy) who talked about X-risk in the Senate hearing in May, while the Blue-Tribe thinkpieces typically take pains to note that t... (read more)

>The fourth thing Bostrom says is that we will eventually face other existential risks, and AGI could help prevent them. No argument here, I hope everyone agrees, and that we are fully talking price.

>It is not sufficient to choose the ‘right level of concern about AI’ by turning the dial of progress. If we turn it too far down, we probably get ourselves killed. If we turn it too far up, it might be a long time before we ever build AGI, and we could lose out on a lot of mundane utility, face a declining economy and be vulnerable over time to other exi... (read more)

Should be noted that while there are indeed tons of people who will fault you for taking steps to survive GCR, in the aftermath of a GCR most of those people will be dead (or at the very least, hypocrites who did the thing they're upset about) and thus not able to fault you for anything. History is written by, if not the winners, at least the survivors.

Admittedly, this is contingent on the GCR happening, but I think there's a pretty-high chance of nuclear war in particular in the near future (the Paul Symon interview in particular has me spooked; a random ... (read more)

I guess it's a claim that advanced civilizations don't hit K2, because they prefer to live in virtual worlds, and have little interest in expanding as fast as possible.

This would be hard. You would need active regulations against designer babies and/or reproduction.

Because, well, suppose 99.9% of your population wants to veg out in the Land of Infinite Fun. The other 0.1% thinks a good use of its time is popping out as many babies as possible. Maybe they can't make sure their offspring agree with this (hence the mention of regulations against designer babi... (read more)

Your scenario does not depend on FTL.

However, its interaction with the Doomsday Argument is more complicated and potentially weaker (assuming you accept the Doomsday Argument at all). This is because P(we live in a Kardashev ~0.85 civilisation) depends strongly in this scenario on the per-civilisation P(Doom before Kardashev 2); if the latter is importantly different from 1 (even 0.9999), then the vast majority of people still live in K2 civilisations and us being in a Kardashev ~0.85 civilisation is still very unlikely (though less unlikely than it would ... (read more)

Pardon my ignorance; I don't actually know what SIA and SSA stand for.