LESSWRONG
LW

All of Sjlver's Comments + Replies

Vegan Nutrition Testing Project: Interim Report
Sjlver20

How does urine testing for iron deficiency work? I was under the impression that one needs to measure blood ferritin levels to get a meaningful result... but I might be wrong?

Reply
2Elizabeth
They're looking for byproducts, but it indeed didn't work very well and that's why I've refocused on blood testing.
What does it take to defend the world against out-of-control AGIs?
Sjlver10

Absolutely agree that the diagram omits lots of good parts from your post!

Is "do big damage" above or below human level? It probably depends on some unspecified assumptions, like whether we include groups of humans or just individuals...; anyway, the difficulty can't be much higher, since we have examples like nuclear weapon states that reach it. It can't be much lower, either, since it is lower-bounded by the ability of "callously-power-seeking humans". Somewhere in that range.

What I like about the diagram:

  • It visualizes the dangerous zone between "do big
... (read more)
Reply
What does it take to defend the world against out-of-control AGIs?
Sjlver*40

I tried to make a summary of how I understood your post, and systemize it a bit. I hope this is helpful for others, but I also want to say that I really liked the many examples in the post.

The central points that I understood from what you wrote:

  • It takes more power to prevent big damage than to do big damage. You give several examples where preventing extinction requires solutions that are illegal or way out of the Overton window, and examples like engineered pandemics that seem easier.
    I believe this point is basically correct for a variety of reasons, fro
... (read more)
Reply
5Steven Byrnes
Thanks! Your diagram does describe one of the things on my mind. One change would be: I would put the “do big damage” below human level. For example, making and releasing novel deadly pandemics seems terrifyingly straightforward (I’m not an expert though). That’s why EA people like Toby Ord often describe engineered pandemics as a serious extinction-level threat even in the absence of AGI (ref). For example, it’s at least plausible that we have COVID-19 because of a lab leak, i.e. well-meaning researchers making a mistake. If things like COVID-19 can plausibly happen by accident, just imagine what could happen with deliberate effort! (Then the argument also involves saying that AGIs are more likely to want deadly pandemics than smart humans, which I currently believe for reasons in Section 3.3.3 but is far from obvious and uncontroversial.) Another thing is that your diagram just uses the word “power” but there are many contributors to that. The obvious one is computational power / insight / smarts / experience / etc., but I’m also thinking a lot about “not being constrained by needing to do things that the human supervisors would approve of” as a source of power, and one which seems important to me. You mention that in your text. “Global agreement against ever building AGIs that use more than X flops” sounds like an interesting idea but I don’t tend to spend time thinking about it because it seems extraordinarily unlikely to happen in the real world. (Well, the agreement might happen, but I don’t expect it to be followed through.) Unless the X is very large, way beyond the point at which AGIs can be terrifyingly capable. I’m not an expert here and if someone is working on it, I certainly wouldn’t want to discourage them.
Lotteries: A Waste of Hope
Sjlver50

Someone once told me that the odds of winning the lottery are smaller than the odds of dying before the draw. Thanks to this insight, lottery tickets have become a kind of "memento mori" for me, and I'm no longer tempted to buy any.

Reply
2[comment deleted]