All of NullSetOwl's Comments + Replies

The universal doomsday argument still relies on SSA, because under SIA, I’m equally surprised to exist as an early person whether most civilizations are short or long. If most civilizations are long, I’m surprised to be early. If most civilizations are short, I’m surprised to exist at all. I could have been any of the late people who failed to exist because most civilizations are short. In other words, the surprise of existing as an early person is equivalent in both cases under SIA, so there's no update. Only under SSA am I certain I will exist but unsure where in the universe I will be.

I tend to explain wages in terms of ease of replacement. Companies will only pay a lot for something they can’t get for cheaper. If AI makes it possible for more people to code, then coders are easier to replace and wages should go down. For entry level jobs this effect is clear, but for senior positions it depends on how easy it is for an employee to get these productivity boosts with AI. Right now there’s a spectrum where almost anyone can make a simple website with AI, but beyond that people start to get filtered out. I expect the distribution to increa... (read more)

I think it’s an important caveat that this is meant for early AGI with human-expert-level capabilities, which means we can detect misalignment as it manifests in small-scale problems. When capabilities are weak, the difference between alignment and alignment-faking is less relevant because the model’s options are more limited. But once we scale to more capable systems, the difference becomes critical.

Whether this approach helps in the long term depends on how much the model internalizes the corrections, as opposed to just updating its in-distribution behav... (read more)

As you mention, the three examples here work regardless of whether SSA or SIA is true because none of the estimated outcomes affect the total number of observers. But the Doomsday Argument is different and does depend on SSA. If SIA is true, the early population of a long world is just as likely to exist as the total population of a short world, so there’s no update upon finding yourself in an early-seeming world.

A total utilitarian observing from outside both worlds will care just as much about the early population of a long world as the total population ... (read more)