All of Roko Jelavić's Comments + Replies

Within the pessimistic hypothesis it does not matter who develops AGI, in any case our death is almost certain.

1Not Relevant
The goal here should be social consensus about individual disincentives to misaligned deployment - stopping individual research labs has a pretty modest ROI. (If they pivot their attention given social consensus, that’s up to them.)

Yes but GPT-3 offers us new evidence we should try to update on. It's debatable to say how many bits of evidence that provides, but we can also update based on this Discontinuous progress in history: an update:

Growth rates sharply changed in many trends, and this seemed strongly associated with discontinuities. If you experience a discontinuity, it looks like there’s a good chance you’re hitting a new rate of progress, and should expect more of that.

AlphaGo was something we saw before we expected it. GPT-3 text generator was something we saw before we expected it. They were discontinuities.

3Adam Zerner
I agree. I'm not sure how much to update on the things you mention or on other things that have happened since 2013, so I think my answer serves as more of jumping off point than something authoritative. I edited it to mention that.

Thanks! From this the first impression is that testing is going to get easier, but actually it still depends on what the antibody test reagents are, do you have any info on that?

Lucky individuals often find something which has a long inference chain. Not a one-step improvement to existing theory, but many steps further. To the experts, it may look like a few new steps, combined masterfully. To the non-expert, they may need to learn about 100 new concepts for it to make sense. That's why 100 non-experts can't just invent general relativity, they need to take a 100 steps, 1 step each, but all in the same direction.

2khafra
How many dimensions is inference space? How many duck-sized horses do we need, to have a 2/3 chance of taking those steps? And are they being modeled as duck-sized monkeys with typewriters, or are they closer to a proper mini-Einstein, who is likely to go the correct direction?

You are correct, and that simpler model gives an even greater risk. I'm skeptical about social distancing because hospitals become overcrowded once 1/1000 of the population gets infected, and they need one month to process the hospitalized. With that pace, the quarantine would need to last 83 years. Even if this estimate is wrong by 10x that implies quarantine duration of 8 years. So much about flattening the curve. The best hope is a vaccine, so the quarantine lasts for approx 1 year, but maybe much shorter if more resources are invested and barriers... (read more)

1Steven Byrnes
Oops, sorry confusing my Roko's!! I was figuring there's 1 open hospital bed per 1000 people where I live (USA), which lets ~0.5% get infected per month. Are you sure it wasn't 0.1% hospitalized, rather than 0.1% infected? It doesn't really matter for this conversation, but it's still something I'd like to know. I think we're largely in agreement. The question is, over a year (antivirals could be faster, fingers crossed), will people get sick of the increasingly large piles of bodies and demand more social distancing, or will they get sick of social distancing and demand less of it? Or swing back and forth each month??? It's hard to say...

Perhaps I should have been more specific, I'm talking about a scenario where there is an actual machine (like a time machine but instead of travelling in time you travel between universes) in which you step and press a button, and then you appear in a parallel universe. In standard probability we have a potential future state of "I'm dead" and "I'm alive" but you can physically travel between those two future states, either one happens or the other happens. In the inter-universe travel scenario you can use the machine to ... (read more)

2Stuart_Armstrong
Before dealing with the implications of the surgeon, I have to understand the general implications of standard many-worlds. It's not clear to me what are the implications of uniformly doubling, or halving, your quantum measure. Until I know that, I don't know if quantum measure can be treated as probability.

Perhaps I should have been more specific, I'm talking about a scenario where there is an actual machine (like a time machine but instead of travelling in time you travel between universes) in which you step and press a button, and then you appear in a parallel universe. It's not a question who claims anything, nor it is a question of random fluctuations, it's a question of whether that kind of machine can be built or not. If it can be built, then increasing quantum diversification reduces xrisk, because then the travelers can travel around... (read more)

Can you please elaborate on your example of resurrection, it sounds interesting but I don't understand it.

2avturchin
The full explanation is in my draft "Classification of the Approaches to the Technological Resurrection" https://philpapers.org/rec/TURCOA-3 Almond suggested the following idea about the resurrection of the dead by the use of a quantum random generator, which create a random mind within a computer (Almond, 2006): If the many-worlds interpretation of quantum mechanics is true, when all possible minds will appear in different timelines starting from the moment of random mind creation, which would mean resurrection of everyone from his own point of view. However, this approach will a) not help an outside observer, who wants to resurrect a relative, for instance, as the observe would see only a random mind, and b) the "measure" of existence of each mind will be infinitely small. The first problem could be overcome by the use of relatives expectations as priors, for example, if I expect to resurrect John, I create all possible "John"s, and use random generator to generate all possible surnames (and all other personal data, which I will here ignore for the simplification of the problem). However, there is still the problem of quantum measure of existence decline, which, according to some authors, is the real problem with ideas like quantum immortality. Measure is, roughly speaking, the share of the worlds where I exist. If we use some expected utility calculations, measure decline results in declining utility of any useful outcome associated with it, so we could just ignore my copies with infinitely small measures. (This position itself is vulnerable, as it takes in account the absolute, but not relative measure. Absolute measure is my share between all possible observers, and relative measure is share of my copies with some property between all my future copies. For example, the relative share of all my future copies in the moment T=today evening, who will eat an apple is something like 0.001. But absolute share of my copies created by the quantum randomness gen

It changes because with ordinary randomness you can't travel between different branches in the decision tree. In the thought experiment with the surgeon he actually physically travels to a parallel universe and saves a life of his copy there. So the expected long term utility is not 1 life saved but 10 lives saved.

1Nebu
It's not clear to me that for all observers in our universe, there'd be a distinction between "a surgeon from a parallel universe suddenly appears in our universe, and that surgeon has memories of existing in a universe parallel to the one he now finds himself in." vs "a surgeon, via random quantum fluctuations, suddenly appears in our universe, and that surgeon has memories of existing in a universe parallel to the one he now finds himself in." In your example, rather than consider all infinitely many parallel universes, you chose to consider 10 specific universes where a surgeon appears and "claims" to have come from a parallel universe, and saves copies of himself. Even in a multiverse where travel between different quantum parallel universes is impossible, you can still find 10 universes where a surgeon appears and "claims" to have come from a parallel universe, and saves copies of himself. You can, in fact, find infinitely many universes where that happens, without requiring any travel between universes.