Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Stuart_Armstrong 21 October 2014 09:10:13PM 0 points [-]

Ok, I don't like gnomes making current decisions based on their future values. Let's make it simpler: the gnomes have a utility function linear in the money owned by person X. Person X will be the person who appears in their (the gnome's) room, or, if no-one appeared, some other entity irrelevant to the experiment.

So now the gnomes have subjectively indistinguishable utility functions, and know they will reach the same decision upon seeing "their" human. What should this decision be?

If they advise "buy the ticket for price $x", then they expect to lose $x with probability 1/4 (heads world, they see a human), lose/gain nothing with probability 1/4 (heads world, they don't see a human), and gain $1-x with probability 1/2 (tails world). So this gives an expected gain of 1/2-(3/4)x, which is zero for x=$2/3.

So this seems to confirm your point.

"Not so fast!" shouts a voice in the back of my head. That second head-world gnome, the one who never sees a human, is a strange one. If this model is vulnerable, it's there.

So let's do without gnomes for a second. The incubator always creates two people, but in the heads world, the second person can never gain (nor lose) anything, no matter what they agree to: any deal is nullified. This seems a gnome setup without the gnomes. If everyone is an average utilitarian, then they will behave exactly as the total utilitarians would (since population is equal anyway) and buy the ticket for x<$2/3. So this setup has changed the outcome for average utilitarians. If its the same as the gnome setup (and it seems to be) then the gnome setup is interfering with the decisions in cases we know about. The fact that the number of gnomes is fixed is the likely cause.

I'll think more about it, and post tomorrow. Incidentally, one reason for the selfish=average utilitarian is that I sometimes model selfish as the average between total utilitarian incubator and anti-incubator (where the two copies hate each other in the tail world). 50%-50% on total utilitarian vs hatred seems to be a good model of selfishness, and gives the x<$1/2 answer.

Comment author: Stuart_Armstrong 21 October 2014 08:36:58PM 0 points [-]

Thanks for engaging with my paper ^_^ I will think about your post and construct a more detailed answer.

The consensus view on LW seems to be that much of the SSA vs. SIA debate is confused and due to discussing probabilities detached from decision problems of agents with specific utility functions.

Really? That's my view, but I didn't know it had spread!

Comment author: Arenamontanus 21 October 2014 07:44:00PM 3 points [-]

Neat. The minimal example would be if each risk had 50% chance of happening: then the observable correlation coefficient would be -0.5 (not -1, since there is 1/3 chance to get neither risk). If the chance of no disaster happening is N/(N+2), then the correlation will be -1/(N+1).

It is interesting to note that many insurance copula methods are used to make size-dependent correlations, but these are nearly always of the type of stronger positive correlations in the tail. This suggests - unsurprisingly - that insurance does not encounter much anthropic risk.

Comment author: Stuart_Armstrong 21 October 2014 07:49:11PM 0 points [-]

When I read this, my first reaction was "I have to show this comment to Anders" ^_^

Comment author: gwern 21 October 2014 07:15:09PM 2 points [-]

The effect is even clearer if we have a probabilistic relation between pandemics, recessions and extinction (something like: extinction risk proportional to product of recession size times pandemic size). Then we would see an anti-correlation rising smoothly with intensity.

So something like the plot of asteroid impact sizes vs time in "The Anthropic Shadow" where the upper-right corner is empty?

Comment author: Stuart_Armstrong 21 October 2014 07:34:25PM 0 points [-]

Similar in that one quadrant is empty, otherwise a distinct effect.

Comment author: TheAncientGeek 21 October 2014 07:04:08PM *  0 points [-]

Would resources have been of use in fighting pandemics before modern medicine?

Comment author: Stuart_Armstrong 21 October 2014 07:33:55PM 0 points [-]

The pandemic/recession example is almost certainly wrong; it was just an illustration of the concept.

Anthropic signature: strange anti-correlations

14 Stuart_Armstrong 21 October 2014 04:59PM

Imagine that the only way that civilization could be destroyed was by a large pandemic that occurred at the same time as a large recession, so that governments and other organisations were too weakened to address the pandemic properly.

Then if we looked at the past, as observers in a non-destroyed civilization, what would we expect to see? We could see years with no pandemics or no recessions; we could see mild pandemics, mild recessions, or combinations of the two; we could see large pandemics with no or mild recessions; or we could see large recessions with no or mild pandemics. We wouldn't see large pandemics combined with large recessions, as that would have caused us to never come into existence. These are the only things ruled out by anthropic effects.

Assume that pandemics and recessions are independent (at least, in any given year) in terms of "objective" (non-anthropic) probabilities. Then what would we see? We would see that pandemics and recessions appear to be independent when either of them are of small intensity. But as the intensity rose, they would start to become anti-correlated, with a large version of one completely precluding a large version of the other.

The effect is even clearer if we have a probabilistic relation between pandemics, recessions and extinction (something like: extinction risk proportional to product of recession size times pandemic size). Then we would see an anti-correlation rising smoothly with intensity.

Thus one way of looking for anthropic effects in humanity's past is to look for different classes of incidents that are uncorrelated at small magnitude, and anti-correlated at large magnitudes. More generally, to look for different classes of incidents where the correlation changes at different magnitudes - without any obvious reasons. Than might be the signature of an anthropic disaster we missed - or rather, that missed us.

Comment author: Strilanc 21 October 2014 12:07:24AM *  1 point [-]

For instance, if anything dangerous approached the AIXI's location, the human could lower the AIXI's reward, until it became very effective at deflecting danger. The more variety of things that could potentially threaten the AIXI, the more likely it is to construct plans of actions that contain behaviours that look a lot like "defend myself." [...]

It seems like you're just hardcoding the behavior, trying to get a human to cover all the cases for AIXI instead of modifying AIXI to deal with the general problem itself.

I get that you're hoping it will infer the general problem, but nothing stops it from learning a related rule like "Human sensing danger is bad.". Since humans are imperfect at sensing danger, that rule will better predict what's happening compared to the actual danger you want AIXI to model. Then it removes your fear and experiments with nuclear weapons. Hurray!

Comment author: Stuart_Armstrong 21 October 2014 02:27:43PM 0 points [-]

Very valid point.

Comment author: djm 21 October 2014 01:02:15PM 0 points [-]

Therefore, they cannot identify "that computer running the code" with "me", and would cheerfully destroy themselves in the pursuit of their goals/reward.

I am curious as to why an AIXI like entity would need to model itself (and all its possible calculations) in order to differentiate the code it is running with the external universe.

The human in charge of a reward channel could work for initial versions, but once its intelligence grew wouldn't it know what was happening (like the box AI example - not likely to work in the long term).

Comment author: Stuart_Armstrong 21 October 2014 02:25:49PM 0 points [-]

I am curious as to why an AIXI like entity would need to model itself (and all its possible calculations) in order to differentiate the code it is running with the external universe.

See other posts on this problem (some of them are linked to in the post above).

The human in charge of a reward channel could work for initial versions, but once its intelligence grew wouldn't it know what was happening

At this point, the "hope" is that the AIXI will have made sufficient generalisations to keep it going.

Comment author: PhilGoetz 20 October 2014 06:46:03PM 0 points [-]

Therefore, they cannot identify "that computer running the code" with "me", and would cheerfully destroy themselves in the pursuit of their goals/reward.

Why do you think that? They would just use a deictic reference. That's what knowledge representation systems have done for the past 30 years.

Comment author: Stuart_Armstrong 21 October 2014 02:23:26PM 0 points [-]

Do you have a way of tweaking the AIXI or AIXI(tl) equation so that that could be accomplished?

Comment author: sbenthall 21 October 2014 12:08:24AM 1 point [-]

Thanks. That criticism makes sense to me. You put the point very concretely.

What do you think of the use of optimization power in arguments about takeoff speed and x-risk?

Or do you have a different research agenda altogether?

Comment author: Stuart_Armstrong 21 October 2014 02:21:49PM 0 points [-]

As an informal concept, it's a good one (and better than "intelligence"). Just as long as its not taken too literally.

View more: Next