George Bernard Shaw wrote that, "the single biggest problem in communication is the illusion that it has taken place". Much of strawmanning is unconscious. One person says that it is important to be positive, the other person interprets this as it being important to be positive in *all* circumstances, when they are merely making a general statement.

I would suggest that a technique to avoid accidentally strawmanning someone would be to begin by intentionally strawmanning them and then try to back off to something more moderate from there. 

Take for example:

"Just be yourself"

A strawman would be, "Even if you are a serial killer, you should focus on being yourself, than changing how you behave".

Since this is a rather extreme strawman, backing off to something more moderate from here would be too easy. We might very well just back off to another strawman. Instead, we should backoff to a more reasonable strawman first, then backoff to the moderate version of their view.

The more moderate strawman, "You should never change how you act in order to better fit in"

When we back off to something more moderate, we then get, "Changing how you act in order to better fit in is generally not worth it"

You can then respond to the more moderate view. If you had responded to the original, you might have pointed out a single case when the principle didn't hold, such as making a change that didn't affect one's individuality (i.e showering regularly) and used it to attack the more general principle. When you have the more moderate principle, you can see that such a single example only negates the strict reading, not the more moderate reading. You can then either accept the moderate reading or add arguments about why you also disagree with it. If you had skipped this process, you might have made a specific critique and not realised that it didn't completely negate the other person's argument.

ohn_Maxwell_IV argued that revitalising Less Wrong is a lost purpose. I'm also very skeptical about Less Wrong 2.0 - but I wouldn't agree with it being a lost purpose. It is just that we are currently not on a track to anywhere. The #LW_code_renovation channel resulted in a couple of minor code changes, but there hasn't been any discussion for at least a month. All that this means, however, is that if we want a better less wrong that we have to do something other than what we have been doing so far. Here are some suggestions.

Systematic changes, not content production

The key problem currently is the lack of content, so the most immediate solution is to produce more content. However, not many people are an Elizier or a Scott. Think about what percentage of blog are actually successful - now throw on the extra limitation of having to be on topic on Less Wrong. Note that many of Scott's most popular posts would be too political to be posted on Less Wrong. Trying to get a group of people together to post content on Less Wrong wouldn't work. Let's say 10 people agreed to join such a group. 5 would end up doing nothing, 3 would do 2-3 posts and it'd fall on the last 2 to drive the site. The odds would be strongly against them. Most people can't consistently pump out high quality content.

The plan to get people to return to Less Wrong and post here won't work either unless combined with changes. Presumably, people have moved to their own blogs for a reason. Why would they come back to posting on Less Wrong, unless something was changed? We might be able to convince some people to make a few posts here, but we aren't going to return the community to its glory days without consistent content.

Why not try to change how the system is set up instead to encourage more content?

Decide on a direction

We now have a huge list of potential changes, but we don't have a direction. Some of those changes would help bring in more content and solve the key issue, while other changes wouldn't. The problem is that there is currently no consensus on what needs to be done. This makes it so much less likely that anything will actually get done, particularly given that it isn't clear whether a particular change would be approved or not if someone did actually do it. At the moment, what we have is people coming on to the site suggesting features and there is discussion, but there isn't anyone or any group in charge to say if you implement this that we would use it. So people will often never start these projects.

Before we can even tackle the problem of getting things done, we need to tackle the problem of what needs to be done. The current system of people simply making posts in discussion in broken - we never even get to the consensus stage, let alone implementation. I'm still thinking about the best way to resolve this, I think I'll post more about this in a future post. Regardless, practically *any* system, would be better than what we have now where there is *no* decision that is ever made.

Below I'll suggest what I think our direction should be:

Positions

Less Wrong is the website for global movement and has a high number of programmers, yet some societies in my university are more capable of getting things done than we are. Part of the reason is that university societies have positions - people decide to run for a position and this grants them status, but also creates responsibilities. At the moment, we have *no-one* working on adding features the website. We'd actually be better off if we held an election for the position of webmaster and *only* had that person working on the website. I'm not saying we should restrict a single person to being able to contribute code for our website, I'm just saying that *right now* implementing this stupid policy would actually improve things. I imagine that there would be at least *one* decent programmer for whom the status would be worth the work given that half the people here seem to be programmers.

Links

If we want more content, then an easy way would be to have a links section, because posting a link is about 1% of the effort of trying to write a Less Wrong post. In order to avoid diluting discussion, these links would have to be posted in their own section. Given that this system is based upon Reddit, this should be super easy.

Sections

The other easy way to generate more content would be to change the rules about what content is on or off topic. This comes with risks - many people like the discussion section how it is. However, if a separate section was created, then people would be able to have these additional discussions without impacting how discussion works at the moment. Many people have argued for a tag system, but whether we simply create additional categories or use tags would be mostly irrelevant. If we have someone who is willing to build this system, then we can do it, if not, then we should just use another category. Given that there is already Main and Discussion I can't imagine that it would be that hard to add in another category of posts. There have been many, many suggestions of what categories we could have. If we just want to get something done, then the simplest thing is to add a single new category, Open, which has the same rules as the Open Threads that we are already running.

Halve downvotes

John_Maxwell_IV points out that too many posts are getting downvotes and critical comments. We could try to change the culture of Less Wrong, perhaps ask a high status individual like Scott or Elizier to request people to be less critical. And that might even work for even a week or a month, before people forget about it. Or we could just halve downvotes. While not completely trivial, this change would be about as simple as they come. We might want to only halve downvotes on articles, not comments, because we seem to get enough comments already, just not enough content. I don't think it'll lower the quality of content too much - quite often there are more people who would downvote a post, but they don't bother because the content is already below zero. I think this might be worth a go - I see a high potential upside, but not much in the way of downside.

Crowdsourcing

If we could determine that a particular set of features would have a reasonable chance of improving LessWrong, then we could crowd-source putting a bounty on someone implementing these features. I suspect that there are many people who'd be happy to donate some money and if we chose simple, well defined features, then it actually wouldn't be that expensive.

Once upon a time the countries of Alpago and Byzantine had a war. Alpago was mostly undamaged during this war. Byzantine was severely damaged by this war, although they have caught up in some metrics such as education, their economy is still somewhat weaker. Alpago was the clear aggressor, and now, fifty years later, everyone who is reasonable now acknowledges that Alpago was in the wrong. 

There is a major debate within the countries about how to respond to the past. Many Byzantians argue that the views of the Alpagoans are irrelevant. The Alpagoans are "unbombed", this provides them with many systematic advantage over the Byzantians such as career opportunities, indeed most of the top companies in Byzantian still have Alpagoan CEOs since many of the senior management were hired before Byzantian had built anywhere near the number of colleges in Alpago.

Many Byzantians argue that the views of the unbombed deserve very little consideration. Of course the unbombed will want to preserve their advantages. How can the Byzantians ever have their voices heard when unbombed members of parliament are giving their opinions in the Alpago parliament on how much compensation is appropriate? Surely if Alpago was truly sorry, they would accept the demands of the Byzantian government without question.

The Byzantians are undoubtedly correct in their assumption that the Alpagoans have a very strong incentive to underestimate what is owed. They are also correct when they say that the Alpagoans are in a position of power that makes it very easy for them to ignore the issue of compensation, after all, it does not affect them very much if their government decides to pay compensation to the Byzantians, instead of the alternate plan of wasting it on a fleet of nuclear submarines. However, in other areas, the Alpagoans no longer have a power advantage. Many Alpagoan politicians used to say that the war was justified, if a politician said that these days, even the conservative party would demand that they resign because no reasonable person could come to such a conclusion.

In contrast, some of the more extreme Byzantians regularly declare the burning of their capital as a intentional war crime, while the evidence quite clearly shows that the Alpagoans had not targeted their civilian population, only their military base which had inadvertently led to the fire when it was destroyed. During the war, the intentional targetting theory was best supported by the evidence available to the Alpagoans, but advance in forensics have long ago disproven this theory. Many Byzantines consider this forensic technique discredited, because it was originally used to blame the war on the Byzantines. The reason why the Alpagoans did not burn the city was not altruistic. They did not want to burn the city merely because this would make it impossible for them to loot it. It is politically risky for an Alpagoan to point out that the burning was unintentional, since they might be mistaken for a member of the Alpagoan Pillorying Club. These are really legitimately horrible people (even the conservative party consider them to be bigots).

On the other hand, the Alpagoans almost universally insist that they never executed any Byzantine civilians in the brief period that they occupied the country. There are extensive interviews with numerous witnesses who saw this happen with their own eyes, but no hard evidence. The Alpagoans dismiss these accounts as it is impossible for them to conceive that criminals might be telling the truth when their own soldiers (whom they consider honorable - they blame politicians for the war) deny this ever happened. Any Byzantine who mentions this immediately gets dismissed as a "loony conspiracy nut".

If the Byzantians want to consider the incentives of the Alpagoans, they need to also consider their own incentives, as they would be construed by a hardened cynic. They might argue that their incentives are to fight for justice as this would earn them respect, but the cynic would not accept this. The cynic would argue that their incentives are to fight for the maximal amount of compensation, even if a perfectly impartial judge decided that it should be X, their incentive would be to claim that it should be at least X + 1. These incentives exist, even if the Alpagoan government would never offer even half of X.

Some of the Alpagoan are motivated by conscious self-interest to preserve their advantages, while many more who are convinced that they support fair compensation are affect by unconscious self-interest bias. But, the cynic will believe that the Byzantians will have an incentive to position the effect of self-interest on the Alpagoans as greater than it is. The cynic will believe that similarly, some of the Byzantians will be motivated by conscious self-interest, and others by unconscious bias, all while completely convinced that they are being fair.

The Alpagoans are in a position of power when it comes to compensation. The Byzantians lack the ability to force them to pay it, so the resolution will most likely be on the terms of the Alpagoans. The cynic will note that the Byzantians have the incentive to position themselves as being in a position of power for all issues, even when they are the ones in the position of power, such as in relation to the claim that the Alpagoans had intentionally burned their capital. Many Byzantians know that the Alpagoans didn't actually intentionally try to burn their capital, but they see this as a technicality (they started an illegitimate war which resulted in the capital burning) and they do not want to get into an argument with their fellow Byzantians who *really* strongly believe this. Further disagreeing with other Byzantians would undermine their cause which they see as just. The cynic would note that this is a very easy argument for the Byzantians to make. It does not harm them if the actions of the Alpagoans are misrepresented, in fact it helps them. Further, there are social incentives to agree with their fellow Byzantians.

Even though the Alpagoans are correct that they didn't intentionally burn the city, many of them have formed their viewpoint out of self-interest. There is convincing historical evidence, but very few of them have actually seen this, nor do most of them have interest in checking it out as it might disprove their beliefs. Most Alpagoans would be unwilling to acknowledge this, as it would harm their credibility and by used as ammunition by Byzantian activists who believe that they burned it intentionally.

We can see that considering the incentives of all the parties will help both the Byzantians come to a better understanding regarding the situation. The same will be true for the Alpagoans - the Byzantians are right in that the Alpagoans are often unaware of their bias. On the other hand, if either group only considers the incentives of one of the parties, they will most likely come to a more biased conclusion than if they had considered the incentives of neither of the parties. For these purposes, it is very important that the cynic be maximally cynical, without actually being a conspiracy theorist, in order to reduce room for bias.

In my last post, I wrote about how the anthropic principle was often misapplied, that it could not be used within a single model, but only for comparing two or more models. This post will explain why I think that the anthropic principle is valid in every case where we aren't making those mistakes.

There have been many probability problems discussed on this site and one popular viewpoint is that probabilities cannot be discussed as existing by themselves, but only as existing in relation to a series of bets. Imagine that there are two worlds: World A has 10 people and World B has 100. Both worlds have a prior probability of 50% of being correct. Is it the case that World B should instead be given a 10:1 odds due to there being ten times the number of people and the anthropic principle? This sounds surprising, but I would say yes as you’d have to be paid 10 times as much from each person in World A who is correct in order for you to be indifferent between the two worlds. What this means is that if there is a bet that gains or loses you money according to whether you are in world A or world B, you should bet as though the probability of you being in world B is 10 times as much. That doesn’t quite show that the probability is 10:1, but it is rather close. I can’t actually remember the exact process/theorem in order to determine probabilities from betting odds. Can anyone link it to me?

Another way to show that the anthropic principle is probably correct is to note that if world A had 0 people instead, then there would be 100% of observing world B rather than world A. This doesn’t prove much, but it does prove that anthropic effects exist on some level. 

Suppose now that world A has 1 person and world B has 1 million people. Maybe you aren’t convinced that you are more likely to observe world B. Let’s consider an equivalent formulation where world A has 1 person who is extremely unobservant and only has a 1 in a million chance of noticing the giant floating A in world A and the other world has a single person, but this time with a 100% chance of noticing the giant floating B in their world. I think it is clear that it is more likely for you to notice a giant floating B than an A.

One more formulation is to have world A have 10 humans and 90 cyborgs and world B to have 100 humans. We can then ask about the probability of being in world B given that you are a human observing the world. It seems clear here that you have 10 times the probability of being in world B than world A given that you are a human. It seems that this should be equivalent to the original problem since the cyborgs don’t change anything. 

I admit that none of this is fully rigorous philosophical reasoning, but I thought that I’d post it anyway a) to get feedback b) to see if anyone denied the use of the anthropic principle in this way (not the way described in my last post), which would provide me with more motivation to try making all of this more formal.

Update: I thought it was worth adding that applying the anthropic principle to two models is really very similar to null hypothesis testing to determine if it is likely that a coin is biased. If there are a million people in one possible world, but only one in another, it would seem to be an amazing coincidence for you to be that one.

The Fine-tuned Universe Theory, according to Wikipedia is the belief that, "our universe is remarkably well suited for life, to a degree unlikely to happen by mere chance". It is typically used to argue that our universe must therefore be the result of Intelligent Design.

One of the most common counter-arguments to this view based on the Anthropic Principle. The argument is that if the conditions were not such that life would be possible, then we would not be able to observe this, as we would not be alive. Therefore, we shouldn't be surprised that the universe has favourable conditions.

I am going to argue that this particular application of the anthropic principle is in fact an incorrect way to deal with this problem. I'll begin first by explaining one way to deal with this problem; afterwards I will explain why the other way is incorrect.

Two model approach

We begin with two modes:

• Normal universe model: The universe has no bias towards supporting life
• Magic universe model: The universe is 100% biased towards supporting life

Alice notices that Earth survived the cold war. She asks Bob why that is. After all, so much more likely for Earth not to survive. Bob tells her that it's a silly question. The only reason she picked out Earth is that it's her home planet, which is because it survived the cold war. If Earth died and, say, Pandora survived, she (or rather someone else, because it's not going to be the same people) would be asking why Pandora survived the cold war. There's no coincidence.

Content Note: Highly abstract situation with existing infinities

This post will attempt to resolve the problem of infinities in utilitarianism. The arguments are very similar to an argument for total utilitarianism over other forms which I'll most likely write up at some point (my previous post was better as an argument against average utilitarianism, rather than an argument in favour of total utilitarianism).

In the Less Wrong Facebook group, Gabe Bf posted a challenge to save utilitarianism from the problem of infinities. The original problem is from by a paper by Nick Bostrom.

I believe that I have quite a good solution to this problem that allows us to systemise comparing infinite sets of utility, but this post focuses on justifying why we should take it to be axiomic that adding another person with positive utility is good and on why the results that seem to contradict this lack credibility. Let's call this the Addition Axiom or A. We can also consider the Finite Addition Axiom (only applies when we add utility into a universe with a finite number of people), call this A0.

Let's consider what other alternative axioms that we might want to take instead. One is the Infinite Indifference Axiom or I, that is that we should be indifferent if both options provide infinite total utility (of the same order of infinity). Another option would be the Remapping Axiom (or R), which would assert that if we can surjectively map a group of people G onto another group H so that each g from G is mapped onto a person h from H where u(g) >= u(h), then v(H) <= v(G) where v represents the value of a particular universe (it doesn't necessarily map onto the real numbers or represent a complete ordering). Using the Remapping Axiom twice implies that we should be indifferent between an infinite series of ones and the same series with a 0 at one spot. This means that the Remapping Axiom is incompatible with the Addition Axiom. We can also consider the Finite Remapping Axiom (R0) which is where we limit the Remapping Axiom to remapping a finite number of elements.

First, we need to determine what are good properties of a statement we wish to take as an axiom. This is my first time trying to establish an axiom so formally, so I will admit that this list is not going to be perfect.

• Uses well-understood and regular objects, properties or processes. If the components are not understood well, it is highly likely that our attempt to determine the truth of a statement will be misguided.
• An axiom close to the territory is more reliable than one in the map because it is very easy to make subtle errors when constructing a map.
• Leads to minimally weird consequences.
• Extends included axioms in a logical way. If the axiom is an extension of a simpler alternative axiom, then it should be intuitive that the result would extend to the larger set; there should be reasons to expect it to behave the same way.

Let's look first at the Infinite Indifference Axiom. Firstly, it deals purely with infinite objects, which are known to often behave irregularly and results in many problems in which there is no consensus. Secondly, it exists in the map to some extent (but not that much at all). In the territory, there are just objects, infinity is our attempt to transpose certain object configurations into a number system. Thirdly, it doesn't seem to extend from the finite numbers very well. If one situation provides 5 total utility and another provides 5 total utility, then it seems logical to treat them as the same as 5 is equal to 5. However, infinity doesn't seem to be equal to itself in the same way. Infinity plus 1 is still infinity. We can remove infinite dots from infinite dots and end up with 1 or 2 or 3... or infinity. Further, this axiom leads to the result that we are indifferent between someone with large positive utility being created and someone with large negative good being created. This is massively unintuitive, but I will admit it is subjective. I think this would make a very poor axiom, but it doesn't mean it is false (Pythagoras' Theorem would make a poor axiom too).

On the other hand, deciding between the Remapping Axiom and Addition Axiom will be much closer. On the first criteria I think the Addition Axiom comes out ahead. It involves making only a single change to the situation, a primitive change if you will. In contrast, the Remapping Axiom involves Remapping an infinite number of objects. This is still a relatively simple change, but it is definitely more complicated and permutations of infinite series are well known to behave weirdly.

On the second criteria, the Addition Axiom (by itself) doesn't lead to any really weird results. We'll get some weird results in subsequent posts, but that's because we are going to going to make some very weird changes to the situation, not because of the Addition Axiom itself. The failure of the Remapping Axion could very well be because mappings lack the resolution to distinguish between different situations. We know that an infinite series can map onto itself, half of itself or itself twice, which lends a huge amount of support to the lack of resolution theory.

On the other hand, the Addition Axiom being false (because we've assumes the Remapping Axiom) is truly bizarre. It basically states that good things are good. Nonetheless, while this may seem very convincing to me, people's intuitions vary so the more relevant material for people with a different intuition is the material above that suggests the Remapping Axiom lacks resolution.

On the third criteria, a new object appearing is something that can occur in the territory. Infinite remappings initially seem to be more in the map than the territory, but it is very easy to imagine a group of objects moving one space to the right, so this assertion seems unjustified. That is, infinity is in the map as discussed before, but an infinite group of objects and their movements can still be in the territory. However, when we think about it again, we see that we have reduced the infinite group of objects X, to a set objects positioned, for example, on X = 0, 1, 2... This is a massive hint about the content of my following posts.

Lastly, the Addition Axiom in infinite case is a natural extension of the Finite Addition Axiom. In A0 the principle is that whatever else happens in the universe is irrelevant and there is no reason for this to change in the infinite case. For the Remapping Axiom, it also seems like a very natural extension of the finite case, so I'll call this criteria a draw.

In summary, if you don't already find the Addition Axiom more intuitive than the Remapping Axiom, the main reasons to favour the Addition Axiom are 1) it deals with better understood objects, 2) it is closer to the territory than the map 3) there are good reasons to suspect that Remapping lacks resolution. Of these reasons, I believe the the 3rd is by far the most persuasive; I consider the other two more to be hints than anything else.

I only dealt with the Infinite Indifference Axiom and the Remapping Axioms, but I'm sure other people will suggest their own alternative Axioms which need to be compared.

Increasing a person's utility, instead of creating a new person with positive utility is exactly the same. Also, this post is just the start. I will provide a systematic analysis of infinite universes over the coming days, plus an FAQ conditional on sufficient high quality questions.

In order to ensure that this post delivers what it promises, I have added the following content warnings:

Content Notes:

Pure Hypothetical Situation: The claim that perfect theoretical rationality doesn't exist is restricted to a purely hypothetical situation. No claim is being made that this applies to the real world. If you are only interested in how things apply to the real world, then you may be disappointed to find out that this is an exercise left to the reader.

Technicality Only Post: This post argues that perfectly theoretical rationality doesn't exist due to a technicality. If you were hoping for this post to deliver more, well, you'll probably be disappointed.

Contentious Definition: This post (roughly) defines perfect rationality as the ability to maximise utility. This is based on Wikipedia, which defines rational agents as an agent that: "always chooses to perform the action with the optimal expected outcome for itself from among all feasible actions". 

We will define the number choosing game as follows. You name any single finite number x. You then gain x utility and the game then ends. You can only name a finite number, naming infinity is not allowed.

Clearly, the agent that names x+1 is more rational than the agent that names x (and behaves the same in every other situation). However, there does not exist a completely rational agent, because there does not exist a number that is higher than every other number. Instead, the agent who picks 1 is less rational than the agent who picks 2 who is less rational than the agent who picks 3 and so on until infinity. There exists an infinite series of increasingly rational agents, but no agent who is perfectly rational within this scenario.

Furthermore, this hypothetical doesn't take place in our universe, but in a hypothetical universe where we are all celestial beings with the ability to choose any number however large without any additional time or effort no matter how long it would take a human to say that number. Since this statement doesn't appear to have been clear enough (judging from the comments), we are explicitly considering a theoretical scenario and no claims are being made about how this might or might not carry over to the real world. In other words, I am claiming the the existence of perfect rationality does not follow purely from the laws of logic. If you are going to be difficult and argue that this isn't possible and that even hypothetical beings can only communicate a finite amount of information, we can imagine that there is a device that provides you with utility the longer that you speak and that the utility it provides you is exactly equal to the utility you lose by having to go to the effort to speak, so that overall you are indifferent to the required speaking time.

In the comments, MattG suggested that the issue was that this problem assumed unbounded utility. That's not quite the problem. Instead, we can imagine that you can name any number less than 100, but not 100 itself. Further, as above, saying a long number either doesn't cost you utility or you are compensated for it. Regardless of whether you name 99 or 99.9 or 99.9999999, you are still choosing a suboptimal decision. But if you never stop speaking, you don't receive any utility at all.

I'll admit that in our universe there is a perfectly rational option which balances speaking time against the utility you gain given that we only have a finite lifetime and that you want to try to avoid dying in the middle of speaking the number which would result in no utility gained. However, it is still notable that a perfectly rational being cannot exist within a hypothetical universe. How exactly this result applies to our universe isn't exactly clear, but that's the challenge I'll set for the comments. Are there any realistic scenarios where the lack of existence of perfect rationality has important practical applications?

Furthermore, there isn't an objective line between rational and irrational. You or I might consider someone who chose the number 2 to be stupid. Why not at least go for a million or a billion? But, such a person could have easily gained a billion, billion, billion utility. No matter how high a number they choose, they could have always gained much, much more without any difference in effort.

I'll finish by providing some examples of other games. I'll call the first game the Exploding Exponential Coin Game. We can imagine a game where you can choose to flip a coin any number of times. Initially you have 100 utility. Every time it comes up heads, your utility triples, but if it comes up tails, you lose all your utility. Furthermore, let's assume that this agent isn't going to raise the Pascal's Mugging objection. We can see that the agent's expected utility will increase the more times they flip the coin, but if they commit to flipping it unlimited times, they can't possibly gain any utility. Just as before, they have to pick a finite number of times to flip the coin, but again there is no objective justification for stopping at any particular point.

Another example, I'll call the Unlimited Swap game. At the start, one agent has an item worth 1 utility and another has an item worth 2 utility. At each step, the agent with the item worth 1 utility can choose to accept the situation and end the game or can swap items with the other player. If they choose to swap, then the player who now has the 1 utility item has an opportunity to make the same choice. In this game, waiting forever is actually an option. If your opponents all have finite patience, then this is the best option. However, there is a chance that your opponent has infinite patience too. In this case you'll both miss out on the 1 utility as you will wait forever. I suspect that an agent could do well by having a chance of waiting forever, but also a chance of stopping after a high finite number. Increasing this finite number will always make you do better, but again, there is no maximum waiting time.

(This seems like such an obvious result, I imagine that there's extensive discussion of it within the game theory literature somewhere. If anyone has a good paper that would be appreciated).

Link to part 2: Consequences of the Non-Existence of Rationality 

This post won't directly address the Sleeping Beauty problem so you may want to read the above link to understand what the sleeping beauty problem is first.

Half*-Sleeping Beauty Problem

The asterisk is because it is only very similar to half of the sleeping beauty problem, not exactly half.

A coin is flipped. If it is heads, you are woken up with 50% chance and interrogated about the probability of the coin having come up heads. The other 50% of the time you are killed. If it is tails you are woken up and similarly interrogated. Given that you are being interrogated, what is the probability that the coin came up heads? And have you received any new information?

Double-Half*-Sleeping Beauty problem

A coin is flipped. If it is heads, a coin is flipped again. If this second coin is heads you are woken up and interrogated on Monday, if it is tails you are woken up and interrogated on Tuesday. If it is tails, then you are woken up on Monday and Tuesday and interrogated both days (having no memory of your previous interrogation). If you are being interrogated, what is the chance the coin came up heads? And have you received any new information?

Double-Half*-Sleeping Beauty problem with Known Day Variation

Sleeping Couples Problem

A man and his identical-valued wife have lived together for so many years that they have reached Aumann agreement on all of their beliefs, including core premises, so that they always make the same decision in every situation.

A coin is flipped. If it is heads, one of the couple is randomly woken up and interrogated about the probability of the coin having come up heads. The other is killed. If it is tales, both are woken up separately and similarly interrogated. If you are being interrogated, what is the probability that the coin came up heads? And have you received any new information?

Sleeping Clones Problem

A coin is flipped. If it is heads, you are woken up and interrogated about the probability of the coin having come up heads. If it is tails, then you are cloned and both copies are interrogated separately without knowing whether they are the clone or not. If you are being interrogated, what is the probability that the coin came up heads? And have you received any new information?

My expectation is that the Double-Half Sleeping Beauty and Sleeping Clones will be controversial, but I am optimistic that there will be a consensus on the other three.

Solutions (or at least what I believe to be the solutions) will be forthcoming soon.

Caveats: Dependency (Assumes truth of the arguments against perfect theoretical rationality made in the previous post), Controversial Definition (perfect rationality as utility maximisation, see previous thread)

This article is a follow up to: The Number Choosing Game: Against the existence of perfect theoretical rationality. It discusses the consequences of The Number Choosing Game, which is roughly, that you name the decimal representation of any number and you gain that much utility. It takes place in a theoretical world where there are no real world limitations in how large a number you can name or any costs. We can also assume that this game takes place outside of regular time, so there is no opportunity cost. Needless to say, this was all rather controversial.

Update: Originally I was trying to separate the consequences from the arguments, but it seems that this blog post slipped away from it.

What does this actually mean for the real world?

This was one of the most asked questions in the previous thread. I will answer this, but first I want to explain why I was reluctant to answer. I agree that it is often good to tell people what the real world consequences are as this isn't always obvious. Someone may miss out on realising how important an idea is if this isn't explained to them. However, I wanted to fight against the idea that people should always be spoonfed the consequences of every argument. A rational agent should have some capacity to think for themselves - maybe I tell you that the consequences are X, but they are actually Y. I also see a great deal of value from discussing the truth of ideas separate from the practical consequences. Ideally, everyone would be blind to the practical consequences when they were first discussing the truth of an idea as it would lead to a reduction in motivated reasoning.

The consequences of this idea are in one sense quite modest. If perfect rationality doesn't exist in at least some circumstances_(edited), then if you want to assume it, you have to prove that a relevant class of problems has a perfectly rational agent. For example, if there are a finite number of options, each with a measurable, finite utility, we can assume that a perfectly rational agent exists. I'm sure we can prove that such agents exist for a variety of situations involving infinite options as well. However, there will also be some weird theoretical situations where it doesn't apply. This may seem irrelevant to some people, but if you are trying to understand strange theoretical situations, knowing that perfect rationality doesn't exist for some of these situations will allow you to provide an answer when someone hands you something unusual and says, "Solve this!". Now, I know my definition for rationality is controversial, but even if you don't accept it, it is still important to realise that the question, "What would a utility maximiser do?" doesn't always have an answer, as sometimes there is no maximum utility. Assuming perfectly rational agents as defined by utility maximisation is incredibly common in game theory and economics. This is helpful for a lot of situations, but after you've used this for a few years in situations where it works you tend to assume it will work everywhere.

Is missing out on utility bad?

One of the commentators on the original thread can be paraphrased as arguing, "Well, perhaps the agent only wanted a million utility". This misunderstands that the nature of utility. Utility is a measure of things that you want, so it is something you want more of by definition. It may be that there's nothing else you want, so you can't actually receive any more utility, but you always want more utility_(edited).

Here's one way around this problem. The original problem assumed that you were trying to optimise for your own utility, but lets pretend now that you are an altruistic agent and that when you name the number, that much utility is created in the world by alleviating some suffering. We can assume in infinite universe so there's infinite suffering to alleviate, but that isn't strictly necessary, as no matter how large a number you name, it is possible that it might turn out that there is more suffering than that in the universe (while still being finite). So let's suppose you name a million, million, million, million, million, million (by its decimal representation of course). The gamemaker then takes you to a planet where the inhabitants are suffering the most brutal torture imaginable by a dictator that is using their planet's advanced neuroscience knowledge to maximise the suffering. The gamemaker tells you that if you had added an extra million on the end, then these people would have had their suffering alleviated. If rationality is winning, does a planet full of tortured people count as winning? Sure, the rules of the game prevent you from completely winning, but nothing in the game stopped you from saving those people. The agent that also saved those people is a more effective agent and hence a more rational agent than you are. Further if you accept that there is no difference between acts and omissions, then there is no moral difference between torturing those people yourself and failing to say the higher number (Actually, I don't really believe this last point. I think this is more a flaw with arguing acts and omissions are the same in the specific case of an unbounded set of options. I wonder if anyone has ever made this argument before, I wouldn't be surprised if it this wasn't the case and if there was a philosophy paper in this).

But this is an unwinnable scenario, so a perfectly rational agent will just pick a number arbitrarily? Sure you don't get the most utility, but why does this matter?

If we say that the only requirement here for an agent to deserve the title of "perfectly rational" is to pick an arbitrarily stopping point, then there's no reason why we can't declare the agent that arbitrarily stops at 999 as "perfectly rational". If I gave an agent the option of picking a utility of 999 or a utility of one million, the agent who picked a utility of 999 would be quite irrational. But suddenly, when given even more options, the agent who only gets 999 utility counts as rational. It actually goes further than this, there's no objective reason that the agent can't just stop at 1. The alternative is that we declare any agent who picks at least a "reasonably large" number to be rational. The problem is that there is no objective definition of "reasonably large". This would create a situation where our definition of "perfectly rational" would be subjective, which is precisely what the idea of perfectly rational was created to avoid. It gets worse than this still. Let's pretend that before the agent plays the game they lose a million utility (and that the first million utility they get from the game goes towards reversing these effects, time travel is possible in this universe). We then get our "perfectly rational" agent a million (minus one) utility in the red, ie. suffering a horrible fate, which they could have easily chosen to avoid. Is it really inconceivable that the agent who gets positive one million utility instead of negative one million could be more rational?

What if this were a real life situation? Would you really go, "meh" and accept the torture because you think a rational agent can pick an arbitrary number and still be perfectly rational?_(edit)  

The argument that you can't choose infinity, so you can't win anyway, is just a distraction. Suppose perfect rationality didn't exist for a particular scenario, what would this imply about this scenario? The answer is that it would imply that there was no way of conclusively winning, because, if there was, then an agent following this strategy would be perfectly rational for this scenario. Yet, somehow people are trying to twist it around the other way and conclude that it disproves my argument. You can't disprove an argument by proving what it predicts_(edit).

What other consequences are there?

The fact that there is no perfectly rational agent for these situations means that any agent will seem to act rather strangely. Let's suppose that a particular agent who plays this game will always stop at a certain number X, say a Googleplex. If we try to sell them this item for more than that, they would refuse as they wouldn't make money on it, despite the fact that they could make money on it if they chose a higher number.

Where this gets interesting is that the agent might have special code to buy the right to play the game for any price P, and then choose the number X+P. It seems that sometimes it is rational to have path-dependent decisions despite the fact that the amount paid doesn't affect the utility gained from choosing a particular number. 

Further, with this code, you could buy the right to play the game back off the agent (before it picks the number) for X+P+1. You could then sell it back to the agent for X+P+one billion and repeatedly buy and sell the right to play the game back to the agent. (If the agent knows that you are going to offer to buy the game off it, then it could just simulate the sale by increasing the number it asks for, but it has no reason not to simulate the sale and also accept a higher second offer)

Further, if the agent was running code to choose the number 2X instead, we would end up with a situation where it might be rational for the agent to pay you money to charge it extra for the right to play the game.

Another property is that you can sell the right to play the game to any number of agents, add up all their numbers, and add your profit on top and ask for that much utility.

It seems like the choices for these games obey rather unusual rules. If these choices are allowed to count as "perfectly rational" as per the people who disagree with me that perfect rationality exists, it seems at the very least that perfect rationality is something that behaves very differently from what we might expect.

At the end of the day, I suppose whether you agree with my terminology regarding rationality or not, we can see that there are specific situations where we it seems reasonable to act in a rather strange manner.

This will be a short article. I've been seeing a lot of dubious reasoning about consciousness and sleep. One famous problem is the problem of personal identity with a destructive teleporter. In this problem, we imagine that you are cloned perfectly in an alternate location and then your body is destroyed. The question asked is whether this clone is the same person as you.

One really bad argument that I've seen around this is the notion that the fact that we sleep every night means that we experience this teleporter every day.

The reason why this is a very bad argument is that it equivocates with two different meanings of consciousness:

• Consciousness as opposed to being asleep or unconscious, where certain brain functions are inactive
• Consciousness as opposed to being non-sentient, like a rock or bacteria, where you lack the ability to have experiences