Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: orthonormal 22 April 2012 04:36:06PM 16 points [-]

If I'd been one of the participants on Hofstadter's original game, I'd have answered him thusly:

"I know where you're going with this experiment— you want all of us to realize that our reasoning is roughly symmetrical, and that it's better if we all cooperate than if we all defect. And if I were playing against a bunch of copies of myself, then I'd cooperate without hesitation.

However, if I were playing against a bunch of traditional game theorists, then the sensible thing is to defect, since I know that they're not going to reason by this line of thought, and so the symmetry is broken. Even if I were playing against a bunch of people who'd cooperate because they think that's more moral, I ought to defect (if I'm acting according to my own self-interest), because they're not thinking in these terms either.

So what I really need to do is to make my best guess about how many of the participants are thinking in this reflexive sort of way, and how many are basing their decisions on completely different lines of thought. And then my choice would in effect be choosing for that block of people and not for the rest, and so I'd need to make my best guess whether it's better for me if I (and the rest of that block) choose to cooperate or if we choose to defect. That depends on how large that block is, how many of the others I expect to cooperate vs. defect, and on the payoff matrix."

At the time he wrote it, the correct choice would have been to defect, because as Hofstadter noted, none of his friends (as brilliant as they were) took anything like that reflexive line of thought. If it were done now, among a group of Less Wrong veterans, I might be convinced to cooperate.

Comment author: Caspar42 01 February 2017 10:27:31AM 0 points [-]

At the time he wrote it, the correct choice would have been to defect, because as Hofstadter noted, none of his friends (as brilliant as they were) took anything like that reflexive line of thought. If it were done now, among a group of Less Wrong veterans, I might be convinced to cooperate.

I would advocate the opposite: Imagine you have never thought about Newcomb-like scenarios before. Therefore, you also don't know how others would act in such problems. Now, you come up with this interesting line of thought about determining the others' choices or correlating with them. Because you are the only data point, your decision should give you a lot of evidence about what others might do, i.e. about whether they will come up with the idea at all and behave in abidance with it.

Now, contrast this with playing the game today. You may have already read studies showing that most philosophers use CDT, that most people one-box in Newcomb's problem, that LWers tend to cooperate. If anything, your decision now gives you less information about what the others will do.

In response to Lost Purposes
Comment author: Caspar42 03 September 2016 10:26:58AM 0 points [-]
Comment author: amcknight 08 November 2012 10:06:04PM 4 points [-]

A Survey of Mathematical Ethics which covers work in multiple disciplines. I'd love to know what parts of ethics have been formalized enough to be written mathematically and, for example, any impossibility results that have been shown.

Comment author: Caspar42 06 May 2016 10:50:46PM *  1 point [-]

Regarding impossibility results, there is now also Brian Tomasik's Three Types of Negative Utilitarianism.

There are also these two attempted formalizations of notions of welfare:

Publication on formalizing preference utilitarianism in physical world models

5 Caspar42 22 September 2015 04:46PM

About a year ago I asked for help with a paper on a formalization of preference utilitarianism in cellular automata. The paper has now been published in the Springer journal Synthese and is available here. I wonder what you think about it and if you are interested would like to discuss it with you.

Comment author: Caspar42 09 July 2015 08:06:34AM 1 point [-]

Certainly a good compilation! It might be even more useful, though, if it contained references to research papers, Bostrom's superintelligence etc., where the arguments are discussed in full detail.

Comment author: Bugmaster 09 July 2015 04:03:59AM 0 points [-]

My own set of objections to AI risk does not include any of these (except possibly #7); but it's possible that they are unusual and therefore do not qualify as "top 10". Still, FWIW, I remain unconvinced that AI risk is something we should be spending any amount of resources on.

Comment author: Caspar42 09 July 2015 08:01:38AM 1 point [-]

Is there a write-up of your objections anywhere?

Comment author: Manfred 29 June 2015 11:54:13PM *  0 points [-]

because there are no simulations of the agent involved.

The role that would normally be played by simulation is here played by a big evidential study of what people with different genes do. This is why it matters whether the people in the study are good decision-makers or not - only when the people in the study are in a position similar to my own do they fulfill this simulation-like role.

It does not seem to be more evil than Newcomb's problem, but I am not sure, what you mean by "evil". For every decision theory, it is possible, of course, to set up some decision problem, where this decision theory loses. Would you say that I set up the "genetic Newcomb problem" specifically to punish CDT/TDT?

Yeah, that sentence is phrased poorly, sorry. But I'll try to explain. The easy way to construct an evil decision problem (say, targeting TDT) is to figure out what action TDT agents take, and then set the hidden variables so that that action is suboptimal - in this way the problem can be tilted against TDT agents even if the hidden variables don't explicitly care that their settings came from this evil process.

In this problem, the "gene" is like a flag on a certain decision theory that tells what action it will take, and the hidden variables are set such that people with that decision theory (the decision theory that people with the one-box gene use) act suboptimally (people with the one-box gene who two-box get more money). So this uses very similar machinery to an evil decision problem. The saving grace is that the other action also gets its own flag (the two-box gene), which has a different setting of the hidden variables.

Comment author: Caspar42 30 June 2015 08:17:09PM 0 points [-]

The role that would normally be played by simulation is here played by a big evidential study of what people with different genes do. This is why it matters whether the people in the study are good decision-makers or not - only when the people in the study are in a position similar to my own do they fulfill this simulation-like role.

Yes, the idea is that they are sufficiently similar to you so that the study can be applied (but also sufficiently different to make it counter-intuitive to say it's a simulation). The subjects of the study may be told that there already exists a study, so that their situation is equivalent to yours. It's meant to be similar to the medical Newcomb problems in that regard.

I briefly considered the idea that TDT would see the study as a simulation, but discarded the possibility, because in that case the studies in classic medical Newcomb problems could also be seen as simulations of the agent to some degree. The "abstract computation that an agent implements" is a bit vague, anyway, I assume, but if one were willing to go this far, is it possible that TDT conflates with EDT?

Comment author: Brian_Tomasik 29 June 2015 10:15:10PM *  1 point [-]

I assume that the one-boxing gene makes a person generically more likely to favor the one-boxing solution to Newcomb. But what about when people learn about the setup of this particular problem? Does the correlation between having the one-boxing gene and inclining toward one-boxing still hold? Are people who one-box only because of EDT (even though they would have two-boxed before considering decision theory) still more likely to have the one-boxing gene? If so, then I'd be more inclined to force myself to one-box. If not, then I'd say that the apparent correlation between choosing one-boxing and winning breaks down when the one-boxing is forced. (Note: I haven't thought a lot about this and am still fairly confused on this topic.)

I'm reminded of the problem of reference-class forecasting and trying to determine which reference class (all one-boxers? or only grudging one-boxers who decided to one-box because of EDT?) to apply for making probability judgments. In the limit where the reference class consists of molecule-for-molecule copies of yourself, you should obviously do what made the most of them win.

Comment author: Caspar42 30 June 2015 04:08:48PM 1 point [-]

But what about when people learn about the setup of this particular problem? Does the correlation between having the one-boxing gene and inclining toward one-boxing still hold?

Yes, it should also hold in this case. Knowing about the study could be part of the problem and the subjects of the initial study could be lied to about a study. The idea of the "genetic Newcomb problem" is that the two-boxing gene is less intuitive than CGTA and that its workings are mysterious. It could make you be sure that you have or don't have the gene. It could make be comfortable with decision theories whose names start with 'C', interpret genetical Newcomb problem studies in a certain way etc. The only thing that we know is that is causes us to two-box, in the end. For CGTA, on the other hand, we have a very strong intuition that it causes a "tickle" or so that could be easily overridden by us knowing about the first study (which correlates chewing gum with throat abscesses). It could not possibly influence what we think about CDT vs. EDT etc.! But this intuition is not part of the original description of the problem.

Comment author: Wei_Dai 30 June 2015 11:04:01AM 2 points [-]

Why would Omega look at other human genes and not the two-boxing (correlated) gene(s) in any world?

I was trying to create a version of the problem that corresponds more closely to MNP, where the fact that a single gene correlates with both chewing gum and abscess is a coincidence, not the result of some process looking for genes correlated with chewing gum, and giving people with those genes abscesses.

Maybe I overlook something or did not describe the problem very well, but in the second multiverse UDT agents two-box, therefore UDT agents (probably) have the two-boxing gene and don't get the $1M. In the first multiverse, UDT agents one-box, therefore UDT agents (probably) don't have the one-boxing gene and get the $1M. So, the first multiverse seems to be better than the second.

Do you see that assuming Omega worked the way I described, then the number and distribution of boxes containing $1M is exactly the same in the two multiverses, therefore the second multiverse is better?

So, it's part of the GNP that Omega has looked at the "two-boxing gene" or (more realistically perhaps) the "most common gene correlated with two-boxing".

I think this is what makes your version of GNP different from MNP, and why we have different intuitions about the two cases. If there is someone or something who looked the most common gene correlated with two-boxing (because it was the most common gene correlated with two-boxing, rather than due to a coincidence), then by changing whether you two-box, you can change whether other UDT agents two-box, and hence which gene is the most common gene correlated with two-boxing, and hence which gene Omega looked at, and hence who gets $1M in box B. In MNP, there is no corresponding process searching for genes correlated with gum chewing, so you can't try to influence that process by choosing to not chew gum.

Comment author: Caspar42 30 June 2015 03:45:11PM 1 point [-]

Do you see that assuming Omega worked the way I described, then the number and distribution of boxes containing $1M is exactly the same in the two multiverses, therefore the second multiverse is better?

Yes, I think I understand that now. But in your version the two-boxing gene practically does not cause the $1M to be in box B, because Omega mostly looks at random other genes. Would that even be a Newcomblike problem?

I think this is what makes your version of GNP different from MNP, and why we have different intuitions about the two cases. If there is someone or something who looked the most common gene correlated with two-boxing (because it was the most common gene correlated with two-boxing, rather than due to a coincidence), then by changing whether you two-box, you can change whether other UDT agents two-box, and hence which gene is the most common gene correlated with two-boxing, and hence which gene Omega looked at, and hence who gets $1M in box B.

In EY's chewing gum MNP, it seems like CGTA causes both the throat abscess and influences people to chew gum. (See p.67 of the TDT paper ) (It gets much more complicated, if evolution has only produced a correlation between CGTA and another chewing gum gene.) The CGTA gene is always read, copied into RNA etc., ultimately leading to throat abscesses. (The rest of the DNA is used, too, but only determines the size of your nose etc.) In the GNP, the two-boxing gene is always read by Omega and translated into a number of dollars under box B. (Omega can look at the rest of the DNA, too, but does not care.) I don't get the difference, yet, unfortunately.

In MNP, there is no corresponding process searching for genes correlated with gum chewing, so you can't try to influence that process by choosing to not chew gum.

I don't understand UDT, yet, but it seems to me that in the chewing gum MNP, you could not chew gum, thereby changing whether other UDT agents chew gum, and hence whether UDT agents' genes contain CGTA. Unless you know that CGTA has no impact on how you ultimately resolve this problem, which is not stated in the problem description and which would make EDT also chew gum.

Comment author: Wei_Dai 30 June 2015 07:04:46AM 6 points [-]

I think UDT reasoning would go like this (if translated to human terms). There are two types of mathematical multiverse, only one of which is real (i.e., logically consistent). You as a UDT agent gets to choose which one. In the first one, UDT agents one-box in this Genetic Newcomb Problem (GNP), so the only genes that statistically correlate with two-boxing are those that create certain kinds of compulsions overriding deliberate decision making, or for other decision procedures that are not logically correlated with UDT. In the second type of mathematical multiverse, UDT agents two-box in GNP, so the list of genes that correlate with two-boxing also includes genes for UDT.

Which type of multiverse is better? It depends on how Omega chooses which gene to look at, which is not specified in the OP. To match the Medical Newcomb Problem as closely as possible, let's assume that in each world (e.g., Everett branch) of each multiverse, Omega picks a random gene look at (from a list of all human genes), and puts $1M in box B for you if you don't have that gene. You live in a world where Omega happened to pick a gene that correlates with two-boxing. Under this assumption, the second type of multiverse is better because the number and distribution of boxes containing $1M is exactly the same in both multiverses, but in the second type of multiverse UDT agents get the additional $1K.

I presume that most LWers would one-box.

I think the reason we have an intuition that we should one-box in the GNP is that when we first read the story, we implicitly assume something else about what Omega is doing. For example, suppose instead of the above, in each world Omega looks at the most common gene correlated with two-boxing and puts $1M in box B if you don't have that gene. If the gene for UDT is the most common such gene in the second multiverse (where UDT two-boxes), then the first multiverse is better because it has more boxes containing $1M, and UDT agents specifically all get $1M instead of $1K.

Comment author: Caspar42 30 June 2015 08:48:54AM *  2 points [-]

Thank you for this elaborate response!!

Omega picks a random gene look at (from a list of all human genes), and puts $1M in box B for you if you don't have that gene

Why would Omega look at other human genes and not the two-boxing (correlated) gene(s) in any world?

Under this assumption, the second type of multiverse is better because the number and distribution of boxes containing $1M is exactly the same in both multiverses, but in the second type of multiverse UDT agents get the additional $1K.

Maybe I overlook something or did not describe the problem very well, but in the second multiverse UDT agents two-box, therefore UDT agents (probably) have the two-boxing gene and don't get the $1M. In the first multiverse, UDT agents one-box, therefore UDT agents (probably) don't have the one-boxing gene and get the $1M. So, the first multiverse seems to be better than the second.

I think the reason we have an intuition that we should one-box in the GNP is that when we first read the story, we implicitly assume something else about what Omega is doing. For example, suppose instead of the above, in each world Omega looks at the most common gene correlated with two-boxing and puts $1M in box B if you don't have that gene.

Yes, this is more or less the scenario, I was trying to describe. Specifically, I wrote:

Omega has only looked at your DNA: If you don't have the "two-boxing gene", Omega puts $1M into box B, otherwise box B is empty.

So, it's part of the GNP that Omega has looked at the "two-boxing gene" or (more realistically perhaps) the "most common gene correlated with two-boxing".

View more: Next