Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: RobbBB 10 April 2017 06:44:58PM *  3 points [-]

"UDT" is ambiguous and has been used to refer to a number of different approaches; "FDT" is a new name for the central cluster of UDT-ish theories (excluding some similar theories like TDT), intended to be less ambiguous and easier to explain (especially to working decision theorists).

In part it's easier to explain because it's formulated in a more CDT-like fashion (whereas Wei Dai's formulations are more EDT-like), and in part it's easier to explain because it builds in less content: accepting FDT doesn't necessarily require a commitment to some of the philosophical ideas associated with updatelessness and logical prior probability that MIRI, Wei Dai, or other FDT proponents happen to accept. In particular, some of Nate's claims in the linked post are stronger than is strictly required for FDT.

Comment author: siIver 10 April 2017 08:36:00PM 1 point [-]

Thanks! So UDT is integrated. That's good to hear.

Comment author: siIver 09 April 2017 06:02:46PM *  0 points [-]

Can someone briefly explain to me the difference between functional and updateless decision theory / where FDT performs better? That would be much appreciated. I have not yet read FDT because it does not mention UDT (I checked) and I want to understand why UDT needs "fixing" before I invest the time.

Comment author: siIver 05 April 2017 06:55:13AM 1 point [-]

Ugh. When I heard about this first I naively thought it was great news. Now I see it's a much harder question.

Comment author: entirelyuseless 04 March 2017 02:40:52PM 0 points [-]

I don't think you understood.

Consider Newcomb's problem by itself. Omega has already flown away. The million is either there, or it is not.

The only sense that you can change whether the million is there is this: if you decide to take two boxes, you are basically deciding to have been a person who would take two boxes, and therefore deciding that Omega would have not put the million. If you decide to take one box, you are basically deciding to have been a person who would take one box, and therefore deciding that Omega would have put the million there.

In my situation, it is the same: you can "determine" whether your dial is set to the first or second position by making a decision about whether to smoke.

Now consider the Omega situation above, except that after Omega has left, Super-Omega steps in, who cannot be predicted by Omega. Super-Omega changes your decision to the opposite of what it was going to be. If this happens, you can two-box and still get the million, or one-box and get nothing, depending on what your original decision was.

In my situation, it is the same: if someone can actually persuade someone to do the opposite of his dial setting, that persuader is basically like Super-Omega here. In other words, this would be exactly what you were talking about, the situation where convincing someone does not help.

What I was saying was this: in the Alien Implant world, the currently existing people have their dials set to the first or second position in a certain proportion. Let's say that 90% of people have their dials set to the second position (so that most people don't die of cancer), and 10% have their dials set to the first position. I agree that the story says their dials never change place. But new people are constantly being born, and nothing in the story says that the proportion among the new people cannot be different.

Assuming the non-existence of Super-Omegas, it is true that the proportion of people who choose to smoke will never be different from the proportion of people who have dials set to the first position. That does not mean that you cannot convince an individual not to smoke-- it just means that the person you convince already has his dial set to the second position. And it also does not mean that the proportion cannot change, via the existence of new people.

Also, I forgot to remark on your claim that a non-forecasting box is "logically impossible". Is this supposed to be logically impossible with a 51% correlation?

or a 52% correlation? ... or a 98% correlation? or a 99% correlation? or a 99.999999% correlation?

I suppose you will say that it becomes logically impossible at an 87.636783% correlation, but I would like to see your argument for this.

Comment author: siIver 04 March 2017 08:05:13PM *  0 points [-]

In my situation, it is the same: you can "determine" whether your dial is set to the first or second position by making a decision about whether to smoke.

No.

You can not. You can't.

I'm struggling with this reply. I almost decided to stop trying to convince you. I will try one more time, but I need you to consider the possibility that you are wrong before you continue to the next paragraph. Consider the outside view: if you were right, Yudkowksy would be wrong, Anna would be wrong, everyone who read your post here and didn't upvote this revolutionary, shocking insight would be wrong. Are you sufficiently more intelligent than any of them to be confident in your conclusion? I'm saying this only so you to consider the possibility, nothing more.

You do not have an impact. The reason why you believe otherwise is probably that in Newcomb's problem, you do have an impact in an unintuitive way, and you generalized this without fully understanding why you have an impact in Newcomb's problem. It is not because you can magically choose to live in a certain world despite no causal connection.

In Newcomb's problem, the kind of person you are causally determines the contents of the opaque box, and it causally determines your decision to open them. You have the option to change the kind of person you are, i.e. decide you'll one-box in Newcomb's problem at any given moment before you are confronted with it (such as right now), therefore you causally determine how much money you will receive once you play it in the future. The intuitive argument "it is already decided, therefore it doesn't matter what I do" is actually 100% correct. Your choice to one-box or two-box has no influence on the contents of the opaque box. But the fact that you are the kind of person who one-boxes does, and it happens to be that you (supposedly) can't two-box without being the kind of person who two-boxes.

In the Smoking Lesion, in your alien scenario, this impact is not there. An independent source determines both the state of your box and your decision to smoke or not to smoke. A snapshot at all humans at any given time, with no forecasting ability, reveals exactly who will die of cancer and who won't. If superomega comes from the sky and convinces everyone to stop smoking, the exact same people will die as before. If everyone stopped smoking immediately, the exact same people will die as before. In the future, the exact same people who would otherwise have died still die. People with the box on the wrong state who decide to stop smoking still die.

Comment author: entirelyuseless 04 March 2017 03:25:12AM 0 points [-]

I'm not sure what you mean by "autonomy" here. The scientists guess that the device is reading or writing, but a third possibility is that it is doing both, and is a kind of brain computer interface. In essence you might as well say it is part of the thing there: the human-black box combination has just as much autonomy as normal humans have.

"Suppose the decision maker was infallible. Everyone who smokes dies. Sooner or later people would all stop smoking. And this is where the scenario doesn't work anymore. Because the number of people dying can't go down. So either it must be impossible to convince people – in that case, why try? – or the decision maker becomes fallible, in which case your whole argument breaks apart. You don't smoke and still die."

In real life it does seem impossible to convince people; there are plenty of people who are stubborn two-boxers, and plenty of people stubbornly insisting on smoking in the smoking lesion, like yourself. So nothing in my experience rules out it being impossible to convince everyone because of the box. Nonetheless, if the box is writing people's choices, that does not mean it will permanently be impossible to persuade people. It will be impossible to persuade people who already have the opposite written; but if we succeed in the future in persuading everyone, it will mean that everyone in the future had their dial set to the second position. Nothing says that the proportion of people in the population with the dial set one way or another can't change; it may be being beamed in by aliens, and perhaps you are cooperating with them by trying to persuade people.

"Think about this fact again: no forecaster means there is a fixed percentage of the population who has their box on state 1."

So what. The proportion of the population who will in real life die of cancer is a fixed proportion; everyone who is going to die of cancer, is going to die of it, and everyone who isn't, isn't. That doesn't mean the proportion can't change in the future, either in the real life case, or in the box case.

Comment author: siIver 04 March 2017 07:27:51AM *  0 points [-]

it will mean that everyone in the future had their dial set to the second position.

No it won't. Nothing you wrote into the story indicates that you can change the box (in case of no forecaster). If you could, that would change everything (and it wouldn't be the smoking lesion anymore).

Comment author: entirelyuseless 03 March 2017 02:50:42PM 0 points [-]

My example may or may not have a forecaster. The story doesn't say, and that's the point. Even if it turns out that the box is not forecasting anything, but simply making people do things, the winning move is the same.

The Smoking Lesion is used as a counterexample to evidential decision theory. But understood in the way that you just did, it would not be a counterexample. You have the desire to smoke. So you know you have that desire, and you already know that you likely have the lesion. So if you resist the desire, it does not become less probable that you have the lesion.

In order to be a counterexample, your estimate of the probability that you have the lesion has to change depending on whether you decide to smoke or not smoke. This is different from the situation that you just described.

Comment author: siIver 03 March 2017 06:34:04PM *  1 point [-]

I know it was the intention, but it doesn't actually work the way you think.

The thing that causes the confusion is that you introduced an infallible decision maker into the brain that takes all autonomy away from the human (in case of there being no forecaster). This is basically a logical impossibility, which is why I just said "this is newcomb's problem". There has to be a forecaster. But okay, suppose not. I'll show you why this does make a difference.

In Newcomb's problem, you do in fact influence the contents of the opaque box. Your decision doesn't, but the fact that you are the kind of person who makes this decision does. Your algorithm does. In the Alien Implant scenario case no forecaster, you don't affect the state of your box at all.

If there was a forecaster, you could prevent people from dying of cancer by telling them about Timeless Decision Theory. Their choice not to smoke wouldn't affect the state of their box, but the fact that you convince them would: the forecaster predicts that you prevent them from smoking, therefore they do not smoke, therefore it predicts they don't smoke, therefore the box is on state 2.

If there was no forecaster, whether or not you smoke has no effect on your box, causally or otherwise. The state of their box is already determined; if you convinced them not to smoke, they would still get cancer and die and the box would be on state 1. Now this never happens in your scenario, which like I said is pretty close to being impossible, hence the confusion.

But it doesn't matter! Not smoking means you live, smoking means you die!

No, it doesn't. Suppose the decision maker was infallible. Everyone who smokes dies. Sooner or later people would all stop smoking. And this is where the scenario doesn't work anymore. Because the number of people dying can't go down. So either it must be impossible to convince people – in that case, why try? – or the decision maker becomes fallible, in which case your whole argument breaks apart. You don't smoke and still die.

Think about this fact again: no forecaster means there is a fixed percentage of the population who has their box on state 1. If you are still not convinced, consider that "winning" by not smoking would then have to mean that someone else gets cancer instead, since you cannot change the number of people. Obviously, this is not what happens.

If there was a forecaster and everyone stopped smoking, no-one would die. If everyone one-boxes in Newcomb's problem, everyone gets rich.

Comment author: siIver 03 March 2017 06:21:49AM *  2 points [-]

I'm afraid you misunderstand the difference between the Smoking Lesion and Newcomb's problem. In the Smoking Lesion, if you are the kind of person who is affected by the thing which causes lung cancer and the desire to smoke, and you resist this desire, you still die of cancer. Your example is just Newcomb's problem with an infallible forecaster, where if you don't smoke you don't die of cancer. This is an inherent difference. They are not the same.

Comment author: siIver 02 March 2017 06:20:49PM 2 points [-]

I'm pretty happy with this article... though one of my concerns is that the section on how exactly AI could wipe out humanity was a bit short. It wants to cure cancer, it kills all humans, okay, but a reader might just think "well this is easy, tell it not to harm humans." I'd have liked if the article had at least hinted at why the problem is more difficult.

Still, all in all, this could have been much worse.

Comment author: Lumifer 13 February 2017 09:28:31PM 0 points [-]

So if both you and me clearly understand the main point, and if the main point seems reasonably uncontroversial (everyone agrees that it's better to be smart than to be dumb, right?), then why do you describe this post as an epic fail? I'm sure that it makes some people's undergarments undergo interesting topological transformations, but that's hardly unusual or cause for such a.. forceful rejection.

Comment author: siIver 13 February 2017 10:09:02PM *  3 points [-]

I feel like I am repeating myself. Here is the chain of arguments

1) A normal person seeing this article and its upvote count will walk away having a very negative view of LessWrong (reasons in my original reply)

2) Making the valid points of this article is in no way dependent on the negative consequences of 1). You could do the same (in fact, a better job at the same) without offending anyone.

3) LessWrong can be a gateway for people to care about existential risk and AI safety.

4) AI safety is arguably the biggest problem in the world right now and extremely low efforts go into solving it, globally speaking.

5) Due to 4) getting people to care about AI safety is extrmely important. Due to that and 3), harming the reputation of LessWrong is really bad

6) Therefore, this article is awful, harmful, and should be resented by everyone.

Comment author: Lumifer 13 February 2017 07:53:48PM 0 points [-]

Well, there are these words and expression sprinkled throughout your comment:

... promoting elitism and entitlement ... and sexism ... value the thoughts of other people who are more knowledgeable about sexism over yours ... being offensive and harmful ...

All of this seems to go deeper than "mannerisms".

Your basic beef with the post seems to be that it is mean and insensitive and I think such an approach missed the post's main point. It seems that you think the main point is to stigmatize stupid people, label them sub-human, and, possibly, subject them to mandatory treatments with drugs and such. I think the main point is to stress that stupidity is not an unchanging natural condition ("sky is blue, water is wet, some people are stupid") but something that could be changed.

Comment author: siIver 13 February 2017 09:02:00PM 2 points [-]

No, I fully acknowledge that the post tries to do those things, see the second half of my reply. I argue that it fails at doing so but is harmful for our reputation etc.

View more: Next