Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Eliezer_Yudkowsky 28 September 2014 04:51:20PM 1 point [-]

This sounds to me like you're arguing against letting a different set of properties of the decision system dominate 'far' decisions as opposed to 'near' decisions. But some of the earliest operations we do are loading in the AI's model into the human's decision system, and it seems to me like a pretty good case for modeling this as the counterfactual 'the human anticipates the same things happening that the AI anticipates, conditional on these actions or strategies', not the counterfactual where you tell a Christian fundamentalist the verbal statement 'God doesn't exist' and model them screaming at the AI where it's wrong. In other words, the first answer that occurs to me is along the lines of, "The counterfactuals we are doing mostly eliminate far mode except insofar as it would actually apply to particular, concrete, lived-in scenarios."

Comment author: Stuart_Armstrong 01 October 2014 03:25:12PM 0 points [-]

Out of interest, was this always the plan, or is this a new patch to CEV?

Comment author: Mitchell_Porter 25 September 2014 02:14:25AM -2 points [-]

What a coincidence - I could make use of the Future of Humanity Institute's money, too.

Comment author: Stuart_Armstrong 26 September 2014 01:41:29PM 3 points [-]

By donating it to the top altruistic cause, I assume ;-)

Comment author: pcm 23 September 2014 04:19:59PM 1 point [-]

Large mindspace does not by itself imply problems for CEV.

The obvious way for CEV to converge is for people to compromise and cooperate on some joint utility function rather than try to conquer those they disagree with. Historical trends suggest increasing cooperation. As long as that continues, coherent agreements about volition will become easier if human thought is extrapolated first.

Comment author: Stuart_Armstrong 24 September 2014 10:02:07AM 1 point [-]

Small mindspaces would make CEV easier, so large mindspaces have to be a problem at some level.

Historical trends suggest increasing cooperation.

CEV is an algorithm, not a continuation of historical trends. Getting the algorithm right might make use of stuff like those trends, though.

Comment author: Decius 24 September 2014 05:19:51AM 1 point [-]

Silly me.

"Suppose I had a hypothesis that all Earth-era and earlier planets shared some feature that some later planets don't."

Given that we don't know what the requirements are to be a galactic phenomenon, figuring out which one we don't have is impossible.

Comment author: Stuart_Armstrong 24 September 2014 09:58:13AM 2 points [-]

"Suppose I had a hypothesis that all Earth-era and earlier planets shared some feature that some later planets don't."

Then we can do the reverse approach - crush all the data we do know, and see what changed about the time our Earth came around. We can then take all these candidates, and check whether any seem plausible, then do some further investigation.

Comment author: Stabilizer 23 September 2014 08:30:02PM 21 points [-]

$30 donated. It may become quasi-regular, monthly.

Thanks for letting us know. I wanted to donate to x-risk, but I didn't really want to give to MIRI (even though I like their goals and the people) because I worry that MIRI's approach is too narrow. FHI's broader approach, I feel, is more appropriate given our current ignorance about the vast possible varieties of existential threats.

Comment author: Stuart_Armstrong 24 September 2014 09:47:36AM 5 points [-]


Comment author: Decius 22 September 2014 09:55:26PM 1 point [-]

Suppose I had a hypothesis about something that was present in many Earth-era planets but not astronomically younger planets. How would I falsify such a hypothesis?

I don't think something with lottery-low odds like the moon would adequately explain the paradox.

I also recognize the possibility that Earth might not ever become a Tegmark I civilization, much less a III.

Comment author: Stuart_Armstrong 23 September 2014 10:34:22AM 3 points [-]

Suppose I had a hypothesis about something that was present in many Earth-era planets but not astronomically younger planets. How would I falsify such a hypothesis?

Depends on the hypothesis. We have some pretty good picture about the development of the Milky Way. If your theory was about rates of supernovas or a certain proportion of heavy metals, then we could check whether those happened recently. If it was more specifically about Earth-like planets, we could record the hypothesis, and check it in a few years or decades as our picture of Earth-like planets around other stars gets clearer.

Comment author: diegocaleiro 23 September 2014 04:43:44AM 3 points [-]

you mean: the human mindspace is much narrower in near mode than in far mode.

Comment author: Stuart_Armstrong 23 September 2014 10:31:36AM 2 points [-]

corrected, thanks!

Comment author: Lumifer 22 September 2014 04:49:12PM 12 points [-]

To summarise: the human mindspace is much narrower is near mode than in far mode.

The conclusion I draw from your examples is that the human mindspace is much narrower when constrained by reality then when allowed to engage in flights of fantasy.

I agree that that there is tension between "coherent" and "extrapolated", however it looks to me secondary to more basic problems. The "extrapolated" part essentially assumes that all humans have the same value system at the core, and the "coherent" part implies that human preferences are coherent when, empirically speaking, they are not.

Comment author: Stuart_Armstrong 22 September 2014 05:02:55PM 4 points [-]

when constrained by reality

Yes. But it's mainly constrained by social reality, which gives you convergence for cheap.

Comment author: Stuart_Armstrong 22 September 2014 12:11:15PM 3 points [-]

Therefore, serious design work on a self-replicating spacecraft should have a high priority.

What you're suggesting will reduce uncertainty, but won't change the mean probability. Suppose we assume the remaining filter risk is p. Then you're proposing a course of action which, if successful, would reduce it to q < p. So:

Either we reduce p to q immediately without doing any work (because we "could have worked on a self-replicating spacecraft", which gives us as much probability benefit as actually doing it), or it means that there is a residual great filter risk (p-q) between now and the completion of the project. This great filter risk would likely come from the project itself.

Essentially your model is playing Russian roulette, and you're encouraging us to shoot ourselves rapidly rather than slowly. This would make it clearer faster what the risk actually is, but wouldn't reduce the risk.

CEV: coherence versus extrapolation

14 Stuart_Armstrong 22 September 2014 11:24AM

It's just struck me that there might be a tension between the coherence (C) and the extrapolated (E) part of CEV. One reason that CEV might work is that the mindspace of humanity isn't that large - humans are pretty close to each other, in comparison to the space of possible minds. But this is far more true in every day decisions than in large scale ones.

Take a fundamentalist Christian, a total utilitarian, a strong Marxist, an extreme libertarian, and a couple more stereotypes that fit your fancy. What can their ideology tell us about their everyday activities? Well, very little. Those people could be rude, polite, arrogant, compassionate, etc... and their ideology is a very weak indication of that. Different ideologies and moral systems seem to mandate almost identical everyday and personal interactions (this is in itself very interesting, and causes me to see many systems of moralities as formal justifications of what people/society find "moral" anyway).

But now let's more to a more distant - "far" - level. How will these people vote in elections? Will they donate to charity, and if so, which ones? If they were given power (via wealth or position in some political or other organisation), how are they likely to use that power? Now their ideology is much more informative. Though it's not fully determinative, we would start to question the label if their actions at this level seemed out of synch. A Marxist that donated to a Conservative party, for instance, would give us pause, and we'd want to understand the apparent contradiction.

Let's move up yet another level. How would they design or change the universe if they had complete power? What is their ideal plan for the long term? At this level, we're entirely in far mode, and we would expect that their vastly divergent ideologies would be the most informative piece of information about their moral preferences. Details about their character and personalities, which loomed so large at the everyday level, will now be of far lesser relevance. This is because their large scale ideals are not tempered by reality and by human interactions, but exist in a pristine state in their minds, changing little if at all. And in almost every case, the world they imagine as their paradise will be literal hell for the others (and quite possibly for themselves).

To summarise: the human mindspace is much narrower in near mode than in far mode.

And what about CEV? Well, CEV is what we would be "if we knew more, thought faster, were more the people we wished we were, had grown up farther together". The "were more the people we wished we were" is going to be dominated by the highly divergent far mode thinking. The "had grown up farther together" clause attempts to mesh these divergences, but that simply obscures the difficulty involved. The more we extrapolate, the harder coherence becomes.

It strikes me that there is a strong order-of-operations issue here. I'm not a fan of CEV, but it seems it would be much better to construct, first, the coherent volition of humanity, and only then to extrapolate it.

View more: Next