All of Academian's Comments + Replies

On reflection, I endorse the conclusion and arguments in this post. I also like that it's short and direct. Stylistically, it argues for a behavior change among LessWrong readers who sometimes make surveys, rather than being targeted at general LessWrong readers. In particular, the post doesn't spend much time or space building interest about surveys or taking a circumspect view of them. For this reason, I might suggest a change to the original post to add something to the top like "Target audience: LessWrong readers who often or occasionally make form... (read more)

I've been trying to get MIRI to switch to stop calling this blackmail (extortion for information) and start calling it extortion (because it's the definition of extortion). Can we use this opportunity to just make the switch?

I support this, whole-heartedly :) CFAR has already created a great deal of value without focusing specifically on AI x-risk, and I think it's high time to start trading the breadth of perspective CFAR has gained from being fairly generalist for some more direct impact on saving the world.

"Brier scoring" is not a very natural scoring rule (log scoring is better; Jonah and Eliezer already covered the main reasons, and it's what I used when designing the Credence Game for similar reasons). It also sets off a negative reaction in me when I see someone naming their world-changing strategy after it. It makes me think the people naming their strategy don't have enough mathematician friends to advise them otherwise... which, as evidenced by these comments, is not the case for CFAR ;) Possible re-naming options that contrast well with "signal boosting"

• Score boosting
• Signal filtering
• Signal vetting

This is a cryonics-fails story, not a cryonics-works-and-is-bad story.

Seems not much worse than actual-death, given that in this scenario you could still choose to actually-die if you didn't like your post-cryonics life.

Seems not much worse than actual-death, given that in this scenario you (or the person who replaces you) could still choose to actually-die if you didn't like your post-cryonics life.

Seems not much worse than actual-death, given that in this scenario you could still choose to actually-die if you didn't like your post-cryonics life.

This is an example where cryonics fails, and so not the kind of example I'm looking for in this thread. Sorry if that wasn't clear from the OP! I'm leaving this comment to hopefully prevent more such examples from distracting potential posters.

Hmm, this seems like it's not a cryonics-works-for-you scenario, and I did mean to exclude this type of example, though maybe not super clearly:

OP: There's a separate question of whether the outcome is positive enough to be worth the money, which I'd rather discuss in a different thread.

(2) A rich sadist finds it somehow legally or logistically easier to lay hands on the brains/minds of cryonics patients than of living people, and runs some virtual torture scenarios on me where I'm not allowed to die for thousands of subjective years or more.

(1) A well-meaning but slightly-too-obsessed cryonics scientist wakes up some semblance of me in a semi-conscious virtual delirium for something like 1000 very unpleasant subjective years of tinkering to try recovering me. She eventually quits, and I never wake up again.

See Nate's comment above:

http://lesswrong.com/lw/n39/why_cfar_the_view_from_2015/cz99

And, FWIW, I would also consider anything that spends less than $100k causing a small number of top-caliber researchers to become full-time AI safety researchers to be extremely "effective".

[This is in fact a surprisingly difficult problem to solve. Aside from personal experience seeing the difficulty of causing people to become safety researchers, I have also been told by some rich, successful AI companies earnestly trying to set up safety research divisions (y... (read more)

Just donated $500 and pledged $6500 more in matching funds (10% of my salary).

I would expect not for a paid workshop! Unlike CFAR's core workshops, which are highly polished and get median 9/10 and 10/10 "are you glad you came" ratings, MSFP

• was free and experimental,

• produced two new top-notch AI x-risk researchers for MIRI (in my personal judgement as a mathematician, and excluding myself), and

• produced several others who were willing hires by the end of the program and who I would totally vote to hire if there were more resources available (in the form of both funding and personnel) to hire them.

1) Logical depth seems super cool to me, and is perhaps the best way I've seen for quantifying "interestingness" without mistakenly equating it with "unlikeliness" or "incompressibility".

2) Despite this, Manfred's brain-encoding-halting-times example illustrates a way a D(u/h) / D(u) optimized future could be terrible... do you think this future would not obtain, because despite being human-brain-based, would not in fact make much use of being on a human brain? That is, it would have extremely high D(u) and therefore be pena... (read more)

Great question! It was in the winter of 2013, about a year and a half ago.

Thanks, fixed!

you cannot use the category of "quantum random" to actual coin flip, because an object to be truly so it must be in a superposition of at least two different pure states, a situation that with a coin at room temperature has yet to be achieved (and will continue to be so for a very long time).

Given the level of subtlety in the question, which gets at the relative nature of superposition, this claim doesn't quite make sense. If I am entangled with a a state that you are not entangled with, it may "be superposed" from your perspective ... (read more)

Not justify: instead, explain.

I disagree. Justification is the act of explaining something in a way that makes it seem less dirty.

If you're curious about someone else's emotions or perspective, first, remember that there are two ways to encode knowledge of how someone else feels: by having a description of their feelings, or by empathizing and actually feeling them yourself. It is more costly --- in terms of emotional energy --- to empathize with someone, but if you care enough about them to afford them that cost, I think it's the way to go. You can ask them to help you understand how they feel, or help you to see things the way they do. If you succeed, they'll appreciate having someone who can share their perspective.

My summary of this idea has been that life is a non-convex optimization problem. Hill-climbing will only get you to the top of the hill that you're on; getting to other hills requires periodic re-initializing. Existing non-convex optimization techniques are often heuristic rather than provably optimal, and when they are provable, they're slow.

And the point of CFAR is to help people become better filtering good ideas from bad. It is plainly not to produce people who automatically believe the best verbal argument anyone presents to them without regard for what filters that argument has been through, or what incentives the Skilled Arguer might have to utter the Very Convincing Argument for X instead of the Very Very Convincing Argument for Y. And certainly not to have people ignore their instincts; e.g. CFAR constantly recommends Thinking Fast and Slow by Kahneman, and teaches exercises to extract more information from emotional and physical senses.

What if we also add a requirement that the FAI doesn't make anyone worse off in expected utility compared to no FAI?

I don't think that seems reasonable at all, especially when some agents want to engage in massively negative-sum games with others (like those you describe), or have massively discrete utility functions that prevent them from compromising with others (like those you describe). I'm okay with some agents being worse off with the FAI, if that's the kind of agents they are.

Luckily, I think people, given time to reflect and grown and learn, are not like that, which is probably what made the idea seem reasonable to you.

Non-VNM agents satisfying only axiom 1 have coherent preferences... they just don't mix well with probabilities.

Dumb solution: an FAI could have a sense of justice which downweights the utility function of people who are killing and/or procreating to game their representation in AI's utility function, or something like that do disincentivize it. (It's dumb because I don't know how to operationalize justice; maybe enough people would not cheat and want to punish the cheaters that the FAI would figure that out.)

Also, given what we mostly believe about moral progress, I think defining morality in terms of the CEV of all people who ever lived is probably okay... they'd probably learn to dislike slavery in the AI's simulation of them.

Thanks for writing this up!

I don't see how it could be true even in the sense described in the article without violating Well Foundation somehow

Here's why I think you don't get a violation of the axiom of well-foundation from Joel's answer, starting from way-back-when-things-made-sense. If you want to skim and intuit the context, just read the bold parts.

1) Humans are born and see rocks and other objects. In their minds, a language forms for talking about objects, existence, and truth. When they say "rocks" in their head, sensory neurons associated with the presence ... (read more)

That was imprecise, but I was trying to comment on this part of the dialogue using the language that it had established

Ah, I was asking you because I thought using that language meant you'd made sense of it ;) The language of us "living in a (model of) set theory" is something I've heard before (not just from you and Eliezer), which made me think I was missing something. Us living in a dynamical system makes sense, and a dynamical system can contain a model of set theory, so at least we can "live with" models of set theory... we in... (read more)

Help me out here...

One of the participants in this dialogue ... seems too convinced he knows what model he's in.

I can imagine living a simulation... I just don't understand yet what you mean by living in a model in the sense of logic and model theory, because a model is a static thing. I heard someone once before talk about "what are we in?", as though the physical universe were a model, in the sense of model theory. He wasn't able to operationalize what he meant by it, though. So, what do you mean when you say this? Are you consideri... (read more)

Until I'm destroyed, of course!

... but since Qiaochu asked that we take ultrafinitism seriously, I'll give a serious answer: something else will probably replace ultrafinitism as my preferred (maximum a posteriori) view of math and the world within 20 years or so. That is, I expect to determine that the question of whether ultrafinitism is true is not quite the right question to be asking, and have a better question by then, with a different best guess at the answer... just because similar changes of perspective have happened to me several times already in my life.

I also wish both participants in the dialogue would take ultrafinitism more seriously.

For what it's worth, I'm an ultrafinitist. Since 2005, at least as far as I've been able to tell.

If you want to make this post even better (since apparently it's attracting massive viewage from the web-at-large!), here is some feedback:

I didn't find your description of the owl monkey experiment very compelling,

If a monkey was trained to keep a hand on the wheel that moved just the same, but he did not have to pay attention to it… the cortical map remained the same size.

because it wasn't clear that attention was causing the plasticity; the temporal association of subtle discriminations with rewards could plausibly cause plasticity directly, wit... (read more)

I'm pretty sure that the idea of the previous two paragraphs has been talked about before, but I can't find where.

On LessWrong: VNM expected utility theory: uses, abuses, and interpretation (shameless self-citation ;)

On Wikipedia: Limitations of the VNM utility theorem

+1 for sharing; you seem the sort of person my post is aimed at: so averse to being constrained by self-image that you turn a blind eye when it affects you. It sounds to me like you that you are actively trying to suppress having beliefs about yourself:

People around me have a model of what Dan apparently is which is empathetic, nice, generous etc. I'm always the first to point out a bias such as racism or nonfactual emotional opinions etc. I don't have to see myself as any of those things though.

I've been there, and I can think of a number of possible... (read more)

If you're a "people-hater" who is able to easily self-modify, why do you still "hate" people? Are you sure you're not rationalizing the usefulness of your dislike of others? What do you find yourself saying to yourself and others about what being a "people-hater" achieves for you. Are there other ways for you to achieve those without hating people? What do you find yourself saying to yourself and others about why it's hard to change? What if in a group of 20+ people interested in rationality, someone has a cool trick you ... (read more)

You're describing costly signaling. Contrary to your opening statement,

The word 'signalling' is often used in Less Wrong, and often used wrongly.

people on LessWrong are usually using the term "signalling" consistently with its standard meaning in economics and evolutionary biology. From Wikipedia,

In economics, more precisely in contract theory, signalling is the idea that one party credibly conveys some information about itself to another party

Within evolutionary biology, signalling theory is a body of theoretical work examining communic

tl;dr: I was excited by this post, but so far I find reading the cited literature uncompelling :( Can you point us to a study we can read where the authors reported enough of their data and procedure that we can all tell that their conclusion was justified?

I do trust you, Yvain, and I know you know stats, and I even agree with the conclusion of the post --- that people are imperfect introspectors --- but I'm discouraged to continue searching through the literature myself at the moment because the first two articles you cited just weren't clear enough on w... (read more)

Sometimes I have days of low morale where I don't get much done, and don't try to force myself to do things because I know my morale is low and I'll likely fail. I'm experimenting with a few different strategies for cutting down on low-morale days... I'd like to have ... better motivation (which might allow me to work on things with less willpower/energy expenditure),

Morale, and reducing the need for willpower / conscious effort, are things I've had success with using self-image changes, e.g. inspired by Naruto :) So...

those things seem to me to be

I'm not saying rationalists should avoid engaging in ritual like the plague; but I do a lot of promoting of CFAR and rationality to non-LW-readers, and I happen to know from experience that a post like this in Main sends bad vibes to a lot of people. Again, I think it's sad to have to worry so much about image, but I think it's a reality.

Thanks for sharing this, Quiet; I'm sad to say I agree with you. I think rationality as a movement can't afford to be associated with ritual. It's just too hard to believe that it's not a failure mode. I personally find Raemon's perspective inspiring and convincing. Raemon, it seems to me that you have a very sane perspective on the role of ritual in people's lives. And I'm all about trying to acknowledge and work with our own emotional needs, e.g. in this post. But I personally think openly associating with Ritual with a Capital R is just too sketch... (read more)

I would still tend to say that 1/3 apiece is the fair division

I'm curious why you personally just chose to use the norm-connoting term "fair" in place of the less loaded term "equal division" ... what properties does equal division have that make you want to give it special normative consideration? I could think of some, but I'm particularly interested in what your thoughts are here!

Kind of, though "intrinsic uncertainty" also suggests the possibility that the subsystems might be generating moral intuitions which simply cannot be reconciled and that the conflict might be unresolvable unless one is willing to completely cut away or rewrite parts of their own mind.

Don't you think that things being perfectly balanced in a way such that there is no resolution is sort of a measure zero set of outcomes? In drift-diffusion models of neural groups in human and animal brains arrive at decisions/actions (explained pretty well here... (read more)

Nice post. Do I understand you correctly that what you call "Intrinsic Moral Uncertainty" is the feeling of unresolved conflict between subsystems of our moral-intuition-generators? If so, I'd suggest calling it "Mere internal conflict" or "Not finished computing" or something more descriptive than "Intrinsic".

The Linguistic Consistency Fallacy: claiming, implicitly or otherwise, that a word must be used in the same way in all instances.

I'm definitely talking about the concept of purpose here, not the word.

in my experience they tend to say something like "Not for anyone in particular, just sort of "ultimate" purpose"... That said, the fact that everywhere else we use the word "purpose" it is three-place is certainly a useful observation. It might make us think that perhaps the three-place usage is the original, well-supported v

Done :)

I actually think people who say "Life has no meaning and everything I do is pointless" are actually making a deeper mistake than confusing connotations with denotations... I think they're actually making a denotational error in missing that e.g. "purpose" or "pointfulness" typically denotes ternary relationships of the form "The purpose of X to Y is Z." In other words, one must ask or tacitly understand "purpose to whom?" and "meaning to whom" before the statement makes any sense.

My favorite conn... (read more)

It would help you and other commenters to have an example in mind of something you want to change about yourself, and what methods you've already tried. Do you already do everything that you think you should? Do you ever procrastinate? Do you ever over-weight short-term pains against long-term gains? Is there anything you don't enjoy such that people who enjoy that thing have better lives than you, in your estimation?

If you answer one of these questions positively, and you have not been paying attention to conscious and unconscious aspects of self-ima... (read more)

The relevant notion of intelligence for a singularity is optimization power, and it's not obvious that we aren't already witnessing the expansion of such an intelligence. You may have already had these thoughts, but you didn't mention them, and I think they're important to evaluating the strength of evidence we have against UFAI explosions:

What do agents with extreme optimization power look like? One way for them to look is a rapidly-expanding-space-and-resource-consuming process which at some point during our existence engulfs our region of space destro... (read more)