The Fundamental Question

What am I doing?: Working at a regular job as a C++ programmer, and donating as much as possible to SIAI. And sometimes doing other useful things in my spare time.

Why am I doing it?: Because I want to make lots of money to pay for Friendly AI and existential risk research, and programming is what I'm good at.

Why do I want this?: Well, to be honest, the original reason, from several years ago, was "Because Eliezer told me to". Since then I've internalized most of Eliezer's reasons for recommending this, but this process still seems kinda backwards.

I guess the next question is "Why did I originally choose to follow Eliezer?": I started following him back when he still believed in the most basic form of utilitarianism: Maximize pleasure and minimize pain, don't bother keeping track of which entity is experiencing the pleasure or pain. Even back then, Eliezer wasn't certain that this was the value system he really wanted, but for me it seemed to perfectly fit my own values. And even after years of thinking about these topics, I still haven't found any other system that more closely matches what I actually believe. Not even Eliezer's current value system. A... (read more)

[-]Jack15y210

Nothing at all against SIAI but

A couple of times I asked SIAI about the idea of splitting my donations with some other group, and of course they said that donating all of the money to them would still be the most leveraged way for me to reduce existential risks.

If you're in doubt and seeking expert advice you should pick an expert that lacks really obvious institutional incentives to give one answer over others.

Regarding the rest of the comment I found it kind of weird and something freaked me out about it, though I'm not sure quite what. That doesn't mean you're doing anything wrong, I might just have biases or assumptions that make what you're doing seem weird to me. I think it has something to do with your lack of skepticism or cynicism and the focus on looking for someone to follow that MatthewB mentioned. I guess your comment pattern matches with things a very religious person would say: I'm just not sure if that means you're doing something wrong or if I'm having an adverse reaction to a reasonable set of behaviors because I have irrationally averse reactions to things that look religious.

[-]PeerInfinity15y100

Yeah, I realized that it was silly for me to ask SIAI what they thought about the idea of giving SIAI less money, but I didn't know who else to ask, and I still didn't have enough confidence in my own sanity to try to make this decision on my own. And I was kinda hoping that the people at SIAI were rational enough to give an accurate and reasonably unbiased answer, despite the institutional incentives. SIAI has a very real and very important mission, and I would have hoped that its members would be able to rationally think about what is best for the mission, rather than what is best for the group. And the possibility remains that they did, in fact, give a rational and mostly unbiased answer.

The answer they gave was that donating exclusively to SIAI was the most leveraged way to reduce existential risks. Yes, there are other groups that are doing important work, but SIAI is more critically underfunded than they are, and the projects that we (yes, I said "we", even though I'm "just" a donor) are working on this year are critical for figuring out what the most optimal strategies would be for humanity/transhumanity to maximize its probability of surviving into a... (read more)

8PhilGoetz15y

If you were Bill Gates, that might be a valid concern. (The "exclusively" part, not the "SIAI" part.) Otherwise, it's most efficient to donate to just one cause. Especially if you itemize deductions.

8MatthewB15y

It may just be me, but why do you need to find someone to follow? I have always found that forging my own path through the wilderness to be far more enjoyable and yield far greater rewards that following a path, no matter how small or large that path may be.

[-]PeerInfinity15y130

Well, one reason why I feel that I need someone to follow is... severe underconfidence in my ability to make decisions on my own. I'm still working on that. Choosing a person to follow, and then following them, feels a whole lot easier than forging my own path.

I should mention again that I'm not actually "following" Eliezer in the traditional sense. I used his value system to bootstrap my own value system, greatly simplifying the process of recovering from christianity. But now that I've mostly finished with that (or maybe I'm still far from finished?), I am, in fact, starting to think independently. It's taking a long time for me to do this, but I am constantly looking for things that I'm doing or believing just because someone else told me to, and then reconsidering whether these things are a good idea, according to my current values and beliefs. And yes, there are some things I disagree with Eliezer about (the "true ending" to TWC, for example), and things that I disagree with SIAI about ("we're the only place worth donating to", for example). I'll probably start writing more about this, now that I'm starting to get over my irrational fear of... (read more)

6Nick_Tarleton15y

If by 'dark arts' you mean 'non-rational methods of persuasion', such things may be ethically questionable (in general; not volunteering information you aren't obligated to provide almost certainly isn't) but are not (categorically) wrong. Rational agents win.

[-]khafra15y170

I like the way steven0461 put it:

...promoting less than maximally accurate beliefs is an act of sabotage. Don’t do it to anyone unless you’d also slash their tires, because they’re Nazis or whatever. Specifically, don’t do it to yourself.

1PeerInfinity15y

I think I agree with both khafra and Nick. I like this quote, and I've used it before in conversations with other people.

2RobinZ15y

I think it's worth distinguishing between "underconfidence" and "lack of confidence" - the former implies the latter (although not absolutely), but under some circumstances you are justified in questioning your competence. Either way, it sounds like you're working on both ends of that balance, which is good. I think this is good thinking.

2PeerInfinity15y

good point about underconfidence versus lack of confidence, thanks

1MatthewB15y

That puts it into an understandable context... I can't quite understand about the having to shake off Christian Beliefs. I was raised with a tremendously religious mother, but about the age of 6 I began to question her beliefs and by 14 was sure that she was stark raving mad to believe what she did. So, I managed to keep from being brainwashed to begin with. I've seen the results of people who have been brainwashed and who have not managed to break completely free from their old beliefs. Most of them swung back and forth between the extremes of bad belief systems (From born-again Christian to Satanist, and back, many times)... So, what you are doing is probably best for the time being, until you learn the tools needed to step off into the wilderness by yourself.

[-]PeerInfinity15y150

In my case, I knew pretty much from the beginning that something was seriously wrong. But since every single person I had ever met was a christian (with a couple of exceptions I didn't realize until later), I assumed that the problem was with me. The most obvious problem, at least for me, was that none of the so-called christians was able to clearly explain what a christian is, and what it is that I need to do in order to not go to hell. And the people who came closest to being able to give a clear explanation, they were all different from each other, and the answer changed if I asked different questions. So I guess I was... partly brainwashed. I knew that there was something really important I was supposed to do, and that people's souls were at stake (a matter of infinite utility/anti-utility!) but noone was able to clearly explain what it was that I was supposed to do. But they expected me to do it anyway, and made it sound like there was something wrong with me for not instinctively knowing what it was that I was supposed to do. There's lots more I could complain about, but I guess I had better stop now.

So it was pretty obvious that I wasn't going to be able to save anyone's sou... (read more)

[-]cousin_it15y270

But it took me a few weeks of swinging back and forth before I finally settled on Singularitarianism.

Here's a quote from an old revision of Wikipedia's entry on The True Believer that may be relevant here:

A core principle in the book is Hoffer's insight that mass movements are interchangeable; he notes fanatical Nazis later becoming fanatical Communists, fanatical Communists later becoming fanatical anti-Communists, and Saul, persecutor of Christians, becoming Paul, a fanatical Christian. For the true believer the substance of the mass movement isn't so important as that he or she is part of that movement.

And from the current revision of the same article:

Hoffer quotes extensively from leaders of the Nazi and communist parties in the early part of the 20th Century, to demonstrate, among other things, that they were competing for adherents from the same pool of people predisposed to support mass movements. Despite the two parties' fierce antagonism, they were more likely to gain recruits from their opposing party than from moderates with no affiliation to either.

Can't recommend this book enough, by the way.

[-]PeerInfinity15y160

Thanks for the link, and the summary. Somehow I don't find that at all surprising... but I still haven't found any other cause that I consider worth converting to.

At the time I converted, Singularitarianism was nowhere near a mass movement. It consisted almost entirely of the few of us in the SL4 mailing list. But maybe the size of the movement doesn't actually matter.

And it's not "being part of a movement" that I value, it's actually accomplishing something important. There is a difference between a general pool of people who want to be fanatical about a cause, just for the emotional high, and the people who are seriously dedicated to the cause itself, even if the emotions they get from their involvement are mostly negative. This second group is capable of seriously examining their own beliefs, and if they realize that they were wrong, they will change their beliefs. Though as you just explained, the first group is also capable of changing their minds, but only if they have another group to switch to, and they do this mostly for social reasons.

Seriously though, the emotions I had towards christianity were mostly negative. I just didn't fit in with the other christians. O... (read more)

8PeerInfinity15y

um... after reviewing this comment, I realize that the stuff I wrote here doesn't actually count as evidence that I don't have True Believer Syndrome. Or at least not conclusive evidence. oh, and did I mention yet that I also seem to have some form of Saviour Complex? Of course I don't actually believe that I'm saving the world through my own actions, but I seem to be assigning at least some probability that my actions may end up making the difference between whether our efforts to achieve a positive Singularity succeed or fail. but... if I didn't believe this, then I wouldn't bother donating, would I? Do other people manage to believe that their actions might result in making the difference between whether the world is saved or not, without it becoming a Saviour Complex?

7cousin_it15y

PeerInfinity, I don't know you personally and can't tell whether you have True Believer Syndrome. I'm very sorry for provoking so many painful thoughts... Still. Hoffer claims that the syndrome stems from lack of self-esteem. Judging from what you wrote, I'd advise you to value yourself more for yourself, not only for the faraway goals that you may someday help fulfill.

3PeerInfinity15y

no need to apologise, and thanks for pointing out this potential problem. (random trivia: I misread your comment three times, thinking it said "I know you personally can't tell whether you have True Believe Syndrome") as for the painful thoughts... It was a relief to finally get them written down, and posted, and sanity-checked. I made a couple attempts before to write this stuff down, but it sounded way too angry, and I didn't dare post it. And it turns out that the problem was mostly my fault after all. oh, and yeah, I am already well aware that I have dangerously low self-esteem. but if I try to ignore these faraway goals, then I have trouble seeing myself as anything more valuable than "just another person". Actually I often have trouble even recognizing that I qualify as a person... also, an obvious question: are we sure that True Believer Syndrome is a bad thing? or that a Saviour Complex is a bad thing? random trivia: now that I've been using the City of Lights technique for so long, I have trouble remembering not to use a plural first-person pronoun when I'm talking about introspective stuff... I caught myself doing that again as I checked over this comment.

4cousin_it15y

I'm pretty sure of that. Not because of what it does to your goals, but because of what it does to you.

3PeerInfinity15y

Please forgive my ignorance, or possibly my deliberate forgetfulness, but... can you please remind me what you think it does to me?

2cousin_it15y

Several comments above you wrote that both Christianity and Singularitarianism drained you of the resources you could've spent on having fun. As far as I can understand, neither ideology gave you anything back.

7PeerInfinity15y

At first I misread what you said and was about to reply with this paragraph: oh. that's mostly because I was Doing It Wrong. I was pushing myself harder than I could actually sustain in the long term, and that ended up being counterproductive to singularitarianism. ( and also counterproductive to fun, though I still don't consider fun to be of any significant inherent value, compared to the value of the mission) But then I noticed that when I read your comment, I was automatically adding the words "and this would be bad for the mission", which probably isn't what you meant. and I might as well admit that as I was thinking about what else to say in reply, everything I thought of was phrased in terms of what mattered to singularitarianism. I was going to resist the suggestion that I should be paying any attention to what the ideology could give back. I was going to resist the suggestion that fun had any use other than helping me stay focused on the mission, if used in moderation. And I'm still undecided about whether this reaction is a bad thing, because I'm still measuring good and bad according to singularitarian values, not according to selfish values. And I would still resist any attempt to change my values to anything that might conflict with singularitarianism, even in a small way. ugh... even if everyone from SIAI told me to stop taking this so seriously, I would probably still resist. And I might even consider this as a reason to doubt how seriously they are taking the mission. ok, so I guess it would be silly of me to claim that I don't have a true believer's complex, or a saviour complex, or just fanaticism in general. though I still need to taboo the word "fanaticism"... I'm still undecided about whether I'm using it as if it means "so sincerely dedicated that the dedication is counterproductive", or "so sincerely dedicated that anyone who hasn't tried to hack their own mind into being completely selfless would say that I'm taking this way too far".

4Jack15y

Say it was the case that promoting a singularity was a bad idea and that, in particular, SIAI did more harm than good. If someone had compelling evidence of this and presented it to you would you be capable of altering your beliefs and behavior in accordance with this new data? I take it the True Believer would not and that we can all agree with would be a bad thing.

5PeerInfinity15y

ah, but Singularitarianism is different: a True Singularitarian is supposed to be able to update on this evidence, even if it means abandoning SIAI entirely. Presented with evidence of the counterproductivity of SIAI, a True Singularitarian would then try to find a better way to help the efforts to achieving a positive Singularity, even if it meant creating an entirely new group for this purpose. Note that "Singularitarian" is not the same as "SIAI Supporter", or "Eliezer Follower"

2PeerInfinity15y

actually, I think the same applies to a True Christian. If a True Christian finds out that the church isn't doing its job properly, and the church refuses to correct what's wrong, then the True Christian is supposed to start their own church. And this actually happened many times through history...

4cupholder15y

Maybe instead of imagining your actions as having some probability of 'making the difference,' try thinking of them as slightly boosting the probability of a positive singularity? At any rate, the survival of someone wheeled in through the doors of a hospital might depend on the EMTs, the nurses, the surgeons, the lab techs, the pharmacists, the janitors and so on and so on. I'd say they're all entitled to take a little credit without being accused of having a savior complex!

2PeerInfinity15y

um... can you please explain what the difference is, between "having some probability X of making the difference between success and failure, of achieving a positive Singularity" and "boosting the probability of a positive Singularity, by some amount Y"? To me, these two statements seem logically equivalent. Though I guess they focus on different details... oh, I just noticed one obvious difference: X is not equal to Y

1cupholder15y

Yeah, what I wrote was intended as an alternative way of thinking about the situation that might make you feel better, rather than an accusation of wrongness.

4PeerInfinity15y

I guess I'll still need to think about this some more... some random observations: if X > 0, then Y > 0 if Y > 0, then X > 0 I was about to question whether maybe X = Y after all, but further thought reveals that X isn't clearly defined, and I really would be better off focusing on Y, because Y is more clearly defined than X, and thinking about Y seems to trigger less panic than thinking about X. So, yeah, thanks again for your comment. It was helpful. :)

1cupholder15y

No problem!

2AdeleneDawner15y

Nitpick for clarity's sake: I've seen no evidence that this was deliberate in the sense implied, and I would expect to have seen such evidence if it did exist. It may have been deliberate or quasi-deliberate for some other reason, such as social anxiety (which I have seen evidence of).

2PeerInfinity15y

er, yes, that's what I meant. sorry for the confusion. I wasn't deliberately trying to prevent anyone from helping, I was deliberately trying to avoid wasting their time, by having no contact with them, which prevented them from being able to help.

6NancyLebovitz15y

I've heard from an ex-fundamentalist that for some people, conversion is a high in itself (I don't know if this is mostly true for Christians, or applies to movements in general. In any case, he said the high lasts for about two years, and then wears off, so that those people then convert to something else.

2juliawise13y

Huh. I knew this was true of me, but didn't realize it was common. I went from being an extreme Christian at 11 to an extreme utilitarian by about 14 (despite not knowing people who were extreme about either thing).

6Utilitarian15y

PeerInfinity, I'm rather struck by a number of similarities between us: * I, too, am a programmer making money and trying to live frugally in order to donate to high-expected-value projects, currently SIAI. * I share your skepticism about the cause and am not uncomfortable with your 1% probability of positive Singularity. I agree SIAI is a good option from an expected-value perspective even if the mainline-probability scenario is that these concerns won't materialize. * As you might guess from my user name, I'm also a Utilitronium-supporting hedonistic utilitarian who is somewhat alarmed by Eliezer's change of values but who feels that SIAI's values are sufficiently similar to mine that it would be unwise to attempt an alternative friendly-AI organization. * I share the seriousness with which you regard Pascal's wager, although in my case, I was pushed toward religion from atheism rather than the other way around, and I resisted Christian thinking the whole time I tried to subscribe to it. I think we largely agree in our current opinions on the subject. I do sometimes have dreams about going to the Christian hell, though. I'm not sure if you share my focus on animal suffering (since animals outnumber current humans by orders of magnitude) or my concerns about the implications of CEV for wild-animal suffering. Because of these concerns, I think a serious alternative to SIAI in cost-effectiveness is to donate toward promoting good memes like concern about wild animals (possibly including insects) so that, should positive Singularity occur, our descendants will do the right sorts of things according to our values.

3PeerInfinity15y

Hi Utilitarian! um... are you the same guy who wrote those essays at utilitarian-essays.com? If you are, we have already talked about these topics before. I'm the same Peer Infinity who wrote that "interesting contribution" on Singularitarianism in that essay about Pascal's Wager, the one that tried to compare the different religions to examine which of them would be the best to Wager on. And, um... I used to have some really nasty nightmares about going to the christian hell. But then, surprisingly, these nightmares somehow got replaced with nightmares of a hell caused by an Evil AI. And then these nightmares somehow got replaced with nightmares about the other hells that modal realism says must already exist in other universes. I totally agree with you that the suffering of humans is massively outweighed by the suffering of other animals, and possibly insects, by a few orders of magnitude, I forget how many exactly, but I think it was less than 10 orders of magnitude. But I also believe that the amount of positive utility that could be achieved through a positive Singularity is... I think it was about 35 orders of magnitude more than all of the positive or negative utility that has been experienced so far in the entire history of Earth. But I don't remember the details of the math. For a few years now I was planning to write about that, but somehow never got around to it. Well, actually, I did make one feeble attempt to do the math, but that post didn't actually make any attempt to estimate how many orders of magnitude were involved Oh, and I totally share your concerns about the possible implications of CEV. Specifically, that it might end up generating so much negative utility that it outweighs the positive utility, which would mean that a universe completely empty of life would be preferable. Oh, and I know one other person who shares your belief that promoting good memes like concern about wild animals would be more cost effective than donating to Friendl

6Utilitarian15y

Bostrom's estimate in "Astronomical Waste" is "10^38 human lives [...] lost every century that colonization of our local supercluster is delayed," given various assumptions. Of course, there's reason to be skeptical of such numbers at face value, in view of anthropic considerations, simulation-argument scenarios, etc., but I agree that this consideration probably still matters a lot in the final calculation. Still, I'm concerned not just with wild-animal suffering on earth but throughout the cosmos. In particular, I fear that post-humans might actually increase the spread of wild-animal suffering through directed panspermia or lab-universe creation or various other means. The point of spreading the meme that wild-animal suffering matters and that "pristine wilderness" is not sacred would largely be to ensure that our post-human descendants place high ethical weight on the suffering that they might create by doing such things. (By comparison, environmental preservationists and physicists today never give a second thought to how many painful experiences are or would be caused by their actions.) As far as CEV, the set of minds whose volitions are extrapolated clearly does make a difference. The space of ethical positions includes those who care deeply about sorting pebbles into correct heaps, as well as minds whose overriding ethical goal is to create as much suffering as possible. It's not enough to "be smarter" and "more the people we wished we were"; the fundamental beliefs that you start with also matter. Some claim that all human volitions will converge (unlike, say, the volitions of humans and the volitions of suffering-maximizers); I'm curious to see an argument for this.

3Nick_Tarleton15y

Who are you thinking of? (Eliezer is frequently accused of this, but has disclaimed it. Note the distinction between total convergence, and sufficient coherence for an FAI to act on.)

3PeerInfinity15y

(edit: The version of utilitarianism I'm talking about in this comment is total hedonic utilitarianism. Maximize the total amount of pleasure, minimize the total amount of pain, and don't bother keeping track of which entity experiences the pleasure or pain. A utilitronium shockwave scenario based on preference utilitarianism, and without any ethical restrictions, is something that even I would find very disturbing.) I totally agree!!! Astronomical waste is bad! (or at least, severely suboptimal) Wild-animal suffering is bad! (no, there is nothing "sacred" or "beautiful" about it. Well, ok, you could probably find something about it that triggers emotions of sacredness or beauty, but in my opinion the actual suffering massively outweighs any value these emotions could have.) Panspermia is bad! (or at least, severely suboptimal. Why not skip all the evolution and suffering and just create the end result you wanted? No, "This way is more fun", or "This way would generate a wider variety of possible outcomes" are not acceptable answers, at least not according to utilitarianism.) Lab-universes have great potential for bad (or good), and must be created with extreme caution, if at all! Environmental preservationists... er, no, I won't try to make any fully general accusations about them. But if they succeed in preserving the environment in its current state, that would involve massive amounts of suffering, which would be bad! I also agree with your concerns about CEV. Though of course we're talking about all this as if there is some objective validity to Utilitarianism, and as Eliezer explained: (warning! the following sentence is almost certainly a misinterpretation!) You can't explain Utilitarianism to a rock, therefore Utilitarianism is not objectively valid. Or, more accurately, our belief in utilitarianism is a fact about ourselves, not a fact about the universe. Well, indirectly it's a fact about the universe, because these beliefs were generated by a proc

7thomblake15y

Your comments are tending to be a bit too long.

1PeerInfinity15y

Thanks for the feedback. I kinda suspected that my comments were too long. So, um... what would you prefer for me to do instead? * split them into multiple comments? * post them somewhere else (the Transhumanist Wiki?) and link to them from here? * refrain from posting the long comments entirely? * find some way to cut them down? * stick to a single topic per comment, and create multiple comments if I want to discuss multiple topics? * wait longer between posting these comments? * something else I haven't thought of?

1thomblake15y

Yes, to various extents. (I should have been more helpful in the grandparent comment.) I think the main problem is you seem to have a "stream of consciousness" style of writing. If you add an additional step of editing after (I'm just assuming you're not doing much of this now), then you can figure out which points are most important to make and put them succinctly. The advantage of this, from a utilitarian point of view, is that you can spend less time editing than it will take any particular person to otherwise figure out what you're trying to say, and thus cause a net benefit to lots of people. (ETA: note that the great-grandparent comment seems less subject to this particular criticism than some others)

4PeerInfinity15y

Thanks again for the feedback. As I was writing the following points, I noticed that I was just making excuses. But instead of deleting them, I left them in, but commented on them, because they felt important and relevant. * I was already aware of the utilitarian argument that it's worth 1 minute of effort at rewriting in order to save 60 people one second each at reading, and I am making at least some attempt to do that. (correction: no, I didn't actually do the math. I should at least try to do the math.) * I already spend lots of time reviewing my comments before I post them. I don't post them until I scan through them once without noticing anything wrong. (correction: no, lately I've been posting them before I complete a full scan without finding any new issues, and I've been fixing some things by editing the comments after posting them. I should be more strict about following this rule. and as I mention below, I should add new issues to the list of things to scan for.) * Normally I have the opposite problem, spending way too much time reviewing what I wrote, which ends up resulting in other important things not getting said, because I'm spending too much time reviewing and never get around to writing the next thing. (correction: this will probably become less of an issue now that I've finished writing all of these "about me" comments.) * It usually feels like there's a sense of urgency, that if I take too long to write a reply, then everyone will have moved on to other topics, and noone will end up reading my comment. (correction: sometimes there is a reason to post stuff asap, other times there isn't. I need to learn how to tell the difference.) But these are just excuses. If I'm going to continue posting comments, then I had better learn how to improve the quality of my comments. The stream-of-consciousness style comments were something I wanted feedback on, and now I got the feedback, thanks. The feedback says that stream-of-consciousness-style co

3David Althaus14y

This is probably too late, but I really love your writing style, especially your stream of consciousness.

3NancyLebovitz15y

The "excuse generator" points at something I suspect is a very fast and active part of a lot of people's minds, but it's probably worth a post or at least an extended open thread comment of its own. As far as I can tell, I write so as to make things clear to the state of mind I was in just before I thought of something I'm trying to get across.

0PeerInfinity15y

Thanks for the feedback, that last sentence sounds like a good idea, I'll go ahead and try it. There probably have already been lots posts about the "excuse generator", though not specifically by that name. For example, Eliezer's post Against Devil's Advocacy Though that's not quite the same thing. And then there's all the posts on rationalization.

3Utilitarian15y

Indeed. It may be rare among the LW community, but a number of people actually have a strong intuition that humans ought to preserve nature as it is, without interference, even if that means preserving suffering. As one example, Ned Hettinger wrote the following in his 1994 article, "Bambi Lovers versus Tree Huggers: A Critique of Rolston"s Environmental Ethics": "Respecting nature means respecting the ways in which nature trades values, and such respect includes painful killings for the purpose of life support." Indeed. Like many others here, I subscribe to emotivism as well as utilitarianism. Yes, that's the ideal. But the planning fallacy tells us how much harder it is to make things work in practice than to imagine how they should work. Actually implementing CEV requires work, not magic, and that's precisely why we're having this conversation, as well as why SIAI's research is so important. :) I hope so. Of course, it's not as though the only two possibilities are "CEV" or "extinction." There are lots of third possibilities for how the power politics of the future will play out (indeed, CEV seems exceedingly quixotic by comparison with many other political "realist" scenarios I can imagine), and having a broader base of memetic support is an important component of succeeding in those political battles. More wild-animal supporters also means more people with economic and intellectual clout. If you include paperclippers or suffering-maximizers in your definition of "anyone," then I'd put the probability close to 0%. If "anyone" just includes humans, I'd still put it less than, say, 10^-3. Yeah, although if we take the perspective that individuals are different people over time (a "person" is just an observer-moment, not the entire set of observer-moments of an organism), then any choice at one instant for pain in another instant amounts to "forcing someone" to feel pain....

0thomblake15y

That is inconsistent. Utilitarianism has to assume there's a fact about the good; otherwise, what are you maximizing? Emotivism insists that there is not a fact about the good. For example, for an emotivist, "You should not have stolen the bread." expresses the exact same factual content as "You stole the bread." (On this view, presumably, indicating "mere disapproval" doesn't count as factual information).

1Utilitarian15y

Sure. Then what I meant was that I'm an emotivist with a strong desire to see suffering reduced and pleasure increased in the manner that a utilitarian would advocate, and I feel a deep impulse to do what I can to help make that happen. I don't think utilitarianism is "true" (I don't know what that could possibly mean), but I want to see it carried out.

0PeerInfinity15y

checking out the wikipedia article... hmm... I think I agree with emotivism too, to some degree. I already have a habit of saying "but that's just my opinion", and being uncertain enough about the validity (validity according to what?) of my preferences, to not dare to enforce them if other people disagree. And emotivism seems like a formalization of the "but that's just my opinion". That could be useful. good point. and yeah, that's that's one of the main issues that's causing me to doubt whether SIAI has any hope of achieving their mission. good point. Have you had any contact with Metafire yet? He strongly agrees with you on this. Just recently he started posting to LW. oh, and "quixotic", that's the word I was looking for, thanks :) heh, yeah, that "significantly less than 50%" was actually meant as an extremely sarcastic understatement. I need to learn how to express stuff like this more clearly. good point! This suggests the possibility of requiring people to go through regular mental health checkups after the Singularity. Preferably as unobtrusively as possible. Giving them a chance to release themselves from any restrictions they tried to place on their future selves. Though the question of what qualifies as "mentally healthy" is... complex and controversial.

2Jack15y

When discussing utilitarianism it is important to indicate whether you're talking about preference utilitarianism or hedonistic utilitarianism, especially in this context.

0PeerInfinity15y

Right, sorry. I'm referring to total hedonic utilitarianism. Maximize the total amount of pleasure, minimize the total amount of pain, and don't bother keeping track of which entity experiences the pleasure or pain. A utilitronium shockwave scenario based on preference utilitarianism, and without any ethical restrictions, is something that even I would find very disturbing.

0Utilitarian15y

Indeed. While still a bit muddled on the matter, I lean toward hedonistic utilitarianism, at least in the sense that the only preferences I care about are preferences regarding one's own emotions, rather than arbitrary external events.

1Strange715y

You could also almost certainly convert a considerable percentage of the planet's mass to computronium without impacting the planet's ability to support life. A planet isn't a very mass-efficient habitat, and I doubt many people would even notice if most of the core was removed, provided it was replaced with something structurally and electrodynamically equivalent.

2NancyLebovitz15y

You need the mass of the core to maintain the gravity. What sort of physics do you have in mind?

2Strange715y

If computronium is of density equal to or greater than iron, physics wouldn't need to be changed. Remove the core, replace it with a roughly spherical wad of perfected brain-matter, plus whatever structural supports are necessary to keep the crust in place, and Newton's Shell Theorem says gravity would be the same. Add some electromagnets for the poles, and channel waste heat from the mechanisms inside to simulate volcanism where appropriate. Even if computronium turns out to have lower density than iron, and for whatever reason it's unacceptable to reduce surface gravity or transplant the luddites to an otherwise earthlike planet of correspondingly greater diameter, some of the core's mass could be converted and the remainder compressed into a black hole. Again, shell theorem means there's no difference from the outside.

1PeerInfinity15y

good point, thanks for mentioning that. heh, that's actually what I meant by leaving the planet "mostly intact", but I should have made that clearer.

6cupholder15y

Guess there's a use for that-guy after all!

5MatthewB15y

A couple of points: I could not tell from your post if you understood that Pascal's Wager is a flawed argument for believing in ANY belief system. You do understand this don't you (That Pascal's Wager is horribly flawed as an argument for believing in anything)? Also, as Counsin it seems to be implying (And I would suspect as well), you seem to be exhibiting signs of the True Believer complex. This is what I alluded to when I discussed friends of mine who would swing back and forth between Born-Again Christian and Satanists. Don't make the same mistake with a belief in the Singularity. One needn't have "Faith" in the Singularity as one would God in a religious setting, as there are clear and predictable signs that a Singularity is possible (highly possible), yet there exists NO SUCH EVIDENCE for any supernatural God figure. Forming beliefs is about evidence, not about blindly following something due to a feel good that one gets from a belief.

1khafra15y

In chapter five of Jaynes, "Queer Uses for Probability Theory," he explains that although a claimed telepath tested 25.8 standard deviations away from chance guessing, that isn't the probability we should assign to the hypothesis that she's actually a telepath, because there are many simpler hypotheses that fit the data (for instance, various forms of cheating). This example is instructive when using Pascal's Wager to minimax expected utility. Pascal's Wager is a losing bet for a Christian, because even though expecting positive infinity utility with infitesimal probability seems like a good bet, there are many likelier ways of getting negative infinity utility from that choice. Doing what you can to promote a friendly singularity can still be called "Pascal's Wager" because it's betting on a very good outcome with a low probability, but the low probability is so many orders of magnitude better than Christianity's that it's actually a rather good bet. Obviously, you don't want to let wishful thinking guide your epistemology, but I don't think that's what PI's talking about.

-2byrnema15y

Pascal's wager is not such a horribly flawed argument. In fact, I wager we can't even agree on why its flawed. Later edit: I assume I am getting voted down for trolling (that is, disrupting the flow of conversation), and I agree with that. An argument about Pascal's wager is not really relevant in this thread. However, especially in the context of being a 'true believer', it is interesting to me that statements are often made that something is 'obvious', when there are many difficult steps in the argument, or 'horrible flawed', when it's actually just a little bit flawed or even controversially flawed. If anyone wants to comment in a thread dedicated to Pascal's wager, we can move this to the open thread, which I hope ultimately makes this comment less trollish of me.

3Nick_Tarleton15y

Partially seconded. (I think most people agree that the primary flaw is the symmetry argument, but I don't think that argument does what they think it does, and I do see people holding up other, minority flaws. I do think the classic wager is horribly flawed for other, related but less commonly mentioned, reasons.) I'll write a top-level post about this today or tomorrow. (In the meantime, see Where Does Pascal's Wager Fail? and Carl Shulman's comments on The Pascal's Wager Fallacy Fallacy.)

1byrnema15y

Thanks for the link to the Overcoming Bias post. I read that and it clarified some things for me. If I had known about that post, above I would have just linked to it when I wrote that the fallacy behind Pascal's wager is probably actually unclear, minor or controversial.

2SilasBarta15y

There aren't many difficult steps in refuting Pascal's wager, and I dont' think there's be much disagreement on it here. The refutation of PW, in short, is this: it infers high utility based on a very complex (and thus highly-penalized) hypothesis, when you can find equally complex (and equally well-supported) hypotheses that imply the opposite (or worse) utility. (Btw, I was one of those who voted you down.)

1byrnema15y

Again, is it the argument that is wrong, or Pascal's application of it? (Can you confirm whether you down-voted me because it's off-topic and inflammatory, or because I'm wrong?)

0SilasBarta15y

It is always wrong to give weight to hypotheses beyond that justified by the evidence and the length penality (and your prior, but Pascal attempts to show what you should do irrespective of prior). Pascal's application is a special case of this error, and his reasoning about possible infinite utility is compounded by the fact that you can construct contradictory advice that is equally well-grounded. I downvoted you not just for being wrong, but for having made such a bold statement about PW without (it seems) having read the material about it on LW. I also think that such over-reaching trivializes the contribution of writers on the topic and so comes off as inflammatory.

0byrnema15y

Are you saying, here, that it is wrong to factor in the utility of the hypothesis when giving weight to the hypothesis? If he didn't consider all the cases, his particular application of the argument was bad, not the argument itself, right? I have read the material, but I disagreed with it, and it's often not clear -- especially when the posts are old -- how I can jump in and chime in that I don't agree. Often it's just the subtext I disagree with, so I wait for someone to make it more explicit (or at least more immediate) and then I bring it up. Thanks for your explanation about the down-voting.

4SilasBarta15y

No (assuming you mean the expected utility of the action given the hypothesis), just that you have to accurately weight its probability. But his argument wouldn't somehow be improved by considering all the cases (not that it would be practical to even consider all the hypotheses of lengths up to that which implies high utility from faith in God!). Considering those cases would find hypotheses that assign the opposite utility to faith, and worse, some would be more probable. To salvage the argument, one would have to not just consider more cases, but provide a lot more epistemic labor -- that is, make arguments that aren't part of PW to begin with.

0byrnema15y

All of your objections to PW seem to be about Pascal's application of the argument (the probabilities he inputted, the number of cases cases he considered) in which case we can agree that his conclusion wouldn't be correct. When I read that Pascal's Wager is flawed as an argument, I interpret this as 'the argument does not have good form'. Did people just mean, all along, that they disagreed with the conclusion of the argument because they didn't agree with the numbers he used?

6SilasBarta15y

I think what they mean is, "If an argument allows you to claim an unreasonably huge amount of utility from actions not seemingly capable of that, then you have a complex enough hypothesis that you can find others with the same complexity and opposite conclusion". PW-type arguments, then, refer to the class of arguments in which someone tries to justify a course of action through (following the action suggested by) an improbable hypothesis by claiming high enough expected utility. That class of arguments has the flaw that when you allow yourself that much complexity, you necessarily permit hypotheses that advise just as strongly against the action. That is not something that you can salvage by using different numbers here and there, and so the argument and similar ones have bad (and unsalvageable) form.

1byrnema15y

That is still fine, because we know how to handle the hypotheses with negative utility. You just optimize over the net utilities of each belief weighted by their probabilities.The fact that there are positive and negative terms together doesn't invalidate the whole argument. You just do the calculation, if you can, and see what you get. If you have the right numbers, and a simple enough case to do the computation, would you find PW an acceptable argument?

5SilasBarta15y

I'm still having trouble understanding your objection. When you decide to have faith based on PW, you're using some epistemology that allows you to pick out the "faith causes infinite utility" hypothesis out of the universe-generating functionspace, and deem it to have some finite probability. The problem is that that epistemology -- whatever it is -- also allows you to pick out numerous other hypotheses, in which some assert the opposite utility from faith (and their existence is provable by inversion of the faith = utility hypothesis elements). In order to show net positive utility from believing, you would have to find some way of counting all hypotheses this complex, and finding out which comes ahead. However, the canonical PW argument relies on such anti-faith hypotheses not existing. You would be treading new ground in finding some efficient way to count up all such hypotheses and find which action comes out ahead -- keeping in mind, of course, that at this level of complexity, there is a HUGE number of hypotheses to consider. So you would be making a new argument, only loosely related to canonical PW. If you think you can pull this off, then go ahead and write the article, though I think you'll soon find it's not as easy as you expect. And I would submit that any hypothesis that allows you to claim something has infinite utility (or necessarily more utility than the result of any other action) must itself be infinitely complex, thus infinitely improbable, canceling out the infinity claimed to come from faith.

3byrnema15y

As you know, I think the essence of Pascal's wager is this: I think there is enough to debate about in that statement alone. But suppose that X = God exists. It seems to me that you are consistently writing that Pascal's Wager fails because in this case the utility of X is impossible to compute due to the complexity of X. I don't believe this makes the argument fail for two reasons: 1. Pascal's Wager says, "If belief in X has positive utility, you should believe in X'. This argument doesn't fail (in form) if the utility is negative or impossible to compute. 2. I disagree that the utility is impossible to compute, despite all your arguments about the complexity of X. My reason is straight-forward: atheists do calculate (or at least estimate) the utility of believing in God. Usually, they come up with a value that is negative. So it's not impossible to estimate the average utility of a complex belief.

2orthonormal15y

That's not quite valid— there is some finite program that unfolds Permutation City-style into a universe that allows for infinite computational power, and thus (by some utility functions) infinite utility as the consequence of some actions. It would be wrong for a scientist living in such a universe to reject that hypothesis.

2JGWeissman15y

The reason I believe Pascal's wager is flawed is that it is a false dichotomy. It looks at only one high utility impact, low probability scenario, while excluding others that cancel out its effect on expected utility. Is there anyone who disagrees with this reason, but still believes it is flawed for a different reason?

1byrnema15y

This is an argument for why the argument doesn't work for theism, it doesn't mean the argument itself is flawed. If you would be willing to multiply the utility of each belief times the probability of each belief and proceed in choosing your belief in this way, then that is an acceptance of the general form of the argument.

0JGWeissman15y

If you assume that changing your belief is an available action (which is also questionable), then the idealized form is just expected utility maximization. The criticism is that Pascal incorrectly calculated the expected utility.

0byrnema15y

Right, one flaw in the idealized form is that it's not clear that you can simply choose the belief that maximizes utility. But in some cases a person can, and does. I think that an incorrect calculation, because one person considered 2 cases instead of N cases, is very different from being flawed as an argument. PeerInfinity was writing about applying Pascal's wager to atheism -- so he must have been referring to the general form of the argument, not a particular application. Matthew B wrote that "Pascal's Wager is a flawed argument for believing in ANY belief system". Well, what about a belief system in which there are exactly two beliefs to choose from and the relative probabilities are (.4, .6) and the relative utilities of having the beliefs if they are true are (1000, 100) ? I would say the conclusion of the idealized form of Pascal's wager is that you should pick the belief that maximizes utility, even though it is lower probability.

2JGWeissman15y

I would distinguish between the general form and the idealized general form. One way to generalize Pascal's wager for belief B, is to compare the expected utilities of believing B and believing one contradictory Belief D in the conditions that B is true and that D is true. This is wrong no matter what belief B you apply it to.

2RobinZ15y

Why would having the beliefs have utility? Isn't utility a function of actions, as a rule? There's no contradiction in thinking "A is unlikely" and yet acting as if A is true - otherwise no-one would wear seat belts.

0byrnema15y

The utility of having a belief is what is being considered in Pascal's wager, and is quite different from the utility of the belief itself. The utility of a belief itself wouldn't sway you to choose one belief over another. Suppose againyou have the two beliefs X and Y, and they each have a certain utility if they are true. If X is true, then you "get" that utility, independently of whether you believed it or not, by virtue of it being true. For example, if there is utility to God existing, then there is that benefit of him existing whether you believe in him or not. In contrast, there is also utility for having a belief. To complicate things, there is a component of the utility that is independent of whether the belief is true or not, and there is a component of the utility that depends on the belief being true. In the case of theism, there is a utility to being a theist (positive or negative, depending on who you ask) regardless of whether God exists, and there would also be an extra utility for believing in him if he does exist (possibly zero, if he doesn't care whether you believe in him or not).

0RobinZ15y

SilasBarta has pointed out a relevant argument regarding that case.

0byrnema15y

You mean the case of the argument applied to theism? I would be willing to forfeit the applicability of the argument for this case, since I'm just interested in discussing the validity of the general argument.

0RobinZ15y

I don't like discussing general cases when I don't have some concrete examples. The only ones I can think of are boring cases of coercion involving unethical mindreaders.

2byrnema15y

Yes, I agree: the utility of having a belief only makes sense when for some reason you are rewarded for actually having the belief instead of acting as though you have the belief. OK, since theism is unique in this aspect, in order to generalize away from the theistic, let's use the utility for acting-as-though-you-believe instead of the utility for actually believing, because in most cases, these should be the same. ... but then, as soon as you do this, the argument become just about choosing actions based on average expected utility and there's nothing controversial about it. So I guess PW might just suffer from lack of application: there are few cases where you are actually differentially rewarded for having a belief (instead of just acting as though you do), and these cases (generalizing from theism) involve hypotheses that are too complex to parametrize (Silas' argument). ---------------------------------------- Back to the immediate object level: PeerInfinity wrote about applying Pascal's Wager to atheism. However, atheism doesn't make a utility distinction between having a belief and acting as though you do. Or does it? Having beliefs motivate actions and make them easier to compute. When PeerInfinity said he chose to believe atheism because it seemed to maximize utility, he might have been summarizing together that acting as though atheism was true was deemed utility maximal, and believing in atheism then followed as utility maximal.

0[anonymous]15y

I also think Pascal's Wager is not horribly flawed in the ways it's most commonly claimed to be, and am aggrieved that this interesting and important discussion is taking place under a downvoted-to-invisibility comment on an unrelated post. I think I'll write a top-level post about it today or tomorrow, but right now, I'd like to humbly ask that the above comment be upvoted until not invisible.

0RobinZ15y

Taboo "Pascal's wager", please.

0byrnema15y

Sure. Here's an argument: Suppose there is a dichotomy of beliefs, X and Y, their probabilities are Px and Py, and the utilities of having each belief is Ux and Uy. Then, the average utility of having belief X is PxUx and the utility of having belief Y is Py\Uy. You "should" choose having the belief (or set of beliefs) that maximizes average utility, because having beliefs are actions and you should choose actions that maximize utility. What is the flaw in this argument? For me, the flaw that you should identify is that you should choose beliefs that are most likely to be true, rather than those which maximize average utility. But this is a normative argument, rather than a logical flaw in the argument.

4Vladimir_Nesov15y

Normally, you should keep many competing beliefs with associated levels of belief in them. The mindset of choosing the action with estimated best expected utility doesn't apply, as actions are mutually exclusive, while mutually contradictory beliefs can be maintained concurrently. Even when you consider which action to carry out, all promising candidates should be kept in mind until moment of execution.

1mattnewport15y

This is complicated in the case of religious beliefs where the deity will judge you by your beliefs and not just your actions.

1RobinZ15y

It is also complicated in the case of religious beliefs where other human beings will judge you by your beliefs, which is one reason why abandoning religions is so hard. But that is off-topic, particularly as you can just lie.

-1mattnewport15y

While we're being off topic, I'm of the opinion that if you are someone who accepts you should one-box then you should also accept Pascal's wager. I think both are wrong but most people here seem to accept one-boxing is correct but not accept Pascal's wager. I don't care enough about either to work the argument out in detail though.

6SilasBarta15y

Newcomb's problem is just a case of making decisions when someone else, who "knows you very well" has already made a decision based on expectation of your decision. There are numerous real-world examples of this. Newcomb's problem only differs in that it takes the limit of the "how well they know you" variable as it approaches "perfect". There needn't be an actual Omega, just a decision theory that is robust for all values of the variable up to and including perfect.

0mattnewport15y

Which sounds a lot like Pascal's wager to me, when your decision is whether to believe in god and god is the person who "knows you very well" and is deciding whether to let you into heaven based on whether you believe in him or not. There are situations which I guess are what you would describe as 'Newcomb-like' where I would do the equivalent of one-boxing. If Omega shows up this evening though I will be taking both his boxes, because there is too big an epistemic gap for me to cross to reach the point of thinking that one-boxing is sensible in this universe.

0RobinZ15y

But the plausibility of a hypothetical is unrelated to the correct resolution of the hypothetical. One could equally say that two-boxing implies that you should push the man off the bridge in the trolley problem - the latter is just as unphysical as Newcomb. The proper objection to unreasonable hypotheticals is to claim that they do not resemble the real-world situations one might compare them to in the relevant aspects.

0mattnewport15y

I actually think that implausible hypotheticals are unhelpful and probably actively harmful which is why I usually don't involve myself in discussions about Omega. I wish I'd stuck with that policy now.

0NancyLebovitz15y

Why do you think implausible hypotheticals are unhelpful and probaby harmful? It seems to me that they're a lot of work for no obvious reward, but I don't have a more complex theory. Anyone have an example of the examination of an implausible hypothetical paying off?

1mattnewport15y

I think implausible hypotheticals are often intuition pumps. If they are used as part of an attempt to convince the audience of a certain point of view I automatically get suspicious. If the point of view is correct, why can't it be illustrated with a plausible hypothetical or a real world example? They often seem to be constructed in a way that tries to move attention away from certain aspects of the situation described and thus allow for dubious assumptions to be hidden in plain sight. Basically, I always feel like someone is trying to pull a philosophical sleight of hand when they pull out an implausible hypothetical to make their case and they often seem to be used in arguments that are wrong in subtle or hard to detect ways. I feel like I encounter them far more in arguments for positions that I ultimately conclude are incorrect than as support for positions I ultimately conclude to be correct.

2NancyLebovitz15y

That's interesting, and might apply to the trolley problem which implies that people can have much more knowledge of the alternatives than they are ever likely to have. Ethical principles and empathy (as a sort of unconscious ethical principle) are needed when you don't have detailed knowledge, but I haven't seen the trolley problem extended to the usual case of not knowing very many of the effects. It might be worth crossing the trolley problem with Protected from Myself. Taking a look at ethical intuitions with specifics: Sex, Drugs, and AIDS: the desire to only help when it will make a big difference and the desire to not help unworthy people add up to worse effects than having a less dramatic view of the world. Having AIDS drugs doesn't mean it makes sense to slack off on prevention as much as has happened.

0mattnewport15y

Yes, the trolley problems are another example of harmful implausible hypotheticals in my opinion. The different reaction many people have to the same underlying ethical question framed as a trolley problem vs. an organ donor problem is I think illustrative of the pernicious influence of implausible hypotheticals on clear thought.

0Jack15y

Well, the fact that they're implausible pretty much means the cash rewards are going to have to wait until they are plausible. But don't we think clear thinking is its own reward? I've found that such things are incredibly crucial for getting people to think clearly about personal identity. In fact I don't know if I have any way of explaining or defending my views on personal identity to the philosophically untrained without implausible hypotheticals. Same goes for understanding skepticism, causality, maybe induction, problems with causal decision theory (obviously), anthropics, simulation... I'm all about being aware that using implausible hypotheticals can generate error but I am bewildered by the sudden resistance to them on this thread: we use them all the time here!

0RobinZ15y

I would be dead chuffed to talk about the wisdom of considering implausible hypotheticals instead, if that's what you'd prefer to do. (: Edit: I would be equally happy to drop the thread entirely, if that's what you prefer.

5mattnewport15y

Ok, let me try and nail down my true objection here. Is Pascal's wager a good reason to believe in God? No. Hypothetically, if you had good reason to believe that the hypothesis of the christian god existing were massively more likely than other hypotheses of similar complexity, would it be a good reason to believe in god? Well, not really - it doesn't add much in that case. Similarly, if Omega showed up at my apartment this evening would I one-box? No. Hypothetically, if I had good reason to believe that an Omega-like entity existed and did this kind of thing (which is the set up for Newcomb's problem) would I one-box? Well, probably yes but you've glossed over the rather radical change to my epistemic state required to make me believe such an implausible thing. I guess I have a general problem with a certain kind of philosophical thought experiment that tries to sneak in a truly colossal amount of implausibility in its premises and ask you not to notice and then whenever you keep pointing to the implausibility telling you to ignore it and focus on the real question. Well I'm sorry, but the staggering implausibility over there in the corner is more significant than the question you want me to focus on in my opinion... (Forgive the casual use of 'you' here - I'm not intending to refer to you specifically).

3Jack15y

I don't understand. A hypothetical can be dangerous if it keeps us from attending to aspects of the problem we're trying to analyze- like the Chinese Room which fails to convey properly the powers it would have to have for us to declare it conscious. The fact that a hypothetical is implausible might make it harder for us to notice that we're not attending to certain issues, I guess. That hardly seems grounds for rejecting them outright (indeed, Dennett uses plenty of intuition pumps). And the implausibility itself really is irrelevant. No one is claiming that the hypothetical will occur, so why should the probability of its occurrence be an issue?

0mattnewport15y

Using Newcomb's problem as an example, it seems like it glosses over important details of how much evidence you would actually need to believe in an Omega like entity and as a result confuses more than it illuminates. Re-reading some of Eliezer's posts on it I get the impression that he is hinting that his resolution of the issue is connected to that problem. It seems to me that it causes a lot of unnecessary confusion because humans are susceptible to stories that require suspension of disbelief in highly implausible occurrences that they would not actually suspend their disbelief for if encountered in real life. This might be an example of Robin Hanson's near/far distinction. Tyler Cowen's cautionary tale about the dangers of stories covers some of the same kinds of human biases that I think are triggered by implausible hypotheticals.

1Jack15y

It certainly does gloss over that... I mean it has to, you'd require a lot of evidence. But the reason it does so is because the question isn't could Omega exists or how can we tel when Omega shows up... the details are buried because they aren't relevant. How does Newcomb's problem confuse more that illuminate? It illustrates a problem/paradox. We would not be aware of that paradox were it not for the hypothetical. I suppose it confuses in the sense that one becomes aware of a problem they weren't previously- but that's the kind of confusion we want. It's a great video and I'm grateful you linked me to it but I don't see where the problems with the kind of stories Cowen was discussing show up in thought experiments.

4PhilGoetz15y

The danger is that you can use a hypothetical to illustrate a paradox that isn't really a paradox, because its preconditions are impossible. A famous example: Suppose you're driving a car at the speed of light, and you turn on the headlights. What do you see?

0Jack15y

This is a danger. Good point.

0mattnewport15y

It confuses because it doesn't really show a problem/paradox. That is not obvious because of the peculiar construction of the hypothetical. If you actually had enough evidence to make it seem like one-boxing was the obvious choice then it wouldn't seem like a paradoxical choice. The problem is people generally aren't able to imagine themselves into such a scenario and so think they should two-box and then think there is a paradox (because you 'should' one-box). They quite reasonably aren't able to imagine themselves into such a scenario because it is wildly implausible. The paradox is just an artifact of difficulties we have mentally dealing with highly implausible scenarios. Specifically what I had in mind was the fact that people seem to have a natural willingness to suspend disbelief and accept contradictory or wildly implausible premises when 'story mode' is activated. We are used to listening to stories and we become less critical of logical inconsistencies and unlikely scenarios because they are a staple of stories. Presenting a thought experiment in the form of a story containing a highly implausible scenario takes advantage of a weakness in our mental defenses which exists for story-shaped language and leads to confusion and misjudgement which we would not exhibit if confronted with a real situation rather than a story.

2Jack15y

No. The choice is paradoxical because no matter how much evidence you have of Omega's omniscience the choice you make can't change the amount of money in the box. As such traditional decision theory tells you to two- box because the decision you make can't affect the amount of money the boxes. No matter how much money is in the boxes you should more by two boxing. Most educated people are causal decision makers by default. So a thought experiment where causal decision makers lose is paradox inducing. If one-boxing was the obvious choice people would feel the need to posit new decision theories as a result.

0mattnewport15y

I disagree, and I think this is what Eliezer is hinting towards now I've gone back and re-read Newcomb's Problem and Regret of Rationality. If you really have had sufficient evidence to believe that Omega is either an omniscient mind reader or some kind of acausal agent such that it makes sense to one-box then it makes sense to one-box. It only look like a paradox because you're failing to imagine having that much evidence. Which incidentally is not really a problem - an inability to imagine highly implausible scenarios in detail is not generally an actual handicap in real world decision making. I'm still going to two-box if Omega appears tomorrow though because there are very many more likely explanations for the series of events depicted in the story than the one you are supposed to take as given.

0byrnema15y

Curiously, what is the average utility you would estimate for belief in God? Or do you feel that trying to estimate this forces suspended disbelief in implausible scenarios?

3mattnewport15y

Which god? The God Of Abraham, Isaac, And Jacob? The Christian, Muslim or Jewish flavour? It would seem this is quite important in the context of Pascal's wager. Some gods are notoriously specific about the form my belief should take in order to win infinite utility. I don't see any compelling evidence to prefer any of the more popular god hypotheses over any other, nor to prefer them over the infinitude of other possible gods that I could imagine. Some of the Norse gods were pretty badass though, they might be fun to believe in.

0byrnema15y

... if I may put the question differently: what average utility do you estimate for not believing in any God?

2mattnewport15y

This strikes me as a rather odd question. I thought we were more or less agreed that beliefs don't generally have utility. The peculiarity of Pascal's wager and religious belief in general is that you are postulating a universe in which you are rewarded for holding certain beliefs independently of your actions. In a universe with no god (which I claim is a universe much like our own) belief in god is merely false belief and generally false beliefs are likely to cause bad decisions and thus lead to sub-optimal outcomes. If the belief in god is completely free-floating and has no implications for actions then it may not have any direct negative effect on expected utility. Presumably given the finite computational capacity of the human brain holding non-consequential false beliefs is a waste of resources and so has slight negative utility. It strikes me that this is not the kind of belief in god that people are usually trying to defend when invoking Pascal's wager however.

1NancyLebovitz15y

I'm not sure that beliefs don't generally have utility. It seems to me that beliefs (or something like beliefs) do a lot to organize action. There's a difference between doing something because of short-term reward and punishment and doing the same thing because one thinks it's generally a good idea.

0byrnema15y

Hmm. I think beliefs do have a utility, whether or not you can act on that utility by choosing a belief or whether or not you can accurately estimate the utility. If you believe something, you will act as though you believe it, so that believing in something inherits the utility of acting as though you do. It seems very strange to think of someone acting as though they believe something, without them actually believing it. There are exceptions, but for the most part, if someone bets on a belief, this is because they believe it.

2mattnewport15y

I don't in general agree with this. Outcomes have utility, actions have expected utility, beliefs are generally just what you use to try and determine the expected utility of actions. As a rule, true beliefs will allow you to make better estimates of the expected utility of actions. This is true for ordinary beliefs: I believe it is raining so I expect the action of taking my umbrella to have higher utility than if I did not believe it was raining. It is possible to imagine certain kinds of beliefs that have utility in themselves but these are unusual kinds of beliefs and most beliefs are not of this type. If there is a god who will reward or punish you in the afterlife partly on the basis of whether you believed in him or not then 'believing in god' would result in an outcome with positive utility but deciding if you live in such a universe would be a different belief that you would need to come to from other kinds of evidence than Pascal's wager. It is possible to imagine other beliefs that could in theory have utility in themselves for humans. For example, it is possible that believing oneself a bit more attractive and more competent than is accurate might benefit ones happiness more than enough to compensate for lost utility due to less accurate beliefs leading to actions with sub-optimal expected utility. If this is true however it is a quirk of human psychology and not a property of the belief in the way that Pascal's wager works. I don't find it at all strange to think of someone acting as if they believe in god even though they don't. This has been common throughout history.

0RobinZ15y

That looks like a good heuristic you are using - it seems related to the idea of the intuition pump. ...wow, that was a short time-to-agreement. :D

2mattnewport15y

Yeah, I think I was always averse to this sort of philosophical sophistry but reading Consciousness Explained probably crystallized my objection to it at a relatively early age.

0RobinZ15y

I think you're mistaken, therefore I would like to see your proof. It would be a shame if I missed an opportunity to be more correct. ;)

4mattnewport15y

They both have an element of privileging the hypothesis. If I had some reason to think I lived in a universe with an Omega/God then I might agree I should one-box/believe in god but since I don't have any reason to think I live in such a universe why am I wasting my time even considering this particular implausible scenario?

2RobinZ15y

I see what you mean, but there exists one of two problems with the symmetry. First, the most annoying form of Pascal's Wager is the epistemological version: "Believing that God exists has positive expected utility, so you should do so". This argument fails logically, for reasons SilasBarta listed, and it is usually this form being refuted when people say, "Pascal's Wager fails". Second, the form of Pascal's Wager concerning worship, "Believing in God, who is known to exist, has positive utility", has moral complexities which are absent from Newcomb's dilemma. Objections in this case usually arise from the normative argument that you should not believe things which are false.

0byrnema15y

I disagree that it fails logically. The argument, written modus ponens, is: "If believing in God has positive expected utility, then you should do so". If you don't believe that believing in God has positive expected utility, then this is not a disagreement in the logic of Pascal's Wager. Pascal's Wager would equally say, "If believing in God has negative expected utility, then you should not do so".

2SilasBarta15y

Okay, now I think I'm starting to see the miscommunication: PW does not simply say what you've quoted there. It's typically associated with an argument about how the possibility of infinite utility from believing (and perhaps infinite disutility from not believing) outweights the small probability of it being true, and the utility of other courses of action, on account of its infinite size. You're taking "Pascal's Wager" to refer only to certain premises the argument uses, not the full argument itself.

0byrnema15y

It occurred to me that you might not agree that my distillation of PW contained all the salient features. (For example, there are no infinitesimals and no infinities written in). However, I think it must have been my more general argument that PeerInfinity was referring to, because he was applying it to atheism.

0byrnema15y

Good point, I edited my form of the argument to include 'sets of beliefs'. If having a set of beliefs maximizes your utility, then having the set is what you "should" do, I think, in the spirit of the argument.

0[anonymous]15y

Accepting God as a probable hypothesis has a lot of epistemic implications. This is not just one thing, everything is connected, one thing being true implies other things being true, other things being false. You won't be seeing the world as you currently believe it to be, after accepting such change, you will be seeing a strange magical version of it, a version you are certain doesn't correspond to reality. Mutilating your mind like this has enormous destructive consequences on your ability to understand the real world, and hence on ability to make the right choices, even if you forget about the hideousness of doing this to yourself. This is the part that is usually overlooked in Pascal's wager. (Belief in belief keeps the human believers out of most of the trouble, but that's not what Pascal's wager advocates! Not understanding this distinction may lead to underestimating the horror of the suggestion.)

0RobinZ15y

Thank you. My response appears in another thread.

2Jack15y

Your story and perspective are very interesting. You don't need to self-censor.

1PeerInfinity15y

Thanks. Actually, the reason why I said "I guess I had better stop writing now" is because this comment was already getting too long.

3thomblake15y

Just a note - don't take Jack's advice to not self-censor too literally. There is much weirdness in you, and even the borders of this place would groan under its weight. Not that there's anything wrong with that.

3AdeleneDawner15y

The above (below? Depends on your settings, I guess) comment, which is now hidden, involves a poll, and would not (I predict) have otherwise become hidden.

3Blueberry15y

It's also hidden depending on your settings: you can change the threshold for hiding comments as well. I don't hide any comments, because seeing a hidden comment makes me so curious I have to click it, and just draws more attention to it for me.

-27PeerInfinity15y

1Unknowns14y

I haven't yet seen an answer to Pascal's Wager on LW that wasn't just wishful thinking. In order to validly answer the Wager, you would also have to answer Eliezer's Lifespan Dilemma, and no one has done that.

2Blueberry14y

I'm pretty sure Peer meant the original version of Pascal's Wager, the argument for Christianity, which has the obvious answer, "What if the Muslims are right? or "What if God punishes us for believing?"

0Unknowns14y

That's not an answer, because the probabilities of those things are not equal. "God punishes us for believing" has a much lower probability, because no one believes it, while many people believe in Christianity. "Muslims are right" could easily be more probable, but then there is a new Wager for becoming Muslim. The probabilities simply do not balance perfectly. That is basically impossible.

3Blueberry14y

Why does the probability have anything to do with the number of people who believe it? There's then the problem that the expected value involves adding multiples of positive infinity (if you choose the right religion) to multiples of negative infinity (if you choose the wrong one), which gives you an undefined result. The probability of any kind of God existing is extremely low, and it's not clear we have any information on what kind of God would exist conditioned on some God existing. There's also the problem that if you know the probability that God exists is very small, you can't believe, you can only believe in belief, which may not be enough for the wager.

1Unknowns14y

The probability has something to do with the number of people who believe it because it is possible that some of those people have a good reason to believe it, which automatically gives it some probability (even if very small.) But for positions that no one believes, this probability is lacking. That adding positive and negative infinity is undefined may be true mathematically, but you have to decide one way or another. And it is wishful thinking to say that it is just as good to choose the less probable way as the more probable way. For example, there are two doors. One has a 99% chance of giving negative infinite utility, and a 1% chance of positive infinite. The second door has a 1% chance of negative infinite utility, and a 99% chance of positive infinite utility. Defined or not, it is perfectly obvious that you should choose the second door. We do have information on what kind of God would exist if one existed: it would probably be one of the ones that are claimed to exist. Anyway, as Nick Bostrom points out, even without this kind of evidence, the probabilities still will not balance EXACTLY, since you will have some evidence even from your intuitions and so on. It may be true that some people couldn't make themselves believe in God, but only in belief, but that would be a problem with them, not with the argument.

3Blueberry14y

This can't be right. The number of people who follow any one religion is affected by how people were raised, by cultural and historical trends, by birth rates, and by the geographic and social isolation of the people involved. None of these things have anything to do with truth. Currently Christianity has twice as many people as any other religion because of historical and political facts; you think this makes it more likely than Islam to be true? Suppose that in 50 years, because of predicted demographic trends, there are twice as many Muslims as Christians. You then seem to be in the strange position of thinking (a) Christianity is more likely to be true now, but (b) because of changing demographics, you will be likely to think Islam is more likely to be true in 50 years. How do people's claims give you that information? Religions are human cultural inventions. At most one could be true, which means the others have to be made up anyway. If a God did exist, why is it more likely that one of them is true than that they were all made up and humanity never came close to guessing the nature of the God that did exist? My intuition tells me that if a God of some sort does exist, the probabilities end up favoring a God that rewards looking at the evidence and believing only what you have reason to be true, but that may just be my bias showing. Intuition about what religion is true is likely to reflect your upbringing and your culture more than the actual truth. Given that there's currently no evidence of any kind of God or afterlife, I can't see how there is any evidence that God X is more likely to exist than God Y. It's also worth noticing that Pascal's Wager uses a spherical cow version of religion. Some religious traditions might require actual belief for infinite utility, others just belief in belief, others just certain behavior or words independent of belief.

0Unknowns14y

I'll answer this later. For now I'll just point out that you aren't addressing my position at all, but other things which I never said. For example, I said that if people believe something, this increases its probability. You respond by asking things like "Currently Christianity has twice as many people... you think this makes it more likely than Islam to be true?" I definitely did not say that the probability of a religion is proportional to the number of people who believe it, just that religions that some people believe are more likely than ones that no one believes.

3TobyBartels14y

Right; or if you don't decide exactly, at least you have to do (believe or not believe) one or the other. I would say that the model breaks down. Mathematics (or at least the particular mathematical model being used) is not capable of describing this situation, but that doesn't make the situation itself meaningless. (That would be a version of the map/territory fallacy.) Here I disagree with you. I would say that you have not given enough information. It is as if you gave the same problem statement but with the word ‘infinite’ removed (so that we only know whether the utilities are positive or negative). It may seem as if you have given all of the information: the probabilities and the utilities. But the mathematics which we use to calculate everything else out of those values breaks down, so in fact you have not given all of the information. One important missing piece of information is the ratio of the first positive utility to the second. That and two other independent ratios would be enough information, if they're all finite. (If not, then we might need more information.) And don't tell me that these ratios are undefined; the mathematical model that calculates the ratios from the information given breaks down, that's all. In fact, there is an atlernative mathematical model of decision which deals only in ratios between utilities; if you'd followed that model from the beginning, then you would never have tried to state the actual utilities themselves at all. (For mathematicians: instead of trying to plot these 4 utilities in a 4-dimensional affine space, plot them in a 3-dimensional projective space.) Right; the proper conclusion of the argument is not to believe, but to try to believe. And if you buy the argument, then you should try very hard!

0Unknowns14y

I agree with everything you've said here, including that in the two door situation the decision could go the other way if you had more information about the ratio of the utilities. Still, it seems to me that what I said is right in this way: if you are given no other information except as stated, you should choose the second door, because your best estimate of the ratios in question will be 1-1. But if you have some other evidence regarding the ratios, or if they are otherwise specified in the problem, your argument is correct.

2PeerInfinity14y

Can you please remind me what the question is, that you're looking for an answer to? And can you please provide a link to an explanation of what Eliezer's Lifespan Dilemma is?

1Unknowns14y

http://lesswrong.com/lw/17h/the_lifespan_dilemma/ If you read the article and the comments, you will see that no one really gave an answer. As far as I can see, it absolutely requires either a bounded utility function (which Eliezer would consider scope insensitivity), or it requires accepting an indefinitely small probability of something extremely good (e.g. Pascal's Wager).

4Blueberry14y

If you believe that there is something with arbitrarily high utility, then by definition, you will accept an indefinitely small probability of it. Assume my life has a utility of 10 right now. My preferences are such that there is absolutely nothing I would take a 99% chance of dying for. Then, by definition, there's nothing with a utility of 1000 or more. The problem comes from assuming that there is such a thing when there isn't. I don't see how this is scope insensitivity; it's just how my preferences are. Someone who really had an unbounded utility function would really take as many steps down the Lifespan Dilemma path as Omega allowed. That's really what they'd prefer. Most of us just don't have a utility function like that.

0Unknowns14y

So you wouldn't die to save the world? Or do you mean hypothetically if you had those preferences? I agree with the basic argument, it is the same thing I said. But Eliezer at least does not, since he has asserted a number of times that his utility function is unbounded, and that it allows for arbitrarily high utilities.

1Blueberry14y

If the world is doomed immediately unless I die for it, I have a 100% chance of dying immediately, so I might as well die to save the world. But if it's a choice between living another 50 years and then the world ending, or dying right now and saving the world, and no one would know, I wouldn't die to save the world. I'm too selfish for that. Then he should keep taking Omega's offers, and any discomfort he has with that is faulty intuition, like the discomfort from choosing TORTURE over SPECKS.

1Unknowns14y

I would die right now to prevent the world from ending 50 years from now. It's actually even hard for me to imagine that you're actually as selfish as you say. If the situation actually came up you might find out differently. But I guess it's possible. You might be right that Eliezer should simply accept the Lifespan dilemma as the necessary consequence of his utility function (at least as he defines it.)

2Blueberry14y

Really? Why? I can't imagine myself dying to save the world; it's completely implausible to me and I have a hard time understanding what it would feel like to be willing to do so. But people often die for much less.

2Unknowns14y

Are you married? If so, would you die to save your wife's life? Or if you're not married, what about your mother? Do you find it hard to imagine those things too?

0XiXiDu14y

It's simple. The 'selfish' terminology is just obscuring matters. Just keep your feelings about one thing (your life) and substitute it with something else (someones life). Unknowns utility functions is of a type that it assigns infinitely high utility to saving the world. Not saving the world is simply no option. That's what Unknowns wants. Edit: Forget what I said about Unknowns previously.

1Unknowns14y

Blueberry was the one who introduced the "selfish" terminology. He said, "I wouldn't die to save the world. I'm too selfish for that."

1XiXiDu14y

I'm really sorry. I confused you with someone else I talked to yesterday. My mistake, I edited my comment and will keep more care in future.

0Unknowns14y

Thank you.

0[anonymous]14y

The standard answer is "But what if the Muslims are right?" You can't be both a Christian and a Muslim, and you lose by guessing wrong. We have no more reason to believe we'll be rewarded for believing in God X than we have to believe we'll be punished for believing in God X, as we would be if God Y were the correct one.

0Oscar_Cunningham14y

All this does is show that the dilemma must have a flaw somewhere, but it doesn't explicitly show that flaw. The same problem occurs with finding the flaws in proposed perpeptual motion machines, you know there must be a flaw somewhere, but it's often tricky to find it. I think the flaw in Pascal's wager is allowing "Heaven" to have infinite utility. Unbounded utilities, fine; infinite utilities, no.

0Nisan14y

See The Pascal's Wager Fallacy Fallacy.

2XiXiDu14y

Betting on infinity.

1Oscar_Cunningham14y

That's a great video.

0Unknowns14y

Elliezer in that article: "The original problem with Pascal's Wager is not that the purported payoff is large. This is not where the flaw in the reasoning comes from. That is not the problematic step. The problem with Pascal's original Wager is that the probability is exponentially tiny (in the complexity of the Christian God) and that equally large tiny probabilities offer opposite payoffs for the same action (the Muslim God will damn you for believing in the Christian God). " This is just wishful thinking, as I said in another reply. The probabilities do not balance.

0Unknowns14y

What about "living forever"? According to Eliezer, this has infinite utility. I agree that if you assign it a finite utility, then the lifespan dilemma fails (at some point), and similarly, if you assign "heaven" a finite utility, then Pascal's Wager will fail, if you make the utility of heaven low enough.

0TraderJoe12y

[comment deleted]

3PhilGoetz15y

Can you write a post about satanism? I'd love to know whether there are any actual satanists, and what they believe/do.

5AdeleneDawner15y

I used to know one, and have done a bit of reading about it. It struck me as a reversed-stupidity version of Christianity, though there were a few interesting memes in the literature.

4MatthewB15y

Depending upon the Type of Satanist, yes, they are often just people looking for a high "Boo-Factor" (A term made-up by many of the early followers of a musical Genre called "Deathrock" (it's more public name is now Goth, although that is like comparing a chain saw to a kitchen pealing knife - the "Goths" are the kitchen knife). Many Satanists, especially those who hadn't really read much of the published Satanic literature would just make something up themselves and it was almost always based in Christian motifs and archetypes. The two institutions who have publicly claimed the title of "Satanist" (The Church of Satan and The Temple of Set) both reject any and all of Christian Theology, Motifs, Archetypes, Symbolism and Characters as being ingenuous and twisted archetypes of older more healthy god archetypes (If you read Jung and Joseph Campbell, this is not uncommon for a rising religious paradigm to hijack an older competing paradigm as its bad-guys) As Phil has suggested, maybe a front page post will come in handy. It should be recognized that some Satanists happen to be very rational people. They are just using the symbolism to manipulate their environment (although most of the more mature ones have found more mature symbols with which to manipulate the environment and their peers and subordinates). The types to which I was referring in my post were the Christian Satanists (people who are worshiping the Christian version of Satan), which is just as bad as worshiping the Christian God. Both the Christian God and the Christian Satan are required for that mythology to be complete.

8wedrifid15y

Wow! We make worshipping the devil sound bad around here by comparing him to God! Excuse me if I take a hint of pleasure at the irony. ;)

1MatthewB15y

Well, they both (according to Christian Myth) are truly bad characters. It is unfortunate for God that Satan (Lucifer) had such a reasonable request "Gee, Jehovah, It would certainly be nice if you let us try out that chair every once in a while." Basically, Lucifer's crime was one that is only a crime in a state where the King is seen as having divine authority to rule, and all else is seen as beneath such things (thus reflecting the Divine Order) It was this act upon which Modern Satanists seized to create a new mythology for Satanism, where it was reason rebelling again an order that was corrupt and tyrannical.

7Jack15y

To be fair this stuff isn't Christian mythology in the way that Adam and Eve, or Loaves and Fishes is Christian mythology. It's just religious fiction. ... Unless someone has declared John Milton a prophet and possessor of divine revelation. Which would be hilarious.

2MatthewB15y

It isn't stuff that made it into the modern canon, but in the Early Christian Church, Myth of this type appeared all over the place from the Jewish Sources, in an attempt to integrate it into various Christian Sects. Isn't it ALL just religious fiction?

7wedrifid15y

The key word in Jack's sentence was "just". The concept of 'canonical' is important, to religious believers and Star Trec fans alike.

2wedrifid15y

The Christian Myth includes a quite specific definition of bad so according to the Christian myth only one of them is bad. Is what you mean that according to you the characters as described in the Christian Myth were both truly bad? That description loses something when the ruler is, in fact, God. One of the bad things about claiming that the king is king because God says so is that it is not the case that any god said any such thing. When the ruler is God then yes, God does say so. The objection that remains is "Who gives a @$@# what God says?" I agree with what (I think) you are saying about the implications of claims of authority but don't like the loaded language. It confuses the issue and well, I would say that technically (that counterfactual) God does have the divine authority to rule. It's just that divine authority doesn't count for squat in my book.

5Blueberry15y

There are Christian Satanists? Correct me if I'm wrong, but I thought Satanism was a religion founded around Rand-like rational selfishness, and explicitly denied any supernatural entities.

4MatthewB15y

Yes, they are "Christian" in the sense that all of the mythology and practices for their worship of Satan are derived from Christianity, and they still believe in a Christian God. It is just that these people believe that they are defying and opposing the Christian God (Fighting for the other team). They still believe in this God, just no longer have it as the object of their worship and devotion. This is also the more traditional form of Satanist in our society, and one which the more modern Satanist tends to oppose. The Modern Satanist is a self-worshiping atheist, and as has been pointed out, tend to place everything in the context of self-interest. It is a highly utilitarian philosophy, but often marred in actual practice by ignorant fools who don't seem to understand the difference between just acting like a selfish dick and acting out of self-interest (doing things which improve one's condition in life, not things which worsen one's condition)

1NancyLebovitz15y

There's an Ayn Rand quote I don't have handy to the effect that if the virtues needed for life are considered evil, people are apt to embrace actual evils in response.

1wedrifid15y

Nope, worshipping the devil is right up there as far as meanings for 'Satanism' go.

0MatthewB15y

You mean, like a main page post? I'd love to. You would be surprised about how rational the real Satanists (and their various offshoots and schisms) are (as the non-Christian based Satanist is an athiest). In fact, the very first Schism of the Church of Satan gave birth to the Temple of Set (Founded by the then head of the Army's Psychological Warfare Division), which was described as a "Hyper-Rational Belief System" (Although in reality it still had some rather unfortunately insane beliefs among its constituents). The Founder was very rational though. He even had quite a bit of science behind his position... It's just that his job caused him to be a rather creepy and scary guy.

0PhilGoetz15y

Has today's Satanism retained any connections to Alistair Crowley?

2Document14y

At least they're maintaining lightness.

3blogospheroid15y

The problems of the world the way the way it is right now and the incentives of the people in power as per the current structure does not seem optimal to me. There are so many obvious things that could be done that are not being done right now. for eg. Competition in the space of governments. Proposing solutions to many present problems of the world does not require a superintelligence. Economists do that everyday. But untangling the entire mess of incentives, power and leverage so that these formerly simple, but now complicated, solutions could be implemented requires a superintelligence. This super intelligence needs to be benevolent today and tomorrow. I have not found a better goal structure than CEV which can maintain this benevolence. Singinst has openly written that they are open to better goal systems. If I find something better, I will move my support there.

3PeerInfinity15y

I agree. I'm aware that there are problems with CEV (mainly: we're probably not going to have enough time to figure out how to actually implement it before the Singularity, and the CEV is biased to exclude only the volition of humanity, which means that there may be a risk of the CEV allowing arbitrary amounts of cruelty to entities that don't qualify as "human") Anyway, I'm aware that there are problems with CEV, but I still don't know of any better plan. Because of the extreme difficulty of actually implementing CEV, I am tempted to advocate the backup plan of coding a purely Utilitarian AI, maximizing pleasure and minimizing pain.An orgasmium shockwave is better than a lifeless universe. The idea would be to not release this AI unless it looks like we're running out of time to implement CEV, but if we are running out of time, then we're not likely to get much warning that we're running out of time. And then there's the complication that according to my current belief system (which I'm still very conflicted about) the orgasmium shockwave scenario is actually better than the CEV scenario, since it would result in greater total utility. But I'm nowhere near confident enough about this to actually advocate the plan of deliberately releasing a pure Utilitarian AI. And this plan has its own dangers, like... shudder... what if we get the utility formula wrong? Oh, and one random idea I had to make CEV easier to actually implement: remove the restriction of the CEV not being allowed to simulate sentient minds. Just try to make sure that these sentient minds have at least a minimum standard of living. Or, if that's too hard, and you somehow need to simulate minds that are actually suffering, you could save a backup copy of them, rather than deleting them, and after the CEV has finished applying its emergency first-aid to the human condition, you can reawaken these simulated minds, and give them full rights as citizens. There should be more than enough resources availa

2PhilGoetz15y

CEV is too vague to call a plan. It bothers me that people are dedicating themselves to pursuing a goal that hasn't yet been defined.

1Strange715y

That was part of my motivation for proposing an alternative.

[-]Vladimir_Golovin15y140

My way of asking these questions:

What is the single most important thing you should be doing?

Are you doing it?

(I'm writing the damn help file. Why? Because nobody on the team has the necessary domain experience and writing skills and English knowledge needed for that. Why is the help file important? Because it's the single biggest chunk of work needed for the final release of our software. Very few users read help, but those who do are important. Why release the software? Because a release of a major new version brings in additional revenue and new customers -- and because abandoning a project at 95% completion is (usually) a stupid idea. Why do we need more revenue? To explore a more mainstream, less nerdy business than our current one. Why explore a more mainstream business? I could go on and on and on, but sorry -- time to write the help file.)

2alasarod15y

Nice. It's possible to feel meaning without those questions having a final answer. As in, those whys really can string on indefinitely, but when I'm involved in a task, the meaning can be apparent to me, but not in a way that language captures. I'm not satisfied with the answer that a hidden, higher-order goal or a secondary reinforcer is at work here. I think the action of carrying out a meaningful task has meaning in itself, not something that terminates in a final "because." Is this clumsy of me to say? I honestly don't know what value this community would place on a claim that starts with "language is unable to capture it" - sounds pretty fishy, no? Am I just giving too much credit to what is really a preference?

[-]sketerpot15y120

I'm writing some programs to take some numbers from a typical "new renewable energy plant under construction!!" news article and automatically generate summaries of how much it costs compared to other options, what the expected life span will be, and so on. I intend to go on a rampage across the internet, leaving concise summaries in the comment sections of these news articles, so that people reading the comments will be able to see past the press release bullshit.

Why? Because I believe that people will be more rational if the right thing is obvious. A simple table of numbers and a few sentences of commentary can strip away layers of distortions and politics in a matter of seconds, if you do it right.

Essentially, it's a more elaborate and more automated version of what I did in this comment thread on Reddit: give the perspective that lazy journalists don't, and do the simple arithmetic that most journalists can't do.

It's very simple, but maybe it'll be effective. A lot of people respond well to straight talk, if they don't have a strongly-held position already.

1aausch15y

Does anyone know of studies which measure how much of an effect access to reliable information has on decision making?

3sketerpot15y

I'm not sure how you would even define that well enough to measure it. How do you define "access to reliable information"? Does a large, confusing web site with lots of reliable information constitute "access to reliable information"? What I do know is that the vast majority of "renewable energy" articles are worse than worthless, because they give the average reader the illusion of understanding, while systematically distorting the facts. Case in point: every wind farm announcement I've ever seen has conflated the maximum power output with the average power output. This is off by a factor of 2.5--5, which is similar to saying that I'm 15--30 feet tall. (That's 4.5--9 meters.) Simply pointing this out to people can help a lot, if my experience is anything to go by. This is anecdotal evidence, I know, but it should work and it looks like it does work.

[-]thomblake15y120

What are you doing?

Are our answers confined to 140 characters or less?

0CannibalSmith15y

Yes.

[-]Morendil15y110

You forgot one high-leverage component in this kind of inquiry: ask the question "why" not one but five times or more.

Just before reading the above, I was looking at instructions for building a laser show from cheap parts and the Arduino microcontroller I've been playing with lately.

Why - because I'm getting an interest lately in programming that affects the physical world.

Why - because, in turn, I believe that will broaden my horizon as a programmer.

Why - because I think learning about programming is one of the more important things anyone interested in the origins and improvement of thinking can learn. (Post about this coming sometime in the next few weeks.)

Why - because I want to improve my thinking in general. Which is also the reason I stopped here after I figured I had collected enough information about the laser stuff.

Why - because my thinking is my highest leverage tool in dealing with the world.

(ETA: to be quite honest, another reason is "because it's fun", but that tends to apply to a lot of the things I do.)

-2[anonymous]15y

I would (presumptuously) bet that that was a rather more important causal factor than the higher-level goals you listed at first. Not that there's anything wrong with that (no, seriously).

[-]Roko15y100

Reading PJ Eby's book chapters on positive vs. negative motivations

because

I think that now is a good time in my life to read up on the skills of real life, of motivation and of success.

[-]BenAlbahari15y90

Actions speak louder than words. A thousand "I love you"s doesn't equal one "I do". Perhaps our most important beliefs are expressed by what we do, not what we say. Daniel Dennett's Intentional Stance theory uses an action-oriented definition of belief:

Here is how it works: first you decide to treat the object whose behavior is to be predicted as a rational agent; then you figure out what beliefs that agent ought to have, given its place in the world and its purpose. Then you figure out what desires it ought to have, on the same consi

... (read more)

[-]Johnicholas15y90

3 priorities, in no particular order: support myself, become more capable, enhance rationality by publishing "seed exoshell" software.

An exoshell, as I understand it, is the software that you think with, in much the same way that you think with a piece of paper or a whiteboard. Current exoshell-ish software might include emacs ("emacs as operating system") or unix shell scripting ("go away or I will replace you with a small shell script"). Piotr Wozniak clearly uses SuperMemo as an exoshell. Mark Hurst's "Bit Literacy&quo... (read more)

4XFrequentist15y

To a fledgling computer geek, this sounds absolutely awesome, and I would love some elaboration!

0Johnicholas15y

Well... the idea is that the tiniest exoshell would simply be one that continuously verifies / trains the user to make changes to the exoshell. So I took a standard design for a quine, and modified it so it injects a random error into the source code that it spits out. I call it "ExoMustard" Source is at: http://www.johnicholas.com/exomustard.c So the idea is if you took this, and repeatedly fixed it, then ran it, then fixed it, then ran it, et cetera, you would soon be comfortable adding other features. Maybe it acts as a little to-do list maintainer program, as well as its previous features. Maybe it also acts as a compiler or a shell or a virtual machine emulator - or stores recipes for how to download and install the compiler, shell, and virtual machine that you like to use. I've done about 10 iterations from that starting point, combining it with SQLite and adding a self-test framework and so on, but I haven't gotten to the point of using it routinely for interacting with the world. Recently, I've started studying tiny self-compilers (Fabrice Bellard's otcc, and Edward Grimley-Evans's bcompiler and cc500). Maybe an ideal seed exoshell would have the functionality of Lennart Augustsson's 1996 ioccc entry, the readability of Darius Bacon's Halp and the host-independence of Rob Landley's Firmware Linux..

2gwillen15y

Has there been a 'what is your exoshell' thread on LW yet? Would it be appropriate to have one? In the purest definition, mine is pretty small (a thousand lines of Perl or so), but if you include 'software you communicate with', which I think I do, it grows rather large, to include most of what's running on my server.

1PeerInfinity15y

My exoshell used to be plain text files, then it was MediaWiki, now it's Google Wave. Not just the raw text itself, but scripts to extract XML tags and other data from the text files, and do stuff with the data. That was a relatively straightforward process with raw text files. It was a bit more complicated with MediaWiki, but that seemed to work even better. Google Wave has the advantage of collaborative editing in realtime, and some advanced search features, but it has lots of serious disadvantages, (including there currently being no way to export to XML!) but hopefully this limitation will soon be overcome. For now I'm using the Ferry extension to export to Google Documents, and from there I batch-export to html. Oh, and I recently started some experiments with scripts to extract tags from this data and make some fancy quantifiedself graphs. If anyone is interested in hearing more about this, please let me know.

2aausch15y

I think you are the first person I know of, who actively uses Google Wave.

0PeerInfinity15y

Adelene Dawner and some of my other friends use Google Wave too, but mostly to read stuff I wrote, and to comment on it, or chat live about it. Oh, and another friend, MetaFire Horsley, aka MetaHorse, uses Google Wave to write some awesome sci-fi stories, and get live feedback on them. And that's working pretty good. I might as well ask: does anyone else here use Google Wave? Or does anyone here have a Wave account that they're not using because they don't know anyone else who uses Wave? oh, and here's some more information about what I'm using Wave for.

2mattnewport15y

I've read a few attempted descriptions of what Google Wave is and have not really been able to make sense of it or understand how it might be useful to me. Several of these descriptions have admitted to difficulty expressing either its function or purpose clearly as well. I haven't been motivated to try to understand it further because I'm not aware of any problem I have which it appears to be a solution to.

0PeerInfinity15y

The main useful feature of Wave is the realtime collaborative editing, and the ability to be instantly alerted to any update to any wave you're monitoring. There's more, but there's probably not much point for me to list all of the other features here. And I'm reluctant to try to convince other people to join, because it can be extremely addictive, and it's still kinda annoyingly glitchy, and is still missing some important features. If you're not the sort of person who tries new things just for the sake of trying them, or if you didn't get immediately excited about Wave when you first heard about it, or you don't think you have any use for realtime collaborative editing, then you're probably better off waiting until someone you know is using Wave for something specific that you want to join in on. and yes, it can be used as a persistent, HTML form of IRC, where you can leave or resume a conversation at any time, or return to an old branch of the conversation and visually branch it off, or even have multiple branches running simultaneously, in separate parts of the wave, rather than the different threads constantly overlapping, which always ends up happening in IRC.

0gwern15y

As far as I can tell, it's a HTML form of IRC, but persistent.

[-]Scott Alexander15y70

I'm doing exactly what I would be doing if I had never found Less Wrong, but now I'm telling myself this is provably the best course because it will make me a lot of money which I can donate to the usual worthy projects. This argument raises enough red flags that I'm well aware of how silly it sounds, but I can't find any particular flaws in the logic.

5Roko15y

which is? I am intrigued as to what the eminent Yvain does in RL ...

[-][anonymous]15y60

What am I doing?: I am studying to be an electrical engineer at SPSU.

Why am I doing it?: Because I want to make a lot of money to pay for Friendly AI, anti-aging, and existential risk research, while following my own interest in power and similar technology.

Of course, these are my LessWrong what and whys, and not necessarily the other what and whys of other sectors of my life. If you took a look at the percentage of my time being devoted towards these activities... lets just say my academic transcript would not flatter me well....

(PS: Asking what and why is a good way to get far-mode about what you're doing, which increases motivation. So thanks for getting me kick-started on my studying ^_^)

[This comment is no longer endorsed by its author]Reply

[-]Kevin15y100

You should go and donate $10 to the SIAI if you haven't, because people that donate any amount of money are much more likely to later donate more amounts of money. Anti-akrasia, etc. etc.

1timtyler15y

These seem like the most relevant links to the associated cognitive biases: http://en.wikipedia.org/wiki/Sunk_costs http://en.wikipedia.org/wiki/Psychology_of_previous_investment

0aoxfordca15y

SPSU? I would express lesswrong folk to be at GaTech ;). Glad to know there are others in the neighborhood (You are an uber-slingshot projectile away)

0JGWeissman15y

Have you considered participating in SIAI's Visiting Fellow Program? It is a good opportunity to learn what is being done for existential risk reduction.

0[anonymous]15y

Oh, you bet I have! But I have a few more responsibilities to deal with here before I can join in the effort in California... but it is my goal to start helping you guys with the LessWrong wiki from my residence here at the very least, and I've always wanted to start a rationalist youtube video playlist! Oh procrastination...

5Kevin15y

At the SIAI house, surfing Less Wrong doesn't count as procrastination! Except when it does.

[-]james_edwards15y40

What am I doing? Trying to write a few thousand words on legal reasoning about people's dispositions.

Why? To finish my dissertation, and graduate with an Honours degree in law.

Why am I doing that? To increase my status and career opportunities, but also due to inertia. I've almost finished this degree, and a few more weeks of work for this result seem worthwhile. Also, doing otherwise would make me look weird and potentially cut off valuable opportunities.

Why does that matter? Much intellectually interesting and well paid work seems to require signalling a... (read more)

[-]magfrump15y40

What am I doing?: Bug testing a program for my thesis. (A mathematical computation.)

Why?: Because I want it to work perfectly (and it doesn't).

Why?: Because I want to impress my advisor.

Why?: two reasons: a) to have a good working relationship since I will be at the same school for a while. (why?: because it's unpleasant to work with people you don't get along with; terminal value.)

b) to get better recommendations.

Why?: to have better career prospects.

Why?: to make more money.

Why?: a) to make my life more pleasant (terminal value) b) to donate to reducing... (read more)

1magfrump15y

Other questions: why am I studying math? Why am I staying in school? Because it is interesting; because if I don't do something interesting I will be (a) less happy, (b) less effective.

[-]roland15y40

I think these two questions are the basic questions of rationality that you should be asking yourself as much as possible. There is this great quote that I have on my desktop:

Only the real determinants of our beliefs can ever influence our real-world accuracy, only the real determinants of our actions can influence our effectiveness in achieving our goals. -- Eliezer Yudkowsky

[-]olimay15y30

Background: two years ago, I dropped out of college with a tremendous amount of debt. I'd failed several classes right before I dropped out, and generally made a big mess of things.

Still alive today, I'm beginning to step free of a lot of social conventions, letting go of shame and the habit of groveling, and learning to really value (and not just know I should value) important things. I am searching for how to make my strongest contribution. In the short term, that probably has to do with making a lot of money, but on the side, I have an inkling that work... (read more)

[-]Will_Newsome15y30

What am I doing? Working for SIAI. For the last hour or so I've been making a mindmap of the effects of 'weird cosmology' on strategies to reduce existential risk: whether or not the simulation hypothesis changes how we should be thinking about the probability of an existential win (conditional on the probability (insofar as probability is a coherent concept here) that something like all possible mathematical/computable structures exist); whether or not we should look more closely at possible inflationary-magnetic-monopole-infinite-universe-creation-horror... (read more)

0Metafire15y

That's very interesting. It sounds like you start digging into the problems with level 4 multiverse ethics I plan to write a series of sci-fi novels about. What I have already written (not too much) can be read with Google Wave if you add "our-ascent-noofactory@googlegroups.com" to you contacts and then display the group waves. There are a couple of underlying concepts and questions which come together in my fictional work: 1. How could a possible positive post-singularity world look like? 2. How could sci-fi in a possible post-singularity world look like? -> My answer to that question is that "post-singularity sci-fi" doesn't make sense and the corresponding works of fictions would be about alternative possible worlds (which exist by modal realism or "classical" multiverse theories) and their hypothetical interactions. 3. How does modal realism multiverse ethics work, if at all? 4. How to portrait a civilization whichs ascends rather infinitely in the most impressive and interesting way? 5. How can a positive post-singularity world be reached after all? Why do I care about the hypothetical consequences of modal realism? Because I think it's the best ontology for modelling the world as correctly as possible, and I'm pretty convinced that it's true (for reasons similar to those in the map that is the territory, but more founded in mathematical logic and philosophy of mathematics). Trying to apply "pure" utilitarianist reasoning to a modal realism multiverse leads to serious problems, for example: A) The amount of joy and suffering are actually infinite, which destroys the point of summing it up or integrating it. You could fix the problem by doing "local" computations, but then how do you define the "locations" in the right way, if worlds are nested or infinite in space or time. All of this is a huge headache for a convinced hedonistic utilitarianist, which I used to be before. (I find it hard to tell what I am, in an ethical sense, now. Possibly, "confused"

1PeerInfinity15y

I should mention that Metafire and I have spent lots of time discussing these concepts, and I am familiar with the ideas he is presenting in this comment. The metaphysics that Metafire is trying to describe here is pretty much identical to what ata described in the post on The mathematical universe: the map that is the territory. Oh, and please excuse the confusing grammar, English isn't Metafire's native language. Metafire lives in Germany. Metafire, please check today's post in my waveblog for some suggestions on how to make the grammar of that comment less confusing. I know I've gotten negative feedback when I've tried to do polls before, but... did anyone actually understand what Metafire was saying in this comment? Even I didn't understand that last paragraph. It will need lots more explanation. Oh, and Metafire and I disagree on the implications of this metaphysics. I believe that this metaphysics doesn't invalidate total hedonic utilitarianism, but Metafire thinks it does, afaik. Oh, and I also disagree with ata about some of the moral conclusions drawn from that post, though these disagreements are probably a matter of intuition and interpretation. Though I suspect that ata did something similar to that mathematical proof that tries to prove that 1=0, by misusing the concept of infinity.

1Larks14y

I understood it, though if you're in regular contact, maybe I only think I do. Essentially, the belief that to be definable is to exist leads to moral nihilism, much as some thing Many-Worlds does ( no matter what you do, it'll still be undone in other worlds. Each 'good' world as an equal and opposite 'bad' world). The second half I understand the sentences of, but don't think Metafire has provided enough details here. Certainly, I don't see how some kind of weighted function over all ethical systems or volitions could help expand their scope.

[-]Metafire15y30

What I am doing:

I registered here on LW today, because this is the first posting I thought I really should comment on. Those two questions are one of the most important and most thought provoking of all. They are even more important than the questions "What do you believe, and why do you believe it?", because most of your believes might not be relevant for your actions at all.

1. I study mathematics and physics, because
1.1. I want to know what the world is and how it works.
1.1.1. Understanding the world better is generally useful, although

... (read more)

0PeerInfinity15y

Is it too late to modify your curriculum? It sounds like you would be much better off leaving yourself enough time to study math and to write your stories.

1Metafire15y

Actually my curriculum is pretty flexible. The real issue isn't time, but setting clear priorities. I'm not really good at that. I find it pretty difficult to make myself write a story, compared to almost everything else. Also, I could need a better spoon-management system.

0PeerInfinity15y

Well, you just finished setting some clear priorities in that last comment, didn't you? It sounds like you would be better off leaving yourself enough free time or spoons or whatever to continue writing your stories whenever you feel inspired to do so. But I would advise against setting a schedule where you're forced to write your stories even if you don't feel inspired. That just seems like a bad idea. Anyway, that's just my opinion, I'm no expert on these matters. Oh, and I guess I should admit that my advice may be biased by the fact that I like your stories, and would like to read more of them. Heh, I was about to say that I still think that utilitarianism is the optimal ethical system, but I just realized that utilitarianism is about what you value, not about how to ethically go about achieving those values. There's a rather extreme difference between a utilitronium shockwave scenario without any ethical restrictions, and a version of the utilitronium shockwave scenario with, for example, ethical restrictions against murder and coersion. And, um... who downvoted Metafire's last comment, and why?

[-]alasarod15y30

I'm sorry to do this because I'm sure it's off topic, but Tim Minchin (comedian) just did a 10 minute piece that will make skeptic that's had to sit through exchanges about auras, and magic, and how science is "just a theory too," just holler.

http://www.youtube.com/watch?v=V0W7Jbc_Vhw

Isn't this enough? Just this world?

5Alicorn15y

This would make a good Open Thread comment. Also, "Storm" isn't especially new; you had me excited for a moment that Minchin had "just" done something.

7alasarod15y

You're right on both accounts. I admit I'm new to commenting on LW. It's intimidating but I've decided to learn from practice rather than observation. Thanks for the input!

[-]Alicorn15y20

I think this was created independently but wow.

2nhamann15y

For those of you as confused as I was, the above post should actually link here.

1Alicorn15y

Whoops, thanks.

[-]JoshB15y20

Listening to Autechre's new album

Because it contains sufficient audial texture and sophisticated sound modulation combined with an intermittent hip-hop beat, so as to sit nicely within my current, self-imposed tolerances of what good electronic music should sound like....which results in a state of favorable brain chemistry.

[-][anonymous]15y10

I do not know if there is any causal connection here, but this piece of art is precisely relevant:

http://www.johnnywander.com/comics/173

1Alicorn15y

Hey.

1abramdemski15y

Apologies!

[-]Matt_Simpson15y10

What am I doing?

Finally responding to this post on LessWrong.

Why am I doing it?

I don't quite feel tired yet, and I don't know which book to pick up for pre-sleep reading: Wicked, so I have some context when I see the musical with my girlfriend in June. The Ancestor's Tale because I find evolution extremely interesting, and there's the off chance that it will be relevant to my future research (and I'm obligated to read it since it was a gift). Or The Theory of Moral Sentiments because I find the moral sense theorist to be interesting precursors to Eliezer... (read more)

1Matt_Simpson15y

I think I'm going with Wicked

[-]anonym15y10

A bit off topic, but you've got me thinking about Babylon 5, so have a few more question:

Who are you? (The Vorlon Question)
What do you want? (The Shadow Question)
Why are you here? & Do you have anything worth living for? (Lorien's Questions)
Where are you going? (Techno Mage's Question)

2Alicorn15y

Ahem.

1anonym15y

Ha, I'd totally forgotten about that excellent post of yours, which I did actually read at the time. Thanks for the reminder.

0MartinB15y

It continues with 'who do you serve and who do you trust'. http://www.youtube.com/watch?v=RWKNGNh1I-4 Now B5 is an amazingly well done show, but its not particularly singularitarian. (http://en.wikipedia.org/wiki/Deathwalker)

[-]thatrenfrewguy15y10

I was, before reading this, reading the Selfish Gene by Dawkins Why? Because it is well written Why? Because it relates to the topic I chose for my IB EE Why? Because I enjoy learning about alternative descriptions of functional units of evolution and of organisms, and Dawkins treats genes like the puppet master of evolution. Why? Because I am trying to bridge a cultural gap with my father (reasons for that are for somewhere else) Why? Because it is refreshingly different from the magic realism I had been reading.

[-]MartinB15y10

That looks to me like applied rationality.

5RobinZ15y

Yes – and rationality must be applied, or else it collapses into sophistry. That was the essential idea behind Something to Protect, for example. Secondarily, asking this question may prevent a subset of disputes about definitions.

3MartinB15y

Fully agreed! I still struggle with applying what I learn, or even having it available at the right time. But i make progress. How does the question prevent disputes about definitions? I fail to see that.

2RobinZ15y

If Fred says, "The test scores of the group trained by method A is greater than that by method B at a 98% significance level, and therefore method A should be preferred", and Sheila says, "The Bayes factor between hypothesis M1, which assumes that method A and method B produce a similar distribution of test results, and M2, which predicts superior results from A, is 1:38, suggesting that method A is superior" ... they don't actually disagree. Both Fred and Sheila would recommend training by method A. It's not a traditional dispute about definitions, but (for example) Sheila sniping at Fred for using frequentist methods would be inappropriate. If he genuinely deserves criticism, she will not need to wait long for an occasion where he is wrong.

[-]Sperling15y00

"Define And thus expunge The ought The should ... Truth's to be sought In Does and Doesn't "

-B. F. Skinner (an interesting soundbite from an otherwise misguided disagreement with Chomsky over language acquisition)

[-][anonymous]15y00

[quote]Define And thus expunge The ought The should

Truth's to be sought In Does and Doesn't [/quote]

-B. F. Skinner (an interesting soundbite from an otherwise misguided disagreement with Chomsky over language acquisition)

[-][anonymous]15y00

[quote]Define And thus expunge The ought The should

Truth's to be sought In Does and Doesn't [/quote]

-B. F. Skinner (an interesting soundbite from an otherwise misguided disagreement with Chomsky over language acquisition)

[-]XiXiDu15y00

I'm doing my best.

"Narns, Humans, Centauri… we all do what we do for the same reason: because it seems like a good idea at the time." -- G’Kar, Babylon 5

[-]Thomas15y00

I do this: http://critticall.com/SQU_cir.html

In fact, the machine on my left does it, I do something else.

[-]byrnema15y00

I think that what you do (and why you do it) follow your beliefs, and that's why interrogating beliefs is the more fundamental question.

For example, you might do 'X' because you believe 'X' matters, or, more meta -- and more fundamental -- you might believe that whether you do 'X' or not matters because you believe that what you do matters. This is only true within a particular belief structure.

3JRMayne15y

I think what you do and why you do it generates beliefs and actions more than people think. One particular example is high-powered law students joining big firms for just a little while until they end up doing [thing they like/thing they think is good for society]. Walking away from buckets of money once the buckets are coming is very, very hard and few can do it. At that point, rationalization sets in. The valuation of money increases, because that becomes a self-worth measurement. As has been pointed out on LW, people do things that they want to do and then make up reasons in their head why that's good. (Being a jerk educates the other guy/The government's just going to waste the money if I pay the proper amount of taxes/The government uses money very efficiently, but I shouldn't pay extra because that would defeat the system/If he didn't want his money taken, he shouldn't have been so stupid as to trust me/I cheat because I should win, and only bad luck causes me not to win, so cheating brings a more just result.) This also applies to jobs. People find reasons to value/overvalue their jobs because they've landed there. Part of this may be pre-existing belief, but this gets cemented in. I think people's actions and jobs end up morphing beliefs - which is one reason why examining actions is important.

1Oscar_Cunningham15y

The problem being that we often find ourselves doing things for reasons other than the ones we think we do. Robin Hanson will tell you that.

0byrnema15y

Why is this a problem? (Along the lines of, why do you need to accurately know the reasons why you do things?) I'm trying to relate. I see beliefs as something I need in order to decide what to do. As long as I'm doing what I decide to do, why would I worry about varied reasons for doing it?

2roundsquare15y

One reason that comes to mind is that you might be avoiding something you should be doing.

[-][anonymous]15y00

What are you doing?

Voting this post down.

And why are you doing it?

Because it contributes nothing substantive to the site; we're aware of the problem of grounding behavior, and this doesn't help solve it.

[-]SilasBarta15y00

What are you doing?

Voting this post down.

And why are you doing it?

Because it contributes nothing substantive to the site; we're aware of the problem of grounding actions, and this doesn't help solve it.

ETA: On reflection, I made this point way too harshly, in an attempt to be cute. Sorry about that.

ETA2: Someone seems to be modding down everyone who's replied to this. Just so you know, it's not me. I don't vote on comments in arguments I'm directly involved in, and I've made a big deal about adhering to this in the past.

3wedrifid15y

I think these questions are important. In fact, I have built the habit of asking myself the question "What do I want?", which prompts thinking along similar lines. (I did actually vote the post down, but just because it presents something as "fundamental to rationality" when it just isn't. It is rather a useful tool of applied rationality. What would have been particularly interesting is if MBlume presented some insights into what he has gained from this kind of introspection, including any changes he has made to "what he is doing" based on the "why he is doing it" sucking.

4SilasBarta15y

I don't think our positions are very different on this. Like I said to Morendil and JGW, the fact that this is a good question for discussion makes it belong on the site -- but in Open Thread. In its top-level form, it needs to do a more thorough handling of the topic.

0wedrifid15y

Yes, I was expressing a similar position, so included it here to reduce clutter. I wonder why the grandparent was downvoted. It wasn't particularly controversial position. Presumably either because it was in reply to your comment so someone voted systematically, assuming it was a fundamental disagreement (other replies to your comment were downvoted at some stage) or because of a parenthetical confession that I too downvoted the post (although I'd expect more umbrage to be taken at your burn!)

3JGWeissman15y

I see a lot of value in the post, which can help a person to expose a disconnect between their normative ideal of what they should do and what they actually are doing. A large part of the point is that this rationality stuff is not just theoretical, it has practical implications, and we should remember to apply the practical lessons in our daily lives. The short, easy to remember "What are you doing? And why are you doing it?" is great for prompting oneself to examine how well they are practically applying rationality, how effective they are at achieving their goals.

7SilasBarta15y

Perhaps, but at its current state, it still seems more appropriate for open thread. For a top-level comment, I would expect to see more detail, take us through examples, and generally provide a more thorough exposition of how to best go about the process.

2Oscar_Cunningham15y

Agreed, nothing is lost by posting something in the open thread first, and then posting an expanded version if it generates interest. Personally, I'd like to see the idea expanded.

-1Morendil15y

I agree that the post has little substantive content, but it may generate some interesting discussion even while sitting at 0. You could fault it for timing - the flood of former lurkers who saying Hi, sometimes with interesting commentary, is already too much to keep up with - but I'd rather downvote an overly long post than a short one. Less of a strain on my attention economy.

3SilasBarta15y

Perhaps, but that would better justify making it an open thread comment.

[-]nhamann15y00

Nitpicking, but this "doing" question can't possibly be of equal importance to the fundamental question of rationality, because answering the "Why are you doing it?" part obviously depends on you having come to terms with what you believe, and why you believe it.

That said, I think this "doing" question is fundamental as well, second in importance only to the Fundamental Question. Good post.

0khafra15y

The "Why" in "why are you doing it" could be interpreted as "for what purpose," or "as a result of what causal chain." Neither of these, at first blush, appears all that fundamental or difficult--but perhaps there's another sense I'm missing.

3nhamann15y

Certainly. What's important, however, is that the process of repeating the "why?" question forces you to to 1) think about what it is that you're doing, in detail, 2) understand what ends these actions serve, and 3) confront the beliefs that make these ends seem desirable in the first place. In effect, asking "what are you doing, and why are you doing it?" forces you to look at not only what you believe, but whether or not your actions are in alignment with your beliefs. For example: What are you doing? I am studying information theory, Bayesian statistics and neuroscience. and why are you doing that? I am trying to understand how the brain works, and it appears that the former two areas of mathematics are useful tools in formulating theories about the brain. (Notice I've already had to confront a belief here. A "why do you believe this?" question should go here.) and why are you doing that? 1) It is a very interesting problem. 2) Ultimately, having a good theory of the brain will likely contribute to both AI and WBE technologies (belief!), both of which I view as necessary to confront issues that will likely arise in the world as the world population increases and as more and more dangerous technologies get developed (synthetic biology, nanotechnology, etc.) (a tangled network of beliefs, here, all of which need explaining). If I were to unravel this further, I would have to confront the fact that AI is itself a dangerous technology, so I should address whether my current course of action results in a net positive or net negative impact on the chances of a beneficial Singularity (there's a belief implicit here: that the Singularity is plausible enough to warrant thinking about. This too requires explanation). Of course, this process quickly gets messy, but in my view "what are you doing and why are you doing it" is of fundamental importance to any rationalist.