LESSWRONG
is fundraising!
LW
$

Comment Permalink

I don't know about the rest of the audience, but I'd really appreciate a worldbuilding writeup, or maybe even just a glossary, explaining the cultural/technological backdrop of 3WC in more detail than the story provides.

Eliezer Yudkowsky16y30

There are some worlds for which I have devised huge cultural, technological, and historical backdrops but this is not one of them.

See in context

7 Open Thread: June 2009

by Cyan

1st Jun 2009

1 min read

145

7

I provide our monthly place to discuss Less Wrong topics that have not appeared in recent posts. Work your brain and gain prestige by doing so in E-prime (or not, as you please).

Open Threads

Personal Blog

7

Open Thread: June 2009

New Comment

145 comments, sorted by

top scoring

Click to highlight new comments since: Today at 11:27 AM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

[-]nazgulnarsil16y150

One of the things I worry about the LW community is that we'll get too enamored of signaling rationality and become afraid to bounce ideas off each other that we're not quite sure are rigorous enough. This thought is brought on by the seemingly small number of top level posts that people try to make (given how many members we have). Are people refraining from posting after they see that one such thread gets voted down? I know how discouraging that can be, and it seems to me that it would be worse for the budding rationalist who is unsure of themselves. To some extent these monthly open threads help, but even in these comments seem pretty conservative. What say you LW? Are we capable of entertaining ideas without adopting them or is the risk of diluting the signal to noise ratio too high?

edit: it occurs to me that maybe I'd like to see another tab up top called "passing thoughts" or some such. The S:N ratio of the regular posts could be kept up and shorter posts could go there. We could all agree simply to not hold it against each other if we make a gaffe in these shorter posts (obvious mistake). I think this would be valuable because even if an idea is flawed it can generate great discussion (one of the reasons I enjoy hacker news). As an added bonus, this could serve as a space for "Ask LW" posts without disrupting anything.

9jimrandomh16y

Having posted five top-level posts so far and having two half-written ones pending, I'd estimate that writing a good post takes me about 20 times as much time and effort as writing a long comment. Many people simply can't commit that much time, or can't maintain focus for that long. I don't think fear of rejection is the problem.

1JamesCole16y

I take nazgulnarsil's comment as suggesting that there may be value to more people writing posts that aren't necessarily "good"... in which case that sort of rejection may not be optimal.

3Alicorn16y

I like the "passing thoughts" segment idea. "Ask LW" sounds great too - a community advice column without the Yahoo! Answers nonsensical noise could be a great draw for new contributors, too.

[-]Matt_Simpson16y90

Apropos my comment here, what do you guys think of making The Simple Math of Everything a reality?

The LW community probably has a diverse enough group of scholars to cover most of the major fields, and if not, we all have contacts. Splitting it up into sections for different individuals to write would make the project much easier to complete, provided someone is coordinating everything. What do you guys think?

4Vladimir_Nesov16y

Write a call for top-level-post essays as a top-level post, stating clearly what kind of articles you see in such a collection, and see if enough accumulates for an edited volume (I think it's unrealistic at this point).

3Tom_Talbot16y

I have always liked this idea. Could it be done on the Less Wrong wiki?

1[anonymous]16y

Instead of writing a collection of articles, could we just put together a collection of links? It couldn't be sold, but it would be ready a lot sooner.

3XFrequentist16y

Something like this?

0Matt_Simpson16y

I like this idea, actually. I'm sure there's already a lot of great material out there, we just need to put it together.

1XFrequentist16y

Agreed as well, I always thought this was a great idea. An edited volume might be overly ambitious, but I think a wikibook could be a feasible. Another way to test the waters is to write a top level post calling for suggestions on topics or format. If there's enough interest, put out a call for essays accompanied by a brief style guide. We'll have to make sure to ask Kalid Azad to participate.

[-]dclayh16y80

Speaking of improving the LW website, I'd like to see little homepages for each user to provide a self-introduction, which I know has been suggested before, but in particular I'd like to see an implementation of userboxes as Wikipedia does (see my homepage there). This would allow people a standardized and easy-to-reference way to show their opinions on various common issues on the site.

[-]gwern16y80

Here's an open topic: why did Overcoming Bias and Less Wrong split?

It certainly looks to me like the technical shift was just an excuse for certain parts of that community to split away.

If this social explanation is true, what implications does this have?

4SoullessAutomaton16y

It's worth noting that LW has a lower barrier to participation and a more community-centric focus, while OB has shifted to be explicitly Hanson's blog; these are clearly pursuing distinctly different goals. The simplest hypothesis is that Hanson and Yudkowsky disagreed on direction and decided to part ways. In the About page on OB, Hanson says: The phrasing here (especially the strongly complimentary reference to Eliezer) is a sort that, in my experience, often indicates a professional disagreement after which collaboration is ended, but with no real hard feelings over it and a desire to signal such to third parties. This suggests to me a lack of ulterior motives. It does seem to be the case that the readers and commenters of LW and OB have some amount of non-overlap, but I suspect that is mostly incidental.

4AndyWood16y

I have the strong impression that Eliezer and Robin were better together. I've not yet felt that either individual blog has the same draw as the old OB.

1gwern16y

And why couldn't that goal have been pursued within a LW-powered OB? It's not as if there is all that much competing content, and surely we can expect further technical additions like tags. Entirely separate sites, with all the implied overhead and loss of network effects, to pursue separate goals only makes sense if the goals are contradictory.

3Eliezer Yudkowsky16y

Robin said he was apprehensive about using new software to power his blog. It's almost as if Robin had a bad experience with over-ambitious hypertext software, at some point in his murky past...

1Z_M_Davis16y

Was that a Xanadu reference?

1gwern16y

It was; well done.

[-]QuestionTime16y80

I need relationship advice and I trust the wisdom and honesty of this community more than most of my friends. I created a new account to ask this question.

I'm with an incredibly compassionate, creative woman. She excels at her job, which is a "helping profession," and one which I believe improves social welfare far more than most. The sex is outstanding.

But she loves magical thinking, she is somewhat averse to expected-utility calculations, my atheism, etc. She is, by her own admission, subject to strong swings of emotion and at greater tha... (read more)

[-]Cyan16y100

How should I go about deciding whether to continue this?

With science!

Specifically, the science of John Gottman. Short version: irreconcilable differences of viewpoint are not an intrinsic bar to a long-lasting relationship. The most potent relationship poison is contempt.

0jimmy16y

I was pretty amazed the first time I saw this, and even though I'm pretty confident in my relationship, it seems like this test would still be worth quite a bit. Does anybody still run these microexpression tests or would you have to convince the researchers to get back into it for a one off thing? When I looked, I couldn't find anyone offering to run this test. I hadn't gone far enough as to contact the original researchers and see if they'd be willing to do it, but if there are other people interested, it might be worth a shot.

0Cyan16y

I assume you're talking about the test where the researchers infer the likely fate of the marriage by tracking the "trajectory" of a 20-minute-or-so conversation. My impression is that proper interpretation of the test requires training. Gottman has written popular relationship advice guides based on his research; I'd recommend that anyone interested in maintaining the health of their current relationship have a look. His advice makes the assumption that certain correlations he's observed are causative, but that assumption seems reasonable for the most part. Research into the actual causal effect of his suggested interventions is ongoing.

0jimmy16y

Yes, that is what I was talking about. Proper interpretation may require training, but we know those people exist, at least.

0hrishimittal16y

Thanks for the link Cyan.

7Alicorn16y

It is possible to make a relationship work in which each party has a role in the general neighborhood of "the rational one" or "the emotional one", as long as the relative places and merits of these roles are acknowledged by both parties. Since you say she's prepared to admit to her mood swings, this may be doable. My proposed checklist: * If she suffers from an extended period of depression, is she prepared to address that (therapy, antidepressants, ice cream & sad movies, whatever she finds works for her), or would she let it greatly interfere with her life and your lives together? * How averse to your atheism is she? How averse to her (presumable) theism are you? Do you have enough else to talk about and enough ability to skirt the topic that it can avoid being a major point of contention between you? If you want kids eventually, can you come to an agreement about how to raise them re: religion? * Does her magical thinking lead her to do anything profoundly instrumentally stupid, or does it mostly just make her sound a little silly occasionally? * Can you respect her, as well as love her, in spite of her failures of rationality? Or would you be hoping in the back of your mind forever and always that she'd eventually wise up and be a more rational version of herself that you could respect? * Are you compatible on other long-term axes? (Financially, politically, life goals, desire to live in a particular location, opinions on family and homemaking, etc.) If you can give the "right" answer to all of those questions (I think it should be obvious in each case which answer would be best) then go for it and the best of luck to you. If you can't, you either need to address the situation and fix it, or move on.

0QuestionTime16y

Alicorn, Thanks for responding - see my "Added" to the original comment. She seems to take a fairly reasonable approach to dealing with / working around her emotional issues and tries hard not to let me suffer because of them. The atheism / theism divide could be much worse. I'm not sure her beliefs even have net-negative consequences. At present, the main issue is that we each have important beliefs that we don't think we can share. RE: children, we could probably both accept me having the right to be honest about my beliefs but not pushing them, or going into detail unless they really want to hear it or reached 18. She doesn't generally do things that are obviously and profoundly instrumentally stupid, but its probably fair to worry about whether she might in the future. She'd need some money to spend on her friends more lavishly than I would, and to give to inefficient charities, but it seems unlikely she'd want to spend more than I could indulge.

2Alicorn16y

Being able to share differing beliefs has more to do with whether you can both remain civil about important things than whether you agree. I regularly and enthusiastically pick apart minute disagreements between myself and my friends, and would feel as though something were lacking if I couldn't - but we can switch topics from politics to polenta when someone gets fed up and there are no hard feelings. If you can't do that with your girlfriend, that indicates a deeper-running incompatibility than merely disagreeing on rationality. Even if you agreed on all the big issues, it would be miraculous for you to make it through life without ever arguing, and being able to argue without it having it destroy your relationship is an essential skill.

0QuestionTime16y

A big part of the issue is that I'm not sure whether in depth discussions of my views will a) convince her, or b) help her live a good and happy life, or c) the relationship between a) and b). Regardless, I'll need to push a little more conversation of LWish topics before doing anything crazy like getting married. She realizes this as well.

0QuestionTime16y

Let me explain that sentence a bit more. As you know, preference utilitarianism comes with quite a bit of bullet-swallowing and while I may be less hard core than some, I swallow bullets she seems very hesitant to. Perhaps equally or more importantly, like most people, she doesn't seem to like to taste the bullets, i.e. ponder uncomfortable thoughts, accept uncertainty, etc. I, on the other hand, seem to take some perverse pleasure in thinking and talking about such topics. From her perspective, I sometimes "analyze things e.g. a poem, a play, the proper emotional response to situation X, to death to the point of being distracted from their inherent value."

3dclayh16y

For me, any (serious) talk of "overanalyzing" or "overthinking" things would be a huge red flag. But maybe I'm unusual in that.

0Alicorn16y

Using the word "overanalysis" isn't always a red flag for unwillingness to analyze where appropriate. Sometimes it just means recognizing that it is not worth six hours of nutrition research and blind taste tests to decide what cereal is the optimal breakfast. In a pinch, you can just grab the generic crisped rice or a prettily-packaged granola and call it good.

5dclayh16y

Of course. To clarify a bit, it's obviously possible to give things more thought than they deserve. But someone who habitually makes accusations of overthinking, to my mind, is indicating a contempt for thought itself, which is about the most horrible quality I can think of. (I believe I first came to this conclusion when I read this webcomic, though on looking back at it I'm not sure.)

0saturn16y

Some people just aren't very good at getting right answers through deliberate reasoning, but can get by using implicit reasoning. Combine that with the typical mind fallacy and you get someone who sees "overthinking" everywhere they look. But I think the problem here isn't so much about these implicit reasoners, but rather about contemptuous people in general, with contemptuous implicit reasoners as a special case.

4Vladimir_Nesov16y

So the program of understanding each other doesn't make progress. I agree with Alicorn, it's essential to establish a mode of communication where you can steadily work on disagreements, with the goal of ultimately resolving them in full. The arguments shouldn't turn into color politics, polarizing and alienating. A bit of advice, based on my experience, for a long-term conversion strategy: * Work on understanding your own position better, make sure you know why you believe what you believe before trying to convince another person to change one's mind. Maybe you are wrong. * Make the mode of interaction and your goals clear when you are arguing, distinguish analysis from social interaction. * Educate the person about fundamentals, thus steadily crafting tools for making deeper arguments in specific discussions. * Prefer shifting the discussion towards education about more general mistake that (might have) contributed to a specific mistake or confusion. In long term, it's more important than resolving a specific problem, and it's easier on the other person's feelings, as you are educating on an abstract theme, rather than attacking a conviction. * Don't argue the objects of emotional attachment, ever (unless the person is ready for that, at which point you are probably done with bootstrapping). Instead, work on finding an angle of approach (as suggested above, maybe something more fundamental) that allows you to make progress without directly confronting the issue. * Not everyone is going to change, some people are too dim or too shallow or persistently not interested.

1QuestionTime16y

Thank you, this sounds like very good advice for how to lead someone down the path. But given that she is reluctant to go down the path, do I want to lead her down it? She already believes that I can defend my views better than she can her's. She probably even believes that my views are closer to the truth. My guess is that she is reluctant to discuss and evaluate the fundamental facts of existence and our values, precisely because she cherishes certain aspects of her current worldview that she correctly believes she is likely to lose. I think its plausible that she'll end up less happy, and maybe less productive, after hearing about the preference utilitarianism and the opportunity cost of spending $80 to have flowers delivered to a friend (note: I'd never try to stop her from doing it, I'd just like to explain why I'm not going to) or after explaining why the idea that people have souls is incoherent (note: I would never say something that strongly. As you suggest I'd want to build up to it slowly, by asking questions and letting the conclusions fall out of the discussion.) Religious people report being happier. By many measures they also do more "good works." I wouldn't be surprised if the same were true of deontologists vs. consequentialists. Do I really have reason to believe she'll benefit from serious detailed discussion of our respective worldviews?

4Apprentice16y

I'd be curious to know what sort of power dynamic you have. My spouse believes I am more rational and intelligent than s/he is - but s/he's still the one who makes the decisions. I advise - my spouse decides. We both like it that way and we've had a successful and happy relationship for more than a decade. Now that I think about it, this is reminiscent of Eliezer's "Three Worlds Collide". You want to keep the rationalist cultists around - but you don't want them in charge :p

0jimmy16y

In "Three Worlds Collide", the rational one does have the power to override if necessary, which I think is very important. If you cant agree, you're doing it wrong- but it still happens occasionally. You'll get better results if you defer to the person that is more rational under those circumstances. In general, it seems like the right policy is to let whomever has harder to communicate data decide. This way, the decision maker is as informed as possible.

1Eliezer Yudkowsky16y

Actually, in 3WC the Confessor is supposed to be strictly charged with sedating people who depart the bounds of sanity. He goes outside this bound, which is completely against all the rules, and afterward he can no longer be called a Confessor.

2Alicorn16y

3Eliezer Yudkowsky16y

There are some worlds for which I have devised huge cultural, technological, and historical backdrops but this is not one of them.

1jimmy16y

I was referring to the part where the president went crazy and her confessor sedated her "and recommend to the government that they carry out the evacuation without asking further questions of your ship". If that doesn't count as the "power to override if necessary", then I'm missing a distinction somewhere.

0Eliezer Yudkowsky16y

Well, part of the point there was that their President would have been universally recognized by her own society as crazy, at that point, just as if she'd said she was hearing voices from her teapot. In contrast to say our own society where this would be considered perfectly normal madness in a politician. The reason her Confessor then needs to advise the government is that her Confessor was the only one to listen to an extremely classified conversation; in other words she has private info which she must summarize/convey somehow to the government.

0QuestionTime16y

Thank you very much for this data point.

3Annoyance16y

It's impossible to make someone reason if they don't wish to. It's impossible to force someone to acknowledge the truth if they don't want to acknowledge it. You don't need to lead her down any path, even if she were willing to follow. She probably already knows what the rational approach is and doesn't choose to implement it. In the event that she doesn't, teach her the method of rationality - not just the result - if she comes to you for help. Don't, otherwise. If she's comfortable with letting you be reasonable, and you're comfortable with letting her have her magical thinking, I'd say everything is fine.

0QuestionTime16y

Part of what motivates this post is that research on happiness suggests that people have a hard time predicting how happy they will be in various possible futures. Gilbert has suggested that introspection is so poor that we better off just asking people in that situation how they feel.

2jimmy16y

In my case, it was worth it. There may be a stubborn reluctance to "give in" or other lower level things that get in the way of believing the truth, but if at the top level she would really rather believe the truth, you're probably fine. If you can't get her to say in full honesty "I would rather believe the truth than made up 'feel good' stories", you're probably hosed. My girlfriend started out as a creationist christian and is now pretty much atheist. Overall, she is much more reasonable and can make the right decision on the types of things she previously did not. She seems to be about as happy as before, and when she's not she can recognize the cause so it's not nearly as damaging. I'd call that a success so far. In general, I have a pretty strict policy of calling each other on any BS. I attempt to get the point across in a nice way, but will persist until she understands even if it does make her upset in the short term. The one exception was when I found out that she believed in creationism. That was too big of a bite, so we left it as "agree to disagree", though I made it obvious that I did not respect her belief. I never made an attempt to deconvert her, but it did happen on its own. There's probably a better way to do it that can make the same progress without ever upsetting the person, but in my experience it ended up being worth it to push through anyway.

3hrishimittal16y

I'm in a situation which seems sort of the opposite of yours. I'm with a woman, who's more rational than any other I personally know. But the sex is just not very good, and I find myself getting physically drawn to other women a bit too much. I've struggled for weeks, trying to decide whether to continue or not. I've tried hard to think what I really want. And I think that if I were sexually satisfied, I would be very happy with the relationship because everything else seems perfect. So, I'm trying to work on that now. I'm paying more attention to being a loving and sensuous partner. Let's say I'm experimenting on the weak aspects of my relationship. If I were in your place, I'd take each point of disagreement on its own merit. If it's decisions where the results can be seen clearly I wouldn't argue but just politely point to the results. As far as religious beliefs are concerned, the more I think about it the more I feel, that defining myself as an 'atheist' is only useful in saying that I don't believe in God. Beyond that, it doesn't add anything valuable to my personality. It can't because it's a negative definition. So, I would try and deal with specific issues rather than try to convince my partner that theism is wrong. If she believes in magic, playful humour might lighten things up a bit. I also think it would be useful if you learnt more about her way of thinking, just like she has learnt about yours.

3QuestionTime16y

My advice is first, to talk to her a lot about sex and make it clear how important that is to you. If that doesn't work, consider asking her for permission to sleep with other women. That option would satisfy me in your situation temporarily, but I'd have to think about whether it would satisfy me longer term.

2byrnema16y

With respect to making your decision, I would advise you to just spend more time with her. It usually took me about 18 months to figure out how I would finally feel about someone, long-term, that I was initially attracted to. After that period of time, differences were either sources of annoyance (or something more neutral) or sources of contempt. If the latter, for either one of you, you're "too different". (Things are complicated by the fact that there's always a little contempt in a relationship, but the contempt I'm talking about will tend to grow and feel more important than everything else over time.) In other words, I don't believe, a priori, that in order to be consistently rational, a rationalist should seek out another rationalist. You're probably seeking someone that complements you, and pushes you to more fully experience life, and that's why opposites attract. As long as you have the same core values about what matters to you both. It takes time to determine if you share those. Regarding the magical thinking that perplexes you: it doesn't seem to me that most rationalists actually understand what it is that magical thinkers believe. For example, if you think about it in terms of scientifically true and false, you're probably not thinking about it the right away. Magical thinkers know they're not making scientific statements. For example, it's not false to believe you have a soul. Whatever she really means by having a soul, she does have. That's why rational arguments don't work. I think it's a matter of communication: she's not really expressing what she means by soul, and you're not really arguing (if you were to) that what she doesn't have is the soul she's talking about. Her description of a soul may be naive and if she's says anything about it that is scientifically falsifiable, then she is confused about what she means. But whenever she gives a description that is not scientifically falsifiable, and you see it as false, you are probably interpret

0rhollerith16y

I am pretty sure that most strong male rationalists are better off learning how the typical woman thinks than holding out for a long-term relationship with a strong female rationalist. Since this point is probably of general interest, I put it in a top-level post. Converting her to your worldview sounds like a bad idea in general. An additional consideration that applies in your particular situation is that converting a helping professional from deontologism to consequentialism will more likely than not make her less effective at work (because helping professionals need all the help they can get to care enough about their patients and clients, and worldview is definitely one source of significant help in that regard). Nobody has responded to the following: I, too, will refrain from commenting because you probably mean "strong swings of mood" and I do not have romantic experience with a moody woman. I do have romantic experience with a fiery woman, i.e., a woman easily aroused to strong negative emotions, but I doubt that is what you mean: in what I am calling a "fiery" woman, the emotion always dissipates quickly -- usually in a few minutes. You say, I would consider that a very positive sign in a prospective sexual partner -- maybe an extremely positive sign (the reason for my uncertainty being that I have never been with a woman whose expected global utility was as high as you describe) -- a sign that would make me pursue the woman much more keenly. The fact that you use language such as "would have net-benefits for her and for the world long-term" (emphasis mine) suggests to me that you are like me in the relevant characteristics and consequently should take it to be a very positive sign, too. The most I can say about the global expected utility (i.e., expected effect on the world in the long term) of any of my girlfriends up to now is that (1) she has many close friendships of long duration, and she is very caring and helpful to those friends or that (2) s

1QuestionTime16y

Thanks very much for your thoughts, and for making a top level post on the topic. Yes, her contribution to social welfare is something I find very attractive, and you help me remember just how important and rare that is.

0jimrandomh16y

Rationality sometimes goes badly wrong, when important details don't fit into a neat reasoning structure or a fatal flaw in argument goes undetected. Emotional reasoning sometimes goes badly wrong, when it deals with corner cases or situations too far from the environments it was evolved to handle. Rationality goes wrong less often, but crucially, they go wrong in different and mostly non-overlapping circumstances. If you have a different world view and reasoning style than your partner, then this might produce conflict, but it also gives you both far better sanity checking than any like-minded person could. You can't transform her mind, but you can act as a rationalist oracle. You speak of this as though it were only a flaw, but in fact it has both an upside and a downside. Use the upside, and mitigate the downside.

1Vladimir_Nesov16y

Gaining rational framework doesn't deprive you of emotional reasoning, but allows you to guide it in normally confusing situations.

0Vladimir_Nesov16y

How strong is your own understanding of rationality? Why are you an atheist, (why) do you believe science works, what is the difference between one person who is actually right and another person who is merely confused, etc.? How smart is she (that is, how easy it'll be to positively apply more confused and complex arguments, not requiring of you great feats of persuasion)? I'll be interested in what other people think about applying the corpus of Overcoming Bias directly (doesn't seem like a generally good idea to me, requires a person to be of a kind that'll maintain focus for extended period of time).

-1QuestionTime16y

I think I'm quite rational and have a decent understanding of aspects of rationality that I haven't managed to implement yet. I think karma is a very imperfect measure, but I'll note that I have more than 100 and less than 400. She is probably one standard deviation above average in terms of IQ, and would score more highly when considering other kinds of intelligence. The main problem in convincing her to think more rationally is emotional resistance. Thank you for responding - see my "Added" to the original comment.

1Vladimir_Nesov16y

Huh? Karma on this site primarily shows positive contribution/participation (if you are able to contribute positively), treating it as a "measure of rationality" is a bizarre perspective. Please try to outline your position on the specific questions I suggested, or something of that kind, it's hard to make such a decision in a well-known case, but yet harder to construct fully general advice. For another example, why do you think it's important to get away from magical thinking? Is it? What is your motivation for thinking about rationality, and for dispelling the other person's confusion? "Compatibility" of worldviews?

1QuestionTime16y

I would probably give you a response you liked better if I understood why you were asking what you were asking. Because the evidence favors atheism and suggests science leads to truth more often than other approaches to belief formation? I could link to arguments but I don't see the point in trying to explain these things in my own words. Does it help to know that I usually agree with your comments and with the LW consensus, where it exists? Is the implication that the more rational I am, the more of a problem my partners rationality will be? I don't think I understand this question. I think the importance of getting away from magical thinking varies across people and contexts. I'm not confident I know how important it is, or even whether its helpful, for some people. Its clear that getting away from magical thinking can sometimes help people achieve their personal goals and help make the world a better place. I enjoy the process regardless of the consequences. But I also hope that it will help me in my career and help me contribute to the world. I think my partner and I both experience some level of discomfort at knowing that our worldviews are in significant conflict, even though this conflict seems to coexist with a high degree of respect for how the accomplishments of the other. It is unfortunate that we basically have to avoid certain topics of conversation that we both find important and that our emotional reaction to things often differs. You might check my responses to Alicorn to learn more. Once again, thank you very much for responding.

2orthonormal16y

This is a delicate topic, but I think Vladimir is trying to tell whether you really use rationality to the degree you claim, or whether you rather accept certain opinions of people you see as rationalists, and wish others shared them. In the latter case, it doesn't matter that the clash is between rationalist and irrationalist opinions: the conflict is isomorphic to any relationship between people of different religions or political parties, and much of the advice for those people should work for you. It's the former case that would require more particular advice. I'm afraid that, in the absence of seeing your thought process, much of this looks like guessing the teacher's password to me. I'd be happy to be corrected, though. EDIT: Wow, that sounds really tactless and dismissive of me. I retract my accusation, on the basis of (1) not having any real justification and (2) it would set a bad precedent, especially for the sort of reception newcomers get.

2QuestionTime16y

Its interesting that people seem to a) be as skeptical of my rationality as they seem to be, and b) think that is the crux of the matter. Regarding a), if someone tells me that they've been reading OB/LW for quite a while and that they think they are considerably more rational than their romantic partner, I think it is very likely that they are correct. But maybe if I was on the other side I would react differently. If I knew of an easy way to prove my rationality I would, but I don't. Even writing an original rational essay wouldn't prove much because I could easily be irrational in other domains. Regarding b), I'm not sure exactly how important it is that potential advice-givers have a very accurate estimate of my rationality (and my girlfriend's rationality). Perhaps it would be helpful to focus on more specific aspects of our beliefs and approaches to experiencing and acting in the world. I lean towards preference utilitarianism, though I don't walk the walk as well as I should. I attempt to calculate the costs and benefits of various choices, she does this too sometimes, but doesn't like applying it reflexively. She believes in spirits, I'm into Dennett and Dawkins (though I see positive aspects to religion/spirituality) My partner and I both agree that: She is much more emotional and I am more rational. She is more prone to depression. She has more faith in intuition, I'm more skeptical of it. Lets say you've read everything I've written here and you think I'm probably no more rational than my partner. ok, that's fine, I'd be happy to hear advice that works for two equally irrational people with different beliefs/values/approaches to experiencing and acting in the world.

0[anonymous]16y

So the program of understanding each other doesn't make progress. I agree with Alicorn, it's essential to establish a mode of communication where you can steadily work on disagreements, with the goal of ultimately resolving them in full. The arguments shouldn't turn into color politics, polarizing and alienating. A bit of advice, based on my experience, for a long-term conversion strategy: * Work on understanding your own position better, make sure you know why you believe what you believe before trying to convince another person to change one's mind. Maybe you are wrong. * Make the mode of interaction and your goals clear when you are arguing, distinguish analysis from social interaction. * Educate the person about fundamentals, thus steadily crafting tools for making deeper arguments in specific discussions. * Prefer shifting the discussion towards education about more general mistake that (might have) contributed to a specific mistake or confusion. In long term, it's more important than resolving a specific problem, and it's easier on the other person's feelings, as you are educating on an abstract theme, rather than attacking a conviction. * Don't argue the objects of emotional attachment, ever. Instead, work on finding an angle of approach (as suggested above, maybe something more fundamental) that allows you to make progress without directly confronting the issue. * Not everyone is going to change, some people are too dim or too shallow or persistently not interested.

[-]nazgulnarsil16y50

it would appear to the average person that most rational types are only moderately successful while all the extremely wealthy people are irrational. due to not seeing the whole sample space (that larger proportion of rational people enjoy moderate success vs a tiny fraction of irrational people who enjoy major success) I don't think a lot of our arguments gain traction with people. Most people infer from outliers as a matter of course.

Now combine this with the idea that signaling rationality is also signaling that we think we deserve more status and deci... (read more)

3pjeby16y

This only makes sense if you consider "rational" to equal "geeky Spock-wannabe", in which case the correlation is reasonable. Bill Gates comes close to that stereotype, though, except for being irrationally passionate about controlling everything and yelling profanity a lot. I do know a few wealthy guys - some are more stereotypically rational, others are only moderately instrumentally rational in the context of "do what works, stop doing what doesn't", while being utterly wacko with respect to everything else. Of course, these guys are only moderately wealthy -- businesses with 8-figure annual sales, net worth below $100 mill, so they're certainly not in BillG's league. Anyway... I can see normal people associating people who like to signal an ideal of rationality with a lack of success and wealth, as it seems to me there's a correlation there. However, being rational - at least instrumentally - doesn't require even a fraction of the sort of mathematical rationality promoted here. See, all of the wealthy people I know, got that way because they don't mind losing thousands or even tens of thousands of dollars trying things that might not work, and which they have no guaranteed way to evaluate in advance, and no science to study. They don't mind being wrong, because they understand that they're in a business where the black swans are on the upside, not the downside.

0JamesCole16y

We're talking here about perception, not reality, and I'm sorry to say that "geeky Spock-wannabe" probably does equate to the average person's perception of "most rational types" .

0JamesCole16y

(Most people take a concept to mean whatever most uniquely distinguishes it from other concepts - so 'rational' means whatever, in the characteristics they associate with rationality, is most unique and different from the other concepts they have. i.e. Spock-like).

1alvarojabril16y

Not only that, often people's goals require irrational thinking. If you're hoping to find a mate in a religious community, or if you're a businessman bringing the free market to the boonies there's an obvious rational incentive to believe irrational things.

0[anonymous]16y

A few related hypotheses: Rationality is highly correlated intelligence. Intelligence is highly correlated to proneness to boredom (in the range IQ >= 90). Boredom is likely to make you fail at the task at hand. The most reliable ways to get rich are perceived as boring to intelligent people. It seems to me that boredom occurs when the task at hand is perceived as not leading towards any highly valued goal. Rationality will gladly give you reasons why some of your goals should not be highly valued. For example, I would like to have millions of euros. I have recently seen evidence that people lead happy lives no matter what their circumstances are. I have discovered that nearly anything I think would be great fun costs less than a thousand euros. I am convinced money does not make one more attractive to women. If I was less analytical about these issues, I think I might perceive my goal as having higher value, and working towards it would be less boring.

4jscn16y

According to research K.E. Stanovich, this is not the case: See http://www.project-syndicate.org/commentary/stanovich1

[-][anonymous]16y40

It would be useful if everyone used quote markup more often; it can be difficult to figure out what someone's agreeing to when they say "I agree" and there are a dozen posts separating the agreement from the original post.

[-]Mike Bishop16y30

Is anybody aware of an academic course which tries to teach OB / LW type stuff? I might someday have the opportunity to teach such a class.

Has anyone made and organized a list of important posts that could serve as the rough draft for a syllabus?

0thomblake16y

No, I've never heard of such a course. Depending on what you mean by that 'type stuff', I'd expect it to be in the psychology, anthropology, economics, or polisci departments.

[-]taw16y20

One quick question - quantum game theory - useful or a mathematical gimmick? I can follow the math, but it's not clear to me if it makes any sense in the real world or not. Can anybody who took a closer look at it tell me what they think?

[-][anonymous]16y10

Suppose I decide that I'm going to partake in a pleasurable activity. How far removed must the decision be from the activity before the decision is no longer reinforced by operant conditioning?

[-]Cyan16y10

Educate me, LW hive mind. Robin Hanson has mentioned that prediction markets can give not just probability assessments on discrete sets of outcomes but also probability distributions over such assessments that let us know how (un)certain the market is concerning a particular assessment (or at least, that's how I interpreted his words). Does anyone have links to descriptions of such methods?

1Douglas_Knight16y

The key phrase is "scoring rules." Another phrase is "combinatorial prediction markets," but I think that's something else.

[-][anonymous]16y10

del

[-]mathemajician16y10

In case it hasn't already been posted by somebody, here's a nice talk about irrational behaviour and loss aversion in particular.

[-]Ziphead16y10

Perhaps this is a known issue, but since I haven't seen it discussed, I thought I'd mention that images don't seem to work in some (all?) of the old posts imported from Overcoming Bias. See for example:

Timeless Physics

The first few pics in that particular post is available from an external server if you click them, but I don't see them inline. The last picture seems to have been hosted at Overcoming Bias, and is no longer accessible.

1Eliezer Yudkowsky16y

Work is underway, apparently.

[-]timtyler15y00

Has anyone read: "3:16 Bible Texts Illuminated by Donald E. Knuth"

http://www.amazon.com/3-16-Bible-Texts-Illuminated/dp/0895792524

...?

[-]dumbshow16y00

Would you all please recommend books on many-worlds? I liked The End of Time but I thought the treatment of MWI was too cursory.

[-]spuckblase16y00

Hi,I have a question I haven't seen adressed after a qucik search. A friend of mine has been diagnosed with mild paranoid schizophrenia after he attacked his brother and got hospitalized thereafter. this was 2 years ago.he got (and still gets) medical treatment (some sort of neuroleptica, I suppose), but not much more. it sort of helped, he has a nice job and some superficial friendships (he never had great interest in things social). Now, the paranoia has surfaced again. I guess it was there all along, but nobody knew for sure. We're afraid it'll get wo... (read more)

1Vladimir_Nesov16y

I guess you'd need an expert psychiatrist rationalist to benefit from this angle. Any obvious advice runs too high a risk of being invalidated by more solid facts known about people in such conditions, so one would need to start with that knowledge.

0spuckblase16y

Thanks for the quick response. I have some trouble understanding, probably due to the language barrier. Do you mean 1) if at all, olny an "expert psychiatrist rationalist" might talk him out of it, or, 2) I should seek knowledge from such a person? if 2), where to find them? Any suggestions?

4Vladimir_Nesov16y

I'm not aware of anyone here being a professional psychiatrist, so can't help with that. I'm only warning that seeking advice from people who know nothing about a phenomenon is generally a bad idea, even if those people have some orthogonal advantage. For example, it's (much) better to seek construction advice from an ordinary engineer that from a genius who is not an engineer.

[-][anonymous]16y00

And for another suggestion for the site itself, it should be possible to tag posts (especially articles, possibly comments) by language, and let users pick what languages they want to see. The interface wouldn't necessarily have to be translated; it would just be nice to have some support for multilingualism.

1thomblake16y

there's a thread for requested features.

0[anonymous]16y

Thank you.

[-]hrishimittal16y00

I'm considering donating to World Vision UK. Does anyone know much about them?

More generally, is there an easy way to find out how good a charity is? Are there reviews done by third parties?

0MichaelHoward16y

I recommend you check out GiveWell, they're doing this sort of thing.

[-]Lightwave16y00

This might not be the right place to ask, but I'll try anyway:

I read an online paper/article on global dictatorship / totalitarianism as an existential threat a while ago, but I can't find it anymore. I've probably found it on OB / SIAI's website or something like that in the first place, but can't find it there now. Would anyone know of such an article (or any good article on the topic, for that matter)?

2RobinHanson16y

That was probably Bryan Caplan's chapter The Totalitarian Threat for the GCR book.

0Lightwave16y

Yup, that's the one, thanks!

1[anonymous]16y

Bryan Caplan's chapter in Global Catastrophic Risks

[-]alvarojabril16y00

Two links that might foster discussion:

http://www.philosophersnet.com/games/

Fun online rationality and anti-bias oriented games. I particularly enjoyed "Staying Alive" (testing conceptions of selfhood). And

http://bloggingheads.tv/diavlogs/20086

Great discussion, I hadn't seen Gendler before but Bloom is always good. Reminded me a little of the IAT discussion here a few months ago.

[-]evtujo16y00

Any fellow OB/LW-ers attending OSCON this year? If you are interested in meeting up there let me know. Perhaps respond to this comment or email me directly and we'll see what we can work out. (You can contact me via gmail at user name evtujo).

[-]Tom_Talbot16y00

What a critic might say about Less Wrong:

1) The purpose of the pursuit of rationality is to increase an individual's understanding of and power over their environment and the people in it.

2) The only way to establish rational thinking in a group of people not otherwise disposed towards it* is to establish a group norm of praising rational thinkers and shaming the irrational, by an established standard of rationality.

Therefore:

Rationalists are power-seekers, and the pursuit of rationality is inherently elitist and exclusionary.

*That is to say, the vast majority of people.

1AdeleneDawner16y

Number two is flawed. You can also establish rationality in people not otherwise interested in it by convincing them that it will be useful to them in pursuing other goals that they're interested in.

0Annoyance16y

But recognizing that rationality is useful requires the application of rationality. So you'll only convince people that already agree, to the degree that they do agree. The real obstacle is that people often don't want to apply rationality and prefer to use magical thinking instead. Rationalists aren't those who know how to be rational, they're the ones who choose rationality over magic when push comes to shove.

1AdeleneDawner16y

You don't have to wait until they already agree, just until they come up against a situation where their magical thinking is obviously not working. Show them that rationality fixes the problem in a few of those situations, and there's a decent chance that they'll start trying it on their own.

0Annoyance16y

That's not nearly enough. To work, you have to pick a situation where the emotional obstacles to thinking rationally are weak, the tendency to avoid extended thought is most likely to be overridden because the outcome is important, and then present the rational strategy in a way that doesn't denigrate magical thinking. It's difficult to locate such moments, as they are very rare. Better by far simply to find people who genuinely want to be right.

0AdeleneDawner16y

A situation where the person has already figured out that their way of handling things isn't working, and they're looking for advice, fits those criteria. They're still unlikely to ask for advice from someone who's more likely to mock them than to be useful, though, which is probably why you hear about those situations so infrequently.

1Annoyance16y

But such people usually don't conclude that the magical thinking itself isn't working. They just want help finding a specific magical argument that will end up with their getting what they want.

1AdeleneDawner16y

I don't think I've ever known anyone who was actually against rationality altogether. Most people just want a solution that works, and if the one they wind up using is rational, they don't have a problem with that. And if you can teach them some rationalist skills while you're giving them that solution, all the better. It takes time to bootstrap people that way, but it's far from being impossible.

0Tom_Talbot16y

The imaginary critic points out that convincing wholly irrational people to be rational is unlikely to be possible using rational arguments. A rationalist would need power over the irrationalist in order to change their mind.

0AdeleneDawner16y

Or, we could just use the same systems that less-rational people use to convince each other of things, but in a way that in in line with the rationalist ethos. Since when does valuing rationality limit what other skills we can use?

0Tom_Talbot16y

Critic: "'Other skills' means skills of manipulation, rationalists readily abandon honesty in favour of power." I just want to clarify that this is not what I think, but I have seen comments of this kind made about Overcoming Bias (on other blogs), though never about Less Wrong. I have never written anything in reply. I was wondering how anyone here would respond to this kind social criticism. Would you even care enough to respond?

[-]pjeby16y140

Would you even care enough to respond?

Anticipating critics and responses to them is largely a waste of time, if they are determined to be against you. Whatever you say will only be fodder for the next attack, and you are wasting precious time and energy being pinned down by their fire.

What we want is responses for people who are not the critics, but may have heard the critics' arguments. That's a considerably less-demanding audience.

7Eliezer Yudkowsky16y

This is a key distinction which all rationalists trying to talk to the public should keep in mind in all places at all times.

2AdeleneDawner16y

Power and honesty (or, more accurately, Friendliness?) aren't mutually exclusive, but yes, that's a major issue. Maybe we need to spend more time talking about Friendliness here, even though it's not exactly a rationalist topic?

0nazgulnarsil16y

I'm highly skeptical of the idea that rationalism isn't a strategy for getting laid. It's just a strategy that says the direct methods of using manipulation are worse than using rationality to get rich (or save the world, or something in between). This is because we recognize that most manipulators fall prey to their own shoddy reasoning and are ultimately handicapping themselves even if they have more success in the short term. Of course such rationalizing about short term vs long term success can be thought of as a natural response from high IQ nerds who find little success with women in day-to-day life. I still think this is true even if it turns out to be correct that long term planning pays off (as is usually the case, unless you get hit by a truck :) bring on the downvotes.

2Mike Bishop16y

Rationalism, as discussed on LW, is not particularly helpful for getting laid. Perhaps what you mean is that our interest in rationalism is motivated by mental circuits which are indirectly oriented towards us getting laid. This is true, but this could also be said of peoples interest in sports, dance, conversation, literature and the arts, etc. Is there any reason to believe it is more true of rationalism?

4pjeby16y

Well, if you filter out the epistemic stuff and focus on various instrumental-rationality bits like: * Willingness to accept unpleasant or unpopular ideas * Willingness to try things you see others succeeding with, even if they seem to be based on ideas that are absurd or impossibly wrong * Willingness to suspend disbelief while you are doing something, separating evaluating from doing * Observing reality to see what works, rather than imagining you are more (or less) successful than you actually are, by devising as-objective-as-practical test/success criteria in advance * Accepting others' beliefs and worldviews at face value, without judging them "good" or "bad" Then yeah, you will find some useful things here, though of course perhaps not nearly as useful things as studying some domain-specific materials on the topic. But the above ideas will serve you well in any domain that involves influencing human behavior, whether it's your own behavior or someone else's.

[-]Drahflow16y00

How to design utility functions for safe AIs?

Make a utility function which will only emit positive values if the AI is disabled at the moment the solution to your precise problem is found. Ensure that the utility function will emit smaller values for solutions which took longer. Ensure the function will emit higher values for world which are more similar to the world as it would have been without the AI interfering.

This will not create friendly AI, but an AI which tries to minimize its interference with the world. Depending on the weights applied to the three parts, it might spontaneously deactivate though.

6Eliezer Yudkowsky16y

"AI is disabled" and "world more similar to the world as it would have been without the AI interfering" are both magical categories. Your qualitative ontology has big, block objects labeled "AI" and "world" and an arrow from "AI" to "world" that can be either present or absent. The real world is a borderless, continuous process of quantum fields in which shaking one electron affects another electron on the opposite side of the universe.

1Nick_Tarleton16y

I understand the general point, but "AI is disabled" seems like a special case, in that an AI able to do any sort of reasoning about itself, allocate internal resources, etc. (I don't know how necessary this is for it to do anything useful), will have to have concepts in its qualitative ontology of, or sufficient to define, its disability – though perhaps not in a way easily available for framing a goal system (e.g. if it developed them itself, assuming it could build up to them in their absence), and probably complicated in some other ways that haven't occurred to me in two minutes.

3Eliezer Yudkowsky16y

Suppose the AI builds devices in the environment, especially computational devices designed to offload cognitive labor. What do you want to happen when the AI is "switched off"? Hence, magical category.

0Vladimir_Nesov16y

Interesting, I didn't think of this situation. How do you define "lack of intelligence" or "removal of the effect of intelligence" in the environment, so that an AI can implement that state? How is this state best achieved? Once the system is established, the world will ever be determined by a specific goal system, even if the goal is for the world to appear as if no AI is present, starting from a certain time. The best solution is for AI to pretend of not being present, "pulling the planets along their elliptic orbits".

0Nick_Tarleton16y

D'oh. Yes, of course, that breaks it.

5SoullessAutomaton16y

As an aside, "waiting for Eliezer to find a loophole" probably does not constitute a safe and effective means of testing AI utility functions. This is something we want provable from first principles, not "proven" by "well, I can't think of a counterexample".

1Nick_Tarleton16y

Of course, hence "...and probably complicated in some other ways that haven't occurred to me in two minutes.".

1SoullessAutomaton16y

Right. I know you realize this, and the post was fine in the context of "random discussion on the internet". However, if someone wants to actually, seriously specify a utility function for an AI any description that starts with "here's a high-level rule to avoid bad things" and then works from there looking for potential loopholes is deeply and fundamentally misguided completely independently of the rule proposed.

2Jordan16y

This might make a fun game actually. Player One sets up a hypothetical AI (utility function + restraints), and Player Two, or the rest of the group, must find a situation where a super intelligence with that utility function would lead to a dystopia. Here's my entry: 1) Utility is obtained by 'votes'. Every person can either give +1 utility or -1 utility to the AI each day. To avoid issues of uncertainty over what constitutes a person, before the AI is turned on everyone is given a voting button. Additional voting buttons that may be created provide no utility. 2) Strongly discount future votes, using U(t) = 1 / 2^t for the new utility, where t is time in days.This way a constant vote for all time yields a finite utility. 3) The AI can not take any action until one year after it has been turned on, except for text outputs to satisfy (4) 4) The AI has an oracle component. Any voter can ask how their life on a particular future day will vary from the present day. The AI must answer to the best of its abilities.

2Richard_Kennaway16y

I ask it a question as in (4), and it tells me "build this machine [specifications included] and you will get [something the AI believes it can tempt me with]". It exerts the best of its ability in persuading me. Unknown to me, the machine that it wants me to build takes command of all the voting buttons and jams them into giving the AI +1 every day. I don't think Player One can win this game.

1Jordan16y

I should have been more clear about what I meant by an oracle. The only utility the AI takes into account when answering a question is accuracy. It must be truthful and only answers questions about differences in the days of a voter. A valid question could be something like "A year from now, if you do [horrendous act] would my voter button still vote you down?" If not, I'd vote the AI down for the whole year until it begins acting. While I recognize the huge threat of unfriendly AI, I'm not convinced Player One can't win. I'd like to see the game played on a wider scale (and perhaps formalized a bit) to explore the space more thoroughly. It might also help illuminate the risks of AI to people not yet convinced. Plus it's just fun =D

2Vladimir_Nesov16y

You can't get anything useful out of AI directed to leave the future unchanged, as any useful thing it does will (indirectly) make an impact on the future, through its application by people. Trying to define what kind of impact really results from the intended useful thing produced by AI brings you back to the square one.

0Drahflow16y

Minimize is not "reduce to zero". If the weighting is correct, the optimal outcome might very well be just the solution to your problem and nothing else. Also, this gives you some room for experiments. Start with a function which only values non-interference, and then gradually restart the AI with functions which include ever larger weights for solution finding, until you arrive at the solution.

1Vladimir_Nesov16y

Or until everyone is dead.

-2Drahflow16y

If the solution to your problem is only reachable by killing everybody, yes.

1olimay16y

Um, then this is not a "safe" AI in any reasonable sense.

0Drahflow16y

I claim that it is, as it is averse to killing people as a side effect. If your solution does not require killing people it would not.

3orthonormal16y

Stop, and read The Hidden Complexity of Wishes again. To us, killing a person or lobotomizing them feels like a bigger change than (say) moving a pile of rock; but unless your AI already shares your values, you can't guarantee it will see things the same way. Your AI would achieve its goal in the first way it finds that matches all the explicit criteria, interpreted without your background assumptions on what make for a 'reasonable' interpretation. Unless you're sure you've ruled out every possible "creative" solution that happens to horrify you, this is not a safe plan.

1[anonymous]16y

If it can understand "I have had little effect on the world", it can understand "I am doing good for humanity". A "safe" utility function would be no easier and less desirable than a Friendly one.

2Nick_Tarleton16y

No easier? There's a lot of hidden content in "effect on the world", but presumably not all of Fun Theory, the entire definition of "person", etc. (or shorter descriptions that unfold into these things). An Oracle AI that worked for humans would probably work just as well for Babyeaters or Superhappies (in terms of not automatically destroying things they value; obviously, it'd make alien assumptions about cognitive style, concepts, etc.).

1Eliezer Yudkowsky16y

I agree with that much, but the question is whether there's enough hidden content to force development of a general theory of "learning what the programmers actually meant" that would be sufficient unto full-scale FAI, or sufficient given 20% more work.

0SoullessAutomaton16y

Does moving a few ounces of matter from one location to another count as a significant "effect on the world"? Does it matter to you whether that matter is taken from 1) a vital component of the detonator on a bomb in a densely populated area or 2) the frontal lobe of your brain? If it does matter to you, how do you propose to explain the difference to an AI?

0Nick_Tarleton16y

In general, yes; you can and should be much more conservative here than would fully reflect your preferences, and give it a principle implying your (1) and (2) are both Very Bad. But, the waste heat from its computation will move at least a few ounces of air. Maybe you can get around this by having it not worry (so to speak) about effects other than through I/O, but this is unsafe if it can use channels you didn't think of to deliberately influence the world. Certainly other problems, too – but (it seems to me) problems that have to be solved anyway to implement CEV, which is sort of a special case of Oracle AI.

2SoullessAutomaton16y

Quite so. The waste heat, of course, has very little thermodynamically significant direct impact on the rest of the world--but by the same token, removing someone's frontal lobe or not has a smaller, more indirect impact on the world than preventing the bomb from detonating or not. Now, suppose the AI's grasp of causal structure is sufficient that it will indeed only take actions that truly have minimal impact vs. nonaction; in this case it will be unable to communicate with humans in ways that are expected to result in significant changes to the human's future behavior, making it a singularly useless oracle. My intuition here is that the insights required for any specification of what causal results of action are acceptable is roughly equivalent to what is necessary to specify something like CEV (i.e., essentially what Warrigal said above) in that both require the AI have, roughly speaking, the ability to figure out what people actually want, not what they say they want. If you've done it right, you don't need additional safeguards such as preventing significant effects; if you've done it wrong, you're probably screwed anyways.

0timtyler16y

That's pretty condensed. One of my video/essays discusses the underlying idea. To quote: "One thing that might help is to put the agent into a quiescent state before being switched off. In the quiescent state, utility depends on not taking any of its previous utility-producing actions. This helps to motivate the machine to ensure subcontractors and minions can be told to cease and desist. If the agent is doing nothing when it is switched off, hopefully, it will continue to do nothing. Problems with the agent's sense of identity can be partly addressed by making sure that it has a good sense of identity. If it makes minions, it should count them as somatic tissue, and ensure they are switched off as well. Subcontractors should not be "switched off" - but should be tracked and told to desist - and so on." * http://alife.co.uk/essays/stopping_superintelligence/

0scav16y

I haven't thought this through very deeply, but couldn't the working of the machine be bounded by hard safety constraints that the AI was not allowed to change, rather than trying to work safety into the overall utility function? e.g. the AI is not allowed to construct more resources for itself. No matter how inefficient it may be, the AI has to ask a human for more hardware and wait for the hardware to be installed by humans. What I would want of a super-intellient AI is more or less what I would want of a human who has power over me: don't do things to me or my stuff without asking, don't coerce me, don't lie to me. I don't know how you would code all that, but if we can't design simple constraints, we can't correctly design more complex ones. I'm thinking layers of simple constraints would be safer than one unprovably-friendly utility function.

1orthonormal16y

There's been a great deal of discussion here on such issues already; you might be interested in some of what's been said. As for your initial point, I think you have a model of a ghost in the machine who you need to force to do your bidding; that's not quite what happens.

1scav16y

Nope, I'm a software engineer, so I don't have that particular magical model of how computer systems work. But suppose you design even a conventional software system, to run something less dangerous than a general AI, like say a nuclear reactor. Would you have every valve and mechanism controlled only by the software, with no mechanical failsafes or manual overrides, trusting the software to have no deadly flaws? Designing that software and proving that it would work correctly in every possible circumstance would be a rich and interesting research topic, but never be completed.

1loqi16y

One difference with AI is that it is theoretically capable of analyzing your failsafes and overrides (and their associated hidden flaws) more thoroughly than you. Manual, physical overrides aren't yet amenable to rigorous, formal analysis, but software is. If we employ a logic to prove constraints on the AI's behavior, the AI shouldn't be able to violate its constraints without basically exploiting an inconsistency in the logic, which seems far less likely than the case where, e.g., it finds a bug in the overrides or tricks the humans into sabotaging them.

0loqi16y

What makes you think these constraints are at all simple?

[-]timtyler15y-10

New rationalist blog: "gullibility is bad for you":

http://gullibilityisbadforyou.blogspot.com/

Moderation Log

145Comments