In particular, how can an AI estimate priors. I'm sure there is a lot of existing work on this.

There is. For example, one can use the Jeffreys prior, which has the desirable property of being invariant under different parametrization choices, or one can pick a prior according to the maximum entropy principle, which says to pick the prior with the greatest entropy that satisfies the model constraints. I don't know if anyone's come up with a meta-rationale that justifies one of these approaches over all others (or explains when to use different approaches), though.

Hopefully an AI will be able to get its hands on large amounts of data. Once it has that, it doesn't matter very much what its priors were.

Sure, if the priors are arbitrary, the Bayesian approach's output is arbitrary. But if the priors make sense, the Bayesian approach works. Or in other words: just like any other algorithm good or bad, GIGO.

I'm a new reader, and I thought you might like to know that this is the post that made me feel like it might be okay to get involved in the LW community. My initial instinct when I started looking around here was trepidation--it reminded me of some people I know who are very smart, intellectual, and rational, who love to debate and analyze ... and to argue with people who might not want to, and who are hopeless at understanding people less rational than themselves, don't acknowledge their own emotions, and don't see how irrational it is to think and behave that way. Before joining the conversation, I needed to hear that this place was not for those people--not an intellectual wankfest but something actually practical, even when it comes to the less reasoned parts of ourselves. So, thanks for that.

Now to salvage the relevance of this comment.

As a practical suggestion for ourselves and each other, it might be interesting to experiment with non-argumentative ways of conveying a point of view: tell an illustrative story, express your idea in the form of an epigram, or even quote a poem or a piece of music or a photograph.

I would have worded this more strongly, myself. In my experience, people who are themselves inclined towards reasoned debate, even civilly, drastically overestimate how much other people are also inclined towards debate and argument. They are of course generalizing from one example, but in this particular case they're also doing intense harm to their social relationships and to the point they're trying to communicate. In their minds, they're engaging in a way which displays and encourages intelligent thought, but to people who dislike a heavily oppositional mode of conversation, they come off as belligerent prats.

The point here is that those who enjoy an adversarial style of heated conversation might find their communication more effective and more readily listened to by a dissimilar audience if they choose to present their ideas in a way that seems to them to be more indirect--perhaps not quite to the level of writing a sonnet about it, but by speaking in general terms, avoiding language which invokes an accusatory tone whether or not personal accusation is intended, and so on. In short, intellectuals that no one will listen to have a lot to learn from poorly-educated but widely-admired poets.

Also, at the risk of exposing my unintellectual taste, my "O Isis Und Osiris" is the bassline of Jet's "Are You Gonna Be My Girl." I briefly worked in QA at EA (many of you know the reputation of that job and also that company, and those who don't can infer it from the tone of this parenthesis). I was testing the original Rock Band, and when I was having a rough morning and didn't want to be there, I'd play through that bassline a couple of times and I'd be doing all right.

Wow, this really brings home the arbitrary nature of the Bayesian approach. If we're trying to get an AI to determine what to do, it can't guess meaningful priors (and neither can we come to that). I presume when it is applied there is a load of theoretical approaches to prior model estimation or is a uniform prior just used as default? In which case are there other occasions when a frequentist and bayesians probability estimates differ?

Does anyone have any suggestions for careers or lifestyles where one can feel a sustained sense of satisfaction?

At the risk of pointing out the obvious, different careers provide satisfaction in different ways. Some jobs, such as that of a beautician, provide the satisfaction of a job well done several times a day. Others, such as computer-game programmer, provide that kind of satisfaction several times a decade. Which appeals to you?

What balance of success and failure do you want? There are jobs in which success is achieved only one time in ten tries, yet the psychic payoff from that one success more than makes up for all the failures. Personally, I couldn't live like that. How about you?

What kinds of social interaction do you want out of your career? There are careers in which you work in almost monastic seclusion and others in which you are in continual interaction with colleagues. How do you feel about interaction with the public as a whole? Repeated contact with complete strangers? Contact restricted to particular age groups or particular social classes? It is up to you.

Are you the kind of person who draws satisfaction out of simply getting the job done, or are you only satisfied when it is done with a certain artistry? Do you want recognition? Do you detest criticism? Would you rather find the cure for disease or cure many patients with various diseases? Prove a theorem or explain a proof? Make a fishing pole, or catch a fish? Or maybe save a species of fish?

The answer to each of these questions may be relevant in making a good career choice. But the trouble is that the typical 20-year-old is completely unequipped to answer them truthfully. Truthful answers to these questions are learned by experience. But the answers that our twenty-year-old actually gives are based on what kind of person he/she wants to be, rather than what kind of person he/she is.

For this reason, I would suggest that every young person take a few years out of the standard educational career track to learn something about his/her self, before committing to a difficult and costly period of training or apprenticeship in some narrow specialty. "Wasting" those years may appear inefficient, but it is far better than wasting a life.

I think your very first step Identify is the key to all this.

Is it rational to pursue an irrational goal rationally?

Our culture focuses on external validation, achievement and winning. My concern is that this is a form of manipulation focused on improving a societies economic measures of value over an individual's personal satisfaction.

In contrast, the science of happiness seems like a good start. This work seems to focus on developing techniques to come to feel satisfaction with ones current state. Perhaps a next step is to look at how communities and organisations can be structured to support this. Speaking for myself I naively assumed that making computer games would be an enjoyable career because I thought that making a game and playing a game would be similar, this is not the case. Does anyone have any suggestions for careers or lifestyles where one can feel a sustained sense of satisfaction? Or indeed a rational means to select/create one?

So it is with charities, I cannot easily experience for myself whether they convey benefit, nor have I found a trusted source of recommendations.

Have you seen GiveWell before? If not, it's well worth looking into.

If you have seen GiveWell before and don't trust its recommendations, why? They're always looking to improve and would welcome suggestions.

For about 3 years now I've been giving to a number of charities through a monthly standing order. Initially setting it up was very satisfying and choosing the charities was a little like purchasing a new gadget, assuming a hands on experience is not available, and there are no trusted reviewers, I look at the various options and go with the ones whose advertising most closely reflected my personality and who looked the least like charlatans. With gadget purchases I find these indirect signals much more informative of the experience with the product than any enumeration of features, while individual statistics can convey value (and certainly attract attention) they often obscure the interaction experience or final quality.

So it is with charities, I cannot easily experience for myself whether they convey benefit, nor have I found a trusted source of recommendations. Charities don't tend to focus on the statistics of their work often preferring to focus on the emotions of the problem, perhaps reflecting the reality that naive, emotional people are the most likely to give to charity. Even when statistics are used they are like the feature lists for the gadgets, I feel no confidence that they reflect the whole picture.

While I have received some satisfaction from my donations, I do feel uncomfortable with the lack of confidence I have that my donations are actually causing an improvement, or whether they are directed towards genuine priorities. From my experiences in work I am familiar with the enormous waste within many organisations and how small steps of progress can be eradicated by later poor decision making.

What I really want is a kind of economics of suffering, a measurement of the various problems that I can apply my own weights and hypotheses to. I would like to see a comparison of the expected loss of life, and loss of quality of life, due to temperature rises, famines, AIDs, cancer etc. as well as the expected rate of return on my investment: have any of the charities I can give money to achieved progress or do they just maintain a status quo. How many charities have done the equivalent of eradicating small pox (i.e. solved a problem). Does anyone know of anything like this? I think it would be a much more valuable step than using rationality to empower the emotional arms race between charities.

I've had the same experiences re: passion and productivity. On your last comment:

"I think there is a real risk of having ones culture and community define goals for ourselves that are not actually what we want."

It's not clear to me what your concern is. You draw a distinction between cultural goals and values, and personal goals and values, but how would you be able to draw the line between the two? (What does it mean to feel something "deep down"?) And even if you could draw that distinction, why is it automatically bad to acquire cultural goals? What would be the consequences of pursuing these "incorrect" goals or values?

The most eye-opening article I've read recently, of possible relation to the subject, is a series on hunter-gatherer tribes by Peter Gray (see http://www.overcomingbias.com/2010/08/school-isnt-about-learning.html). While I'm skeptical of Gray's seemingly oversimplified depiction of hunter-gatherer tribes, the salient point of his argument is that there is a strong anti-authority norm in typical hunter-gather tribes. This leads me to think that the "natural" human psyche is resistant to authority, and conformity has to be "beaten in." Some of my own emotional conflicts have been due to a conflictedness about obeying authority; it seems to me that the "emotional mind" is more in line with these primal psychologies, which are exhibited more strongly in hunter-gatherer tribes than in modern society.

Certainly I would argue that following the emotional mind is not something everyone should do; it seems like there are a few niches in our society for the totally "free", who have the luxury of being able to make a living while largely ignoring the demand for individuals to find and conform to a specific externally-rewarded role in society. The positive and negative feedback individuals receive for following or ignoring their emotional minds, I would hypothesize, plays a large part in determining how much they ultimately listen to their emotional minds.