What is actually the problem?

There are lots of problems. If I had to pick only one, it would be that you seem to think there is a single, simple problem that can be identified from this transcript.

If he kills himself, he hurts only himself. If he's violent toward other people, he can end up doing a lot more damage than that. He mentioned that one incident, but given his casual attitude toward it, there are probably more. It wouldn't surprise me if he was beating his girlfriend. Domestic assault (I call it domestic because it was against someone he lived with, even though housemate is not as usual a target as partner or child) is a huge huge huge warning flag. He had a bad day, and trouble sleeping, and suddenly someone else has to deal with the consequences of having a broken nose for the rest of their lives. The consequences for each of them are disproportionate, asymmetric. If he has another bad day, what next?

His girlfriend's life might be in danger.

Thank you, that was a very nice extension to the story. I should have included the scenario to make her belief relevant. I agree with you, assigning 100% probability is irrational in her case. But, if she is not rationally literate enough to express herself in fuzzy, non-binary way, I think she would maintain rationality through saying "Ceteris paribus, I prefer to be not locked in the same room with Cain because I believe he is a murder because I believe Adam was innocent" (ignoring ad hominem)

I was under the impression that the golden standard for rationality is falsifiability. However, I now understand that Eve is rational despite unfalsifiablity, because she remained Bayesian.

What if we were to take one step back and Adam didn't die. Eve claims that her believe pays rent because it could be falsified if Adam changed in character. In this scenario, I suppose that you would agree to say that Eve is still rational.

Now, I cannot formulate my arguments properly at the moment, but I think it is weird that Adam's death make Eve's belief irrational, as per:

So I do not believe a spaceship blips out of existence when it crosses the cosmological horizon of our expanding universe, even though the spaceship's existence has no further experimental consequences for me.

http://lesswrong.com/lw/ss/no_logical_positivist_i/

This belief pays no rent. It's unfalsifiable precisely because it's irrelevant - there is no prediction that Eve can make which would give different outcomes based on Adam's past behavior. The belief just doesn't matter.

Separately, if she assigns 0.0 probability to anything, she's probably not actually as rational as she claims.

So it would be better to work on computer security? Or on education, so that we raise fewer unfriendly natural intelligences?

Also, AI safety research benefits AI research in general and AI research in general benefits humanity. Again only marginal contributions?

So you think there's not much we can do about x-risk? What makes you think that? Or, alternatively, if you think that only few people who can do much good in x-risk mitigation, what properties enable them to do that?

Oh, and why do you consider AI safety a "theoretical [or] unlikely" problem?

Perhaps you'd do a lot more good with a slight reduction in shipping costs or tiny improvements in safety or enjoyment of some consumer product.

Perhaps you would also do more good by working in a slight increase in shipping costs.

In the likely case that your marginal contribution to x-risk doesn't save the world

So you think that other people could contribute much more to x-risk, so I should go into areas where I can have a lot of impact? Otherwise, if everyone says »I'll only have a small impact on x-risk. I'll do something else.«, nobody would work on x-risk. Are you trying to get a better justification for work on x-risk out of me? At the moment I only have this: x-risk is pretty important, because we don't want to go extinct (I don't want humanity to go extinct or into some worse state than today). Not many people are working on x-risk. Therefore I do work on x-risk, so that there are more people working on it. Now you will tell me that I should start using numbers.

the fact that you won't consider leaving Kagoshima is an indication that you aren't as fully committed as you claim

What did I claim about my degree of commitment? And yes, I know that I would be more effective at improving the state of humanity if I didn't have certain preferences about family and such.

Anyway, thanks for pushing me towards quantitative reasoning.

I have some questions on discounting. There are a lot, so I'm fine with comments that don't answer everything (although I'd appreciate it if they do!) I'm also interested in recommendations for a detailed intuitive discussion on discounting, ala EY on Bayes' Theorem.

• Why do people focus on hyperbolic and exponential? Aren't there other options?
• Is the primary difference between them the time consistency?
• Are any types of non-exponential discounting time-consistent?
• What would it mean to be an exponential discounter? Is it achievable, and if so how?
• What about different values for the exponent? Is there any way to distinguish between them? What would affect the choice?
• Does it make sense to have different discounting functions in different circumstances?
• Why should we discount in the first place?

On a personal level, my intuition is not to discount at all, i.e. my happiness in 50 years is worth exactly the same as my happiness in the present. I'll take $50 right now over $60 next year because I'm accounting for the possibility that I won't receive it, and because I won't have to plan for receiving it either. But if the choice is between receiving it in the mail tomorrow or in 50 years (assuming it's adjusted for inflation, I believe I'm equally likely to receive it in both cases, I don't need the money to survive, there are no opportunity costs, etc), then I don't see much of a difference.

• Is this irrational?
• Or is the purpose of discounting to reflect the fact that those assumptions I made won't generally hold?
• The strongest counterargument I can think of is that I might die and not be able to receive the benefits. My response is that if I die I won't be around to care (anthropic principle). Does that make sense? (The discussions I've seen seem to assume that the person will be alive at both timepoints in any case, so it's also possible this should just be put with the other assumptions.)
• If given the choice between something bad happening now and in 10 years, I'd rather go through it now (assume there are no permanent effects, I'll be equally prepared, I'll forget about the choice so anticipation doesn't play a role, etc). Does that mean I'm "negative discounting"? Is that irrational?
• I find that increasing the length of time I anticipate something (like buying a book I really want, and then deliberately not reading it for a year) usually increases the amount of happiness I can get from it. Is that a common experience? Could that explain any of my preferences?