= 27d567bc2d48d9f40e9ccc20ea370b15)
SforSingularity comments on Optimal Strategies for Reducing Existential Risk - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (21)
Also, suppose that there are and will be 1000 singularitarian activists who can, together, increase the probability of a positive singularity outcome from 0.1 to 0.2, and you are average amongst them. The benefit that accrues to you if you spend time working with the singularitarian movement is then delta U * 0.1/10,000 = 10^(-5) delta U, where delta U is the utility difference between the expected utility of the life you will live conditional upon existential disaster (which won't occur for quite a while - at least 15 years from today) and the utility of the life you will live conditional upon a positive singularity outcome.
I doubt that anyone really has a utility function that supports a delta U of 100,000 times the typical utility differences in our everyday lives, e.g. 100,000 times the utility difference of spending money on a nice house, an expensive family, etc. Therefore the goodness of a post positive singularity outcome cannot incentivize the individual to bring it about, to the singularitarian movement has to rely upon people whose personal notion of goodness comes from being the kind of person who puts others before themselves, even in the face of criticism and ostracism from those others.
That is, unless there is some kind of reward/punishment precommitment going on.
While adopting a virtue ethic of being the sort of person who works against existential risk may result in ostracism IF you reveal it, if we assume that ostracism hurts efforts to reduce that risk then the rational thing for such a person to do would be to keep it to themselves.
But yes, it may happen that it would be rational to bring up such issues, get one (important?) person involved and motivated and simultaneously ostracize yourself from everyone else. Then you would need to be a person who cared more about others' wellbeing than what those people think of you. Which, IMHO, is pretty damn cool.
The larger problem is that people close to one - one's partner, parents, close friends - will all find out sooner or later; indeed attempting to hide it is probably even worse as it erodes trust.
it isn't cool if everyone ostracizes you and your life sucks whilst you work to save everyone, and then afterwards you get no acknowledgment; at least not in my book, especially if the problem is so large that the incremental reduction in risk you can achieve is very very small.
But in reality, I think that there are third options, side benefits to being involved in the risk-reduction movement (the other people in the movement are nice and smart, which means that they are great to be friends with, and they influence you positively, it provides personal motivation beyond what you would normally have, and if you are good the incremental reduction in risk you can achieve is large enough that you make a substantial improvement to your own prospects), so actually I think that being a risk-reducer is a personal gain, at least the way the current situation is.
If the situation changed so that it was a heavy personal loss (e.g. you could maximally reduce risk by sacrificing your life or risking a serious probability of that for the cause) then I would want to heavily advocate incentivization in some form; otherwise, a lot of people will drop away from the movement (not necessarily me, though I would have to do some soul-searching).
Though they end up being small factors in my own considerations, I like the mention of the side benefits of being part of such a group.
You appear to assume that rationalists are selfish? Or that our "real selves" are exclusively sub-deliberative systems that can't multiply benefits to others?