I don't really know the SIAI people, but I have the impression that they're not against AI at all. Sure, an unfriendly AI would be awful - but a friendly one would be awesome.
True. I know the SIAI people pretty well (I'm kind of one of them) and can confirm they agree. But they're pretty heavily against uFAI development, which is what I thought XiXiDu's quote was talking about.
And they probably think AI is inevitable, anyway.
Well... hopefully not, in a sense. SIAI's working to improve widespread knowledge of the need for Friendliness among AGI researchers. It's inevitable (barring a global catastrophe), but they're hoping to make FAI more inevitable than uFAI.
As someone who was a volunteered for SIAI at the Singularity Summit, a critique of SIAI could be to ask why we're letting people who aren't concerned about uFAI speak at our conferences and affiliate with our memes. I think there are good answers to that critique, but the critique itself is a pretty reasonable one. Most complains about SIAI are comparatively maddeningly irrational (in my own estimation).
A stronger criticism, I think, is why the only mention of friendliness at the Summit was some very veiled hints in Eliezer's speech. Again, I think there are good reasons, but not good reasons that a lot of people know, so I don't understand why people bring up other criticisms before this one.
Related: Taking ideas seriously
Let us say hypothetically you care about stopping people smoking.
You were going to donate $1000 dollars to givewell to save a life, instead you learn about an anti-tobacco campaign that is better. So you chose to donate $1000 dollars to a campaign to stop people smoking instead of donating it to a givewell charity to save an African's life. You justify this by expecting more people to live due to having stopped smoking (this probably isn't true, but for the sake of argument)
The consequences of donating to the anti-smoking campaign is that 1 person dies in africa and 20 live that would have died instead live all over the world.
Now you also have the choice of setting fire to many tobacco plantations, you estimate that the increased cost of cigarettes would save 20 lives but it will kill likely 1 guard worker. You are very intelligent so you think you can get away with it. There are no consequences to this action. You don't care much about the scorched earth or loss of profits.
If there are causes with payoff matrices like this, then it seems like a real world instance of the trolley problem. We are willing to cause loss of life due to inaction to achieve our goals but not cause loss of life due to action.
What should you do?
Killing someone is generally wrong but you are causing the death of someone in both cases. You either need to justify that leaving someone to die is ethically not the same as killing someone, or inure yourself that when you chose to spend $1000 dollars in a way that doesn't save a life, you are killing. Or ignore the whole thing.
This just puts me off being utilitarian to be honest.
Edit: To clarify, I am an easy going person, I don't like making life and death decisions. I would rather live and laugh, without worrying about things too much.
This confluence of ideas made me realise that we are making life and death decisions every time we spend $1000 dollars. I'm not sure where I will go from here.