= 27d567bc2d48d9f40e9ccc20ea370b15)
orthonormal comments on Open Thread: July 2010, Part 2 - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (770)
Coherent Extrapolated Volition
This is exactly the argument religious people use to excuse any shortcomings of their personal FAI. Namely, their personal FAI knows better than you what's best for your AND everyone else.
What average people do is follow what is being taught here on LW. They decide based on their prior. Their probability estimates tell them that their FAI is likely to exist and make up excuses for extraordinary decisions based on the possible existence of it. That is, support their FAI while trying to inhibit other uFAI all in the best interest of the world at large.
Yahweh and the associated moral system are far from incomprehensible if you know the cultural context of the Israelites. It's a recognizably human morality, just a brutal one obsessed with purity of various sorts.
It is not about the moral system being incomprehensible but the acts of the FAI. Whenever something bad happens religious people excuse it with an argument based on "higher intention". This is the gist of what I wanted to highlight. The similarity between religious people and those true believers into the technological singularity and AI's. This is not to say it is the same. I'm not arguing about that. I'm saying that this might draw the same kind of people committing the same kind of atrocities. This is very dangerous.
If people don't like anything happening, i.e. don't understand it, it's claimed to be a means to an end that will ultimately benefit their extrapolated volition.
People are not going to claim this in public. But I know that there are people here on LW who are disposed to extensive violence if necessary.
To be clear, I do not doubt the possibilities talked about on LW. I'm not saying they are nonsense like the old religions. What I'm on about is that the ideas the SIAI is based on, while not being nonsense, are posed to draw the same fanatic fellowship and cause the same extreme decisions.
Ask yourself, wouldn't you fly a plane into a tower if that was the only way to disable Skynet? The difference between religion and the risk of uFAI makes it even more dangerous. This crowd is actually highly intelligent and their incentive based on more than fairy tales told by goatherders. And if dumb people are already able to commit large-scale atrocities based on such nonsense, what are a bunch of highly-intelligent and devoted geeks who see a tangible danger able and willing to do? More so as in this case the very same people who believe it are the ones who think they must act themselves because their God doesn't even exist yet.
Yes. I would also drop a nuke on New York if it were the only way to prevent global nuclear war. These are both extremely unlikely scenarios.
It's very correct to be suspicious of claims that the stakes are that high, given that irrational memes have a habit of postulating such high stakes. However, assuming thereby that the stakes never could actually be that high, regardless of the evidence, is another way of shooting yourself in the foot.