From that Future of Life conference: if self-driving cars take over and cut the death rate from car accidents from 32000 to 16000 per year, the makers won't get 16000 thank-you cards -- they'll get 16000 lawsuits.
I'm afraid I won't have time to give you more help. There's a short summary of each sequence under the link at the top of the page, so it won't take you forever to see the relevance.
EDIT: you're wondering elsewhere in the thread why you're not being well received. It's because your post doesn't make contact with what other people have thought on the topic.
Any AI that would do this is unFriendly. The vast majority of uFAIs have goals incompatible with human life but not in any way concerned with it. [...] Therefore there is little to fear in the way of being tortured by an AI.
That makes no sense. The uFAIs most likely to be created are not drawn uniformly from the space of possible uFAIs. You need to argue that none of the uFAIs which are likely to be created will be interested in humans, not that few of all possible uFAIs will.
it's unclear to me how the category of "evolutionary restrictions" could apply to rationality techniques. Suggestions?
Not sure if this simple example is what you had in mind, but -- evolution wasn't capable of making us grow nice smooth erasable surfaces on our bodies, together with ink-secreting glands in our index fingers, so we couldn't evolve the excellent rationality technique of writing things down to remember them. So when writing was invented, the inventor was entitled to say "my invention passes the EOC because of the "evolutionary restrictions" clause".
O.K, demonstrate that the idea of deterrent exists somewhere within their brains.
Evolutionary game theory and punishment of defectors is all the answer you need. You want me to point at a deterrent region, somewhere to the left of Broca's?
You say that science is useful for truths about the universe, whereas morality is useful for truths useful only to those interested in acting morally. It sounds like you agree with Harris that morality is a subcategory of science.
...something can be good science without in any way being moral that Sam Harris would recog
If you claim that evolutionary reasons are a person's 'true preferences'
No, of course not. It's still wrong to say that deterrent is nowhere in their brains.
Concerning the others:
Scientific inquiry percieves facts which are true and useful except for goals which run directly counter to science. Morality perceives 'facts' which are only useful to those who wish to follow a moral route.
I don't see what "goals which run directly counter to science" could mean. Even if you want to destroy all scientists, are you better off knowing some scienc...
Nevertheless, moral questions aren't (even potentially) empirical, since they're obviously seeking normative and not factual answers.
You can't go from an is to an ought. Nevertheless, some people go from the "well-being and suffering" idea to ideas like consequentialism and utilitarianism, and from there the only remaining questions are factual. Other people are prepared to see a factual basis for morality in neuroscience and game theory. These are regular topics of discussion on LW. So calling it "obvious" begs the whole question.
control over the lower level OS allows for significant performance gains
Even if you got a 10^6 speedup (you wouldn't), that gain is not compoundable. So it's irrelevant.
access to a comparatively simple OS and tool chain allows the AI to spread to other systems.
Only if those other systems are kind enough to run the O/S you want them to run.
The unstated assumption is that a non-negligible proportion of the difficulty in creating a self-optimising AI has to do with the compiler toolchain. I guess most people wouldn't agree with that. For one thing, even if the toolchain is a complicated tower of Babel, why isn't it good enough to just optimise one's source code at the top level? Isn't there a limit to how much you can gain by running on top of a perfect O/S?
(BTW the "tower of Babel" is a nice phrase which gets at the sense of unease associated with these long toolchains, (eg) Python - RPython - LLVM - ??? - electrons.)
Ok, but are we optimising the expected case or the worst case? If the former, then the probability of those things happening with no special steps against them is relevant. To take the easiest example: would postponing the "take over the universe" step for 300 years make a big difference in the expected amount of cosmic commons burned before takeover?
That page mentions "common sense" quite a bit. Meanwhile, this is the latest research in common sense and verbal ability.
I don't think it's useful to think about constructing priors in the abstract. If you think about concrete examples, you see lots of cases where a reasonable prior is easy to find (eg coin-tossing, and the typical breast-cancer diagnostic test example). That must leave some concrete examples where good priors are hard to find. What are they?
To be clear, the idea is not that trying to deliberately slow world economic growth would be a maximally effective use of EA resources and better than current top targets; this seems likely to have very small marginal effects, and many such courses are risky. The question is whether a good and virtuous person ought to avoid, or alternatively seize, any opportunities which come their way to help out on world economic growth.
It sounds like status quo bias. If growth was currently 2% higher, should the person then seize on growth-slowing opportunities?
On...
Don't forget that the goal in the Turing Test is not to appear intelligent, but to appear human. If an interrogator asks "what question would you ask in the Turing test?", and the answer is "uh, I don't know", then that is perfectly consistent with the responder being human. A smart interrogator won't jump to a conclusion.
"That which has happened before is less likely to happen again" (a reference to an old Overcoming Bias post I can't locate).
Good point. In fact, that is the type of environment which is required for the No Free Lunch theorems mentioned in the post to even be relevant. A typical interpretation in the evolutionary computing field would be that it's the type of environment where an anti-GA (a genetic algorithm which selects individuals with worse fitness) does better than a GA. There are good reasons to say that such environments can't occur for ...
...In the Soviet Union religion was marginalized for some 70 years, two generations grew up in the environment of state atheism, yet soon after the restrictions were relaxed, the Church has regained almost all of the lost ground. The situation was similar in the rest of the ex-Warsaw bloc (with less time under mandated atheism), and even in China, where the equilibrium was restored after the Cultural Revolution. The standard argument [bold added] for this happening is "but Communism was basically a religion by another name", what with the various C
"loosing" is still incorrect.
In a sense, bookies could be interpreted as "money pumping" the public as a whole. But somehow, it turns out that any single individual will rarely be stupid enough to take both sides of the same bet from the same bookie, in spite of the fact that they're apparently irrational enough to be gambling in the first place.
Suggest making the link explicit with something like this: "in spite of the fact that they're apparently irrational enough to be part of that public in the first place."
I'm hoping in particular that someone used to feel this way—shutting down an impulse to praise someone else highly, or feeling that it was cultish to praise someone else highly—and then had some kind of epiphany after which it felt, not allowed, but rather, quite normal.
I think there is a necessary distinction between matter-of-fact praising someone highly, and engaging in various sucking-up behaviours such as echoing particular forms of words, or quoting-as-authority. The latter do leave an unpleasant taste and in those cases I can understand the "cult" reaction.
For small vices, it is perhaps more important to ask, "What works?"
http://en.wikipedia.org/wiki/Broken_windows_theory
This is a bit like the "look before you leap", "no, who hesitates is lost" game.
Also "the teacher smiled"? Damn your smugness, teacher!