I expect B8 is the major factor. Before social media, if you had a bad idea and two of your five close friends told you they didn’t think it was a good idea, you’d drop it. Now five random ‘friends’ will tell you how insightful you are and how blind everyone else is. You’ve publicly stated your belief in the idea and got social proof. That makes it that much harder to drop.
People individually don’t have more bad ideas than before, but there is much more selection pressure in favor of them.
We want to show that given any daemon, there is a smaller circuit that solves the problem.
Given any random circuit, you can not, in general, show whether it is the smallest circuit that produces the output it does. That's just Rice's theorem, right? So why would it be possible for a daemon?
But within every apple eater lies that hunger -- that deep abyssal craving -- for apples
Which an apple eater could perfectly satisfy without apples, if only they weren’t told ad nauseam that satisfying their cravings without apples is inferior and shameful, that apples are really much better and that they should strive for eating apples. This is a cultural problem, not a natural one.
Don't worry about not being able to convince Lubos Motl. His prior for being correct is way too high and impedes his ability to consider dissenting views seriously.
Given there are friendly human intelligences, what would have to be true about the universe in order for friendly AGI’s to be impossible?
A list that is probably vastly incomplete. It seems very likely that there have been vehicle attacks for as long as vehicles exist. What would be the odds of no one in the past 100 years, no angry spouse, disgruntled ex-employee or lunatic, having thought of taking revenge on the cruel world by ramming a vehicle into people? Wouldn’t a prior on the order of at least one such event per 1 million vehicles per year be more likely to yield correct predictions than 0, for events before, say, the year 2005?
In that triad the meta-contrarian is broadening the scope of the discussion. They address what actually matters, but that doesn’t change that the contrarian is correct (well, a better contrarian would point out the number of deaths due to Ebola is far less than any of those examples and Ebola doesn’t seem a likely candidate to evolve into a something causing an epidemic) and that the meta-contrarian has basically changed the subject.
Suppose the Manhattan project was currently in progress, meaning we somehow had the internet, mobile phones, etc. but not nuclear bombs. You are a smart physicist that keeps up with progress in many areas of physics and at some point you realize the possibility of a nuclear bomb. You also foresee the existential risk this poses.
You manage to convince a small group of people of this, but many people are skeptical and point out the technical hurdles that would need to be overcome, and political decisions that would need to be taken, for the existential risk ...
Could I summarize this as: deontological and virtue-ethical moral principles are often useful approximations to consequentialist reasoning, that are faster to apply to a situation and therefore often preferable as a guide to desired behavior?