The goal of development of FAI is reduction of its existential threat to near 0%, by mathematically proving stability and desirability of its preferences. It's OK, but it reminds me of zero-risk bias.
Excellent point. The goal of FAI should be to increase expected value, not to minimize risk.
If an evolutionary psychologist says that we expect something to be a widespread property of societies, it isn't a valid rebuttal to say, "But the Aché of South America didn't have that property." That's like laughing at the warnings on cigarettes because your grandmother smoked until she was 96.
Eliezer has a bias against evolution, and a bias against randomness, as exhibited in his series ending in Worse than Random, which is factually correct in the details, but misleading in the real world, as demonstrated by repeated times when his acolytes have used it to attack probabilistic search, probabilistic models, etc.
My take all along has been that something about evolution has caused it to reliably make the world a more complicated, more interesting, and better place; and evolution, with randomness, is the only process that can be trusted to continue this. Any attempt to control and direct the course of change will just lock in the values of the controller.
I see E's story about the moties as being one possible source of his bias against evolution, and hence against randomness.
A top ping-pong player told me that when he's playing ping-pong, time slows down. I've also heard this from a martial artist. Possibly extreme expertise and focus uses neural circuits in a way that provides a reaction faster than those circuits do when operating in the general-purpose mode. The back-of-envelope calculations would be based on observation of the general-purpose mode.
I think you're using a "spotlight of attention" model, which supposes that consciousness is like a spotlight that sweeps over the contents of the mind, illuminating only one spot at a time, but shining on all of it at one time or another.
I'm invoking a "special teams" metaphor, which is that the conscious mind is brought to bear only on a subset of problem types, and there are some parts (possibly large parts) of the mind that the spotlight never shines on. But I didn't present data to distinguish between these models; and in fact the cognitive architecture I described is more of a "spotlight of attention" model. (I think that's because it represented only symbolic information, of the type that the conscious mind deals with.)
The post may be misleading. The spotlight may shine on all the parts we care about at one time or another; your goals may all be accessible to you (although the top-level values and goals may be read-only). But there's a giant mass of subconscious learned skills and classifiers that you don't have access to, and information flows from them back up into "your" part of the mind without your being aware of it. A better (and in some ways opposed) metaphor might be that you're the president, but you have to implement everything by delegating it to a large bureaucracy.
SIAI seems poorly suited to generating interest in and concern for existential risk and may very well be lowering the prestige attached to investigating existential risk rather than raising the prestige attached to investigating existential risk.
Why? How would you do it differently?
View more: Next
No - expected value is important. If many successful FAI scenarios could result in negative value, then zero value (universal extinction) would be better.
We should put some thought into whether a negative-value universe is plausible, and what it would look like.