Stupid Questions Thread - January 2014

RomeoStevens

Haven't had one of these for awhile. This thread is for questions or comments that you've felt silly about not knowing/understanding. Let's try to exchange info that seems obvious, knowing that due to the illusion of transparency it really isn't so obvious!

If it turns out that the whole MIRI/LessWrong memeplex is massively confused, what would that look like?

Note that in the late 19th century, many leading intellectuals followed a scientific/rationalist/atheist/utopian philosophy, socialism, which later turned out to be a horrible way to arrange society. See my article on this. (And it's not good enough to say that we're really rational, scientific, altruist, utilitarian, etc, in contrast to those people -- they thought the same.)

So, how might we find that all these ideas are massively wrong?

By their degree of similarity to ancient religious mythological and sympathetic magic forms with the nouns swapped out.

14JoshuaFox12y

I think the whole MIRI/LessWrong memeplex is not massively confused. But conditional on it turning out to be very very wrong, here is my answer: A. MIRI 1. The future does indeed take radical new directions, but these directions are nothing remotely like the hard-takeoff de-novo-AI intelligence explosion which MIRI now treats as the max-prob scenario. Any sci-fi fan can imagine lots of weird futures, and maybe some other one will actually emerge. 2. MIRI's AI work turns out to trigger a massive negative outcome -- either the UFAI explosion they are trying to avoid, or something else almost as bad. This may result from fundamental mistakes in understanding, or because of some minor bug. 3. It turns out that the UFAI explosion really is the risk, but that MIRI's AI work is just the wrong direction; e.g., it turns out that that building a community of AIs in rough power balance; or experimenting by trial-and-error with nascent AGIs is the right solution. B. CfAR 1. It turns out that the whole CfAR methodology is far inferior to instrumental outcomes than, say, Mormonism. Of course, CfAR would say that if another approach is instrumentally better, they would adopt it. But if they only find this out years down the road, this could be a massive failure scenario. 2. It turns out that epistemologically non-rational techniques are instrumentally valuable. Cf. Mormonism. Again, CfAR knows this, but in this failure scenario, they fail to reconcile the differences between the two types of rationality they are trying for. Again, I think that the above scenarios are not likely, but they're my best guess at what "massively wrong" would look like.

14adbge12y

A few that come to mind: * Some religious framework being basically correct. Humans having souls, an afterlife, etc. * Antinatalism as the correct moral framework. * Romantic ideas of the ancestral environment are correct and what feels like progress is actually things getting worse. * The danger of existential risk peaked with the cold war and further technological advances will only hasten the decline.