Major update here.
The state of affairs regarding the SIAI and its underlying rationale and rules of operation are insufficiently clear.
Most of the arguments involve a few propositions and the use of probability and utility calculations to legitimate action. Here much is uncertain to an extent that I'm not able to judge any nested probability estimations. Even if you tell me, where is the data on which you base those estimations?
There seems to be an highly complicated framework of estimations to support and reinforce each other. I'm not sure how you call this in English, but in German I'd call that a castle in the air.
I know that what I'm saying may simply be due to a lack of knowledge and education, that is why I am inquiring about it. How many of you, who currently support the SIAI, are able to analyse the reasoning that led you to support the SIAI in the first place, or at least substantiate your estimations with other kinds of evidence than a coherent internal logic?
I can follow much of the reasoning and arguments on this site. But I'm currently unable to judge their overall credence. Are the conclusions justified? Is the coherent framework build around the SIAI based on firm ground? There seems to be no critical inspection or examination by a third party. There is no peer review. Yet people are willing to donate considerable amounts of money.
I'm concerned that, although consistently so, the SIAI and its supporters are updating on fictional evidence. This post is meant to inquire about the foundations of your basic premises. Are you creating models to treat subsequent models or are your propositions based on fact?
An example here is the use of the Many-worlds interpretation. Itself a logical implication, can it be used to make further inferences and estimations without additional evidence? MWI might be the only consistent non-magic interpretation of quantum mechanics. The problem here is that such conclusions are, I believe, widely considered not to be enough to base further speculations and estimations on. Isn't that similar to what you are doing when speculating about the possibility of superhuman AI and its consequences? What I'm trying to say here is that if the cornerstone of your argumentation, if one of your basic tenets is the likelihood of superhuman AI, although a valid speculation given what we know about reality, you are already in over your head with debt. Debt in the form of other kinds of evidence. Not to say that it is a false hypothesis, that it is not even wrong, but that you cannot base a whole movement and a huge framework of further inference and supportive argumentation on such premises, on ideas that are themselves not based on firm ground.
The gist of the matter is that a coherent and consistent framework of sound argumentation based on unsupported inference is nothing more than its description implies. It is fiction. Imagination allows for endless possibilities while scientific evidence provides hints of what might be possible and what impossible. Science does provide the ability to assess your data. Any hint that empirical criticism provides gives you new information on which you can build on. Not because it bears truth value but because it gives you an idea of what might be possible. An opportunity to try something. There’s that which seemingly fails or contradicts itself and that which seems to work and is consistent.
And that is my problem. Given my current educational background and knowledge I cannot differentiate LW between a consistent internal logic, i.e. imagination or fiction, and something which is sufficiently based on empirical criticism to provide a firm substantiation of the strong arguments for action that are proclaimed by the SIAI.
Further, do you have an explanation for the circumstance that Eliezer Yudkowsky is the only semi-popular person who's aware of something that might shatter the universe? Why is it that people like Vernor Vinge, Robin Hanson or Ray Kurzweil are not running amok using all their influence to convince people of the risks ahead, or at least give all they have to the SIAI? Why aren't Eric Drexler, Gary Drescher or AI researches like Marvin Minsky worried to the extent that they signal their support?
I'm talking to quite a few educated people outside this community. They do not doubt all those claims for no particular reason. Rather they tell me that there are too many open questions to focus on the possibilities depicted by the SIAI and to neglect other near-term risks that might wipe us out as well.
I believe that many people out there know a lot more than I do, so far, about related topics and yet they seem not to be nearly as concerned about the relevant issues than the average Less Wrong member. I could have named other people. That's besides the point though, it's not just Hanson or Vinge but everyone versus Eliezer Yudkowsky and some unknown followers. What about the other Bayesians out there? Are they simply not as literate as Eliezer Yudkowsky in the maths or maybe somehow teach but not use their own methods of reasoning and decision making?
What do you expect me to do, just believe Eliezer Yudkowsky? Like I believed so much in the past which made sense but turned out to be wrong? Maybe after a few years of study I'll know more.
...
2011-01-06: As this post received over 500 comments I am reluctant to delete it. But I feel that it is outdated and that I could do much better today. This post has however been slightly improved to account for some shortcomings but has not been completely rewritten, neither have its conclusions been changed. Please account for this when reading comments that were written before this update.
2012-08-04: A list of some of my critical posts can be found here: SIAI/lesswrong Critiques: Index
Physicists have something else, however, and that is domain expertise. As far as I am concerned, MWI is completely at odds with the spirit of relativity. There is no model of the world-splitting process that is relativistically invariant. Either you reexpress MWI in a form where there is no splitting, just self-contained histories each of which is internally relativistic, or you have locally propagating splitting at every point of spacetime in every branch, in which case you don't have "worlds" any more, you just have infinitely many copies of infinitely many infinitesimal patches of space-time which are glued together in some complicated way. You can't even talk about extended objects in this picture, because the ends are spacelike separated and there's no inherent connection between the state at one end and the state at the other end. It's a complete muddle, even before we try to recover the Born probabilities.
Rather than seeing MWI as the simple and elegant way to understand QM, I see it as an idea which in a way turns out to be too simple - which is another way of saying, naive or uninformed. Like Bohmian mechanics, conceptually it relies on a preferred frame.
The combination of quantum mechanics with special relativity yields quantum field theory. In quantum field theory, everything empirically meaningful is conceptually relativistic. In your calculations, you may employ entities (like wavefunctions evolving in time) which are dependent on a particular reference frame, but you can always do such calculations in a different frame. An example of a calculational output which is frame-independent would be the correlation function between two field operators at different points in space-time. By the time we reach the point of making predictions, that correlation function should only depend on the (relativistically invariant) space-time separation. But in order to calculate it, we may adopt a particular division into space and time, write down wavefunctions defined to exist on the constant-time hypersurfaces in that reference frame, and evolve them according to a Hamiltonian. These wavefunctions are only defined with respect to a particular reference frame and a particular set of hypersurfaces. Therefore, they are somehow an artefact of a particular coordinate system. But they are the sorts of objects in terms of which MWI is constructed.
The truly relativistic approach to QFT is the path integral, the sum over all field histories interpolating between conditions on an initial and a final hypersurface. These histories are objects which are defined independently of any particular coordinate system, because they are histories and not just instantaneous spacelike states. But then we no longer have an evolving superposition, we just have a "superposition" of histories which do not "split" or "join".
At any time, theoretical physics contains many ideas and research programs, and there are always a lot of them that are going nowhere. MWI has all the signs of an idea going nowhere. It doesn't advance the field in any way. Instead, as with Bohmian mechanics, what happens is that specific quantum theories are proposed (field theories, string theories), and then the Everettians, the Bohmians, and so on wheel out their interpretive apparatus, which they then "apply" to the latest theoretical advance. It's a parasitic relationship and it's a sign that in the long run this is a dead end.
I will provide an example of an idea which is more like what I would look for in an explanation of quantum theory. The real problem with quantum theory is the peculiar way its probabilities are obtained. You have complex numbers and negative quasiprobabilities and histories that cancel each other. The cancellation of possibilities makes no sense from the perspective of orthodox probability. If an outcome can come about in one way, the existence of a second way can only increase the probability of the outcome - according to probability theory and common sense. Yet in the double-slit experiment we have outcomes that are reduced in probability through "destructive interference". That is what we need to explain.
There is a long history of speculation that maybe the peculiar structure of quantum probabilities can be obtained by somehow conditioning on the future as well as on the past, or by having causality working backwards as well as forwards in time. No-one has ever managed to derive QM this way, but many people have talked about it.
In string theory, there are light degrees of freedom, and heavy degrees of freedom. The latter correspond to the higher (more energetic) excitations of the string, though we should not expect that strings are fundamental in the full theory. In any case, these heavy excitations should cause space to be very strongly curved. So, what if the heavy degrees of freedom create a non-time-orientable topology on the Planck scale, giving rise to temporally bidirectional constraints on causality, and then the light strings interact (lightly) with that background, and quantum-probability effects are the indirect manifestation of that deeper causal structure, which has nonlocal correlations in space and time?
That's an idea I had during my string studies. It is not likely to be right, because it's just an idea. But it is an explanation which is intrinsically connected to the developing edge of theoretical physics, rather than a prefabricated explanation which is then applied in a one-size-fits-all fashion to any quantum theory. It would be an intrinsically string-theoretic derivation of QM. That is the sort of explanation for QM that I find plausible, for the reason that everything deep in physics is deeply connected to every other deep thing.
Huh? This is what I've always¹ taken MWI in a relativistic context...
Anyway, I'll have to read the works by 't Hooft when I have time. They look quite interesting.