As I get closer to really understanding the theory (that is, knowing all the mathematical detail), I find myself becoming more and more of a string-theory fundamentalist, so I should attempt a response to this.
First, some basic quantum field theory. In quantum field theory, particles and fields are generally regarded as complementary. Particles are energy quanta from field modes, and field states are equivalent to certain superpositions of particle states. Every particle in the Standard Model corresponds to a field, and vice versa. But if you have to pick one description as more fundamental, it's probably the field, since the fundamental variables in the basic equation are field variables.
The classical theory of gravity is general relativity, and it's a field theory, so as a quantum field theory there should be gravitons. But as I just explained, a graviton state is a superposition of field states and vice versa.
The principal reason for the conceptual churn in quantum gravity, at least before string theory, was that quantum general relativity is nonrenormalizable, i.e. you cannot perform calculations with it. Renormalization (in its simplest, original forms) is a procedure for dealing with the divergent (infinite) integrals showing up in "perturbative quantum field theory", which is where the particle states show up. An elementary particle state may be regarded as a certain superposition of states of the noninteracting field. Perturbative approaches to interacting field theory are possible if the interaction is weak enough that the actual dynamics may be regarded as a correction to the free field dynamics. Feynman diagrams are a notation for these corrections which also symbolize some of the space-time processes appearing in the superpositions. But even in perturbative field theory, you still get infinite values for some of these corrections. You will be summing amplitudes over a variety of possible processes, and sometimes these sums just don't converge. Renormalization involves the assumption of fictitious processes whose integrals sum to infinities which cancel out the divergent integrals.
This is one of those concepts which is ubiquitous in popular exposition of physics but which sounds bizarre and which the expositors even describe as bizarre. However, the simplest way to understand the meaning of renormalization is to suppose that the theory you are working with (e.g. quantum electrodynamics) is incomplete, and that the complete theory (the complete set of interactions, which you do not know, and which may include completely unknown fields) is not divergent and undefined. The renormalization procedure is then a way of incorporating this hypothesis that a better theory has no divergences because other interactions cancel the infinities arising from the incomplete theory.
This is a practical process only because, in a renormalizable theory, only a finite number of parameters are required to define the divergence-cancelling "counterterms" at every order of perturbation theory (order corresponds to the complexity of the virtual processes entering into the amplitude sum over histories). When quantum gravity is said to be nonrenormalizable, what is meant is that an infinite number of parameters are needed to construct all the counterterms. In a renormalizable theory, the finite number of parameters are actually set by empirical values like observed mass and charge. But the infinity of parameters for quantum gravity renormalization render it unpredictive.
However, renormalization arose within a particular approach to making QFT calculations - the approximations of perturbation theory. One can instead aim to solve an interacting quantum field theory exactly. Just because perturbation theory breaks down, doesn't mean that the full unapproximated theory is undefined - just that you can't approximate the solution using that method.
So, back to the Standard Model, gravity, and fundamental theory. That gravity is not part of the Standard Model is an arguable proposition. The Standard Model is a codification of what is observed. The Standard Model without gravity consists of various matter fields and various force fields in interaction (and a Higgs field to provide mass; this is a hack and the Higgs has not been seen). All these interactions are renormalizable and so you can calculate with them. But gravity is also observed, of course, and we can write down an equation expressing the idea that the other Standard Model fields interact with a gravitational field as Einstein described it. We just can't do much with it as a quantum field theory, because gravity is nonrenormalizable. The Standard Model including gravity, however, as codified in that equation, really ought to be regarded as our empirically best theory of physics.
Beyond this, there have really been two directions of inquiry. One is unification of everything but gravity in various field theories with large symmetries which are then broken in some way; the other direction of inquiry focuses on quantum gravity, and has seen a great deal of formal invention and philosophical thrashing about, and even one solid result (Hawking radiation). The first path is the one pursued by particle physicists and quantum field theorists, and eventually they constructed unified field theories which did include gravity in a master symmetry; first supergravity, and then finally the string theories. The second path has been pursued by a lot of other people and does not have the methodological unity of the first path. The most substantial criticism of the first-path philosophy from the second path is that quantum field theory generally assumes flat Minkowski space and does not represent many effects from general relativity, such as black holes. However, this criticism has lost a lot of its impact across the years as physicists have learned to apply quantum field theory in curved spacetime backgrounds. There is still the objection that the philosophy of general relativity, in which matter and geometry react to each other, is lost in a philosophy which treats geometry as a classical "background" to quantum fields, but again, I see a slow remediation of this deficiency on the quantum side.
Now to string theory. Mathematically string theory is a work in progress. Tremendous progress has been made, especially when the various string theories of the 1980s were discovered in the 1990s to be intertransformable, but we do not yet know the fundamental variables of the unified string theory, in the way that the undamental variables of quantum field theory are the field degrees of freedom.
It is clear that string theory can produce something qualitatively like the Standard Model - though intriguingly, no-one has yet found a string "vacuum" which has exactly all the qualitative features of the Standard Model (dimensionality, number of generations, particular symmetries and so forth). The problem is that string theory can also describe a large number of other physical outcomes. It depends on what you do with all those dimensions. Until the previous decade (2000s) the mainstream of string research was always hoping that one geometric configuration would end up preferred, with everything else dynamically unstable, and that this would finally produce a falsifiable prediction. However, the anthropic approach crept into string theory in recent years, producing controversy. It seems clear that the answer to the question of string theory's ground-state geometry - is there just one, or are there many, and is the anthropic approach viable - depends on cosmology. The evolution from unstable geometries to stable ones occurs on cosmological scales, and if there are radically different geometries at different places in the universe, that too will only be understood once string cosmology is more advanced.
I think I'll stop there, since there are so many different issues which get raised in a discussion like this, and just field specific questions if there are any. I know I certainly haven't responded to everything in Tom's post, but I hope this clarifies a few things.
Thank you for this excellent summary of some of the technical details. I disagree on a few points, but most of this is a great explanation of how some of this works.
- Eliezer Yudkowsky, Collapse Postulates
In the olden days of physics, circa 1900, many prominent physicists believed in a substance known as aether. The principle was simple: Maxwell's equations of electromagnetism had shown that light was a wave, and light followed many of the same equations as sound waves and water waves. However, every other kind of wave- sound waves, water waves, waves in springs- needs some sort of medium for its transmission. A "wave" is not really a physical object; it is just a disturbance of some other substance. For instance, if you throw a rock into a pond, you cannot pluck the waves out of the pond and take them home with you in your backpack, because the "waves" are just peaks and troughs in the puddle of water (the medium). Hence, there should be some sort of medium for light waves, and the physicists named this medium "aether".
However, difficulties soon developed. If you have a jar, you can pump the air out of the jar, and then the jar will no longer transmit sound, demonstrating that the wave medium (the air) has been removed. But, there was no way to remove the aether from a jar; no matter what the experimentalists did, you could still shine light through it. There was, in fact, no way of detecting, altering, or experimenting with aether at all. Physicists knew that aether must be unlike all other matter, because it could apparently pass through closed containers made of any substance. And finally, the Michelson-Morely experiment showed that the "aether" was always stationary relative to Earth, even though the Earth changed direction every six months as it moved about in its orbit! Shortly thereafter, the inconsistencies were resolved with Albert Einstein's Theory of Special Relativity, and everyone realized that aether was imaginary.
Shortly thereafter, during the 20th century, physicists discovered two new forces of nature: the strong nuclear force and the weak nuclear force. These two forces, as well as electromagnetism, could be described very well on the quantum level: they were created by the influence of mediator particles called (respectively) gluons, W and Z bosons, and photons, and these particles obeyed the laws of quantum mechanics just like electrons and mesons did. The description of these three forces, as well as the particles they act upon, has been neatly unified in a theory of physics known as the Standard Model, which has been our best known description of the universe for thirty years now.
However, gravity is not a part of this model. Making an analogy to the other forces, physicists have proposed a mediator particle known as the "graviton". The graviton is thought to be similar to the photon, the gluon, and the W and Z bosons, except that it is massless and has spin 2. I posit that the "graviton" is essentially the same theory as the "aether": a misguided attempt to explain something by reference to similar-seeming things that were explained in the same way. Consider the following facts:
And, with reference to the graviton itself:
So, what's really going on here? I don't know. I'm not Albert Einstein. But I suspect it will take someone like him- someone brilliant, very good at physics, yet largely outside the academic system- to resolve this mess, and tell us what's really happening.