Greetings everyone.
I am feeling somewhat lethargic at the moment having just gotten off work, but I am pleased to see such a dedicated set of individuals who take the time to debate such a variety of topics and engage in rational discourse. Self-critique is important (love the name; Less Wrong).
As far as I am concerned everything we think we know is wrong. There is only "less wrong." Some things we have a pretty good grasp on and may only be .0000001% wrong. But I have to wonder just how many things actually fall into that category and how much of it is "wishful thinking" or hubris on our part to think that we know more than we actually do.
MTF
I am glad that there are people other than myself who find the notion of "emergent phenomena" to be code word for "magic" or "ignorance." Quite frankly I am mystified by how learned people, presumably taught the scientific method, could be so enamored with so called "emergent phenomena" but then I recall that practically the whole discipline of modern cosmology is little more than dressed up religion (anyone here know what possible cosmological consequences "Dark Energy" or "Dark Matter" have). But in that vein it bares mentioning that if it is impossible to discriminate between two things, then they are logically identical. Magic = achieving work through methods that do not involve "work effecting activity." Emergent phenomena = quality achieving methods that do not involve "quality achieving activity."
In order to deal with the topic of Emergent Phenomena one is required to comprehend what is called "Supervenience." I will leave it to the readers to explore Supervenience at their own discretion. (Standford encyclopedia of philosophy has a good article on it). Suffice it to say that some people have managed to convince themselves that a form of Supervenience can exist whereby "Top down causation" can occur.
Strong claims to emergent phenomena (which are essentially arguments regarding causation) require that the Supervenient qualities occur out of nowhere. This would be like in real life saying that Kinetic Energy occurs without there being any potential energy. Potential energy is not itself energy; hence the POTENTIAL part of its name. So can anyone point out why we "assume" the existence of potential energy in matter but that "potential intelligence" is not present in neurons?
Weak claims to emergent phenomena (which are essentially classification arguments) reference sets which include all objects in the universe (as Eliezer has pointed out), and as such are completely devoid of explanatory powers. Technically all things in the universe are "emergent properties" of physics. But we could potentially explain everything in the universe "if we had perfect knowledge of physics." We would need something like Laplace's daemon, but with perfect predictability we could derive neurochemistry. Once you have perfect neurochemistry knowledge, you can derive economics or any other set of behaviors you wish.
Example: When two triangles are brought together a square is the result (technically a quadrilateral, but I am keeping it simple). There is no property of triangles that is equivalent with "squareness." So "squareness" can be said to be an emergent property of triangles. Right now every person reading this should realize that something fishy is going on. Each triangle has "potential squareness" as part of its "list of qualities that it possesses." So when two triangles are brought together you get a square. You do not get something for nothing. The only difference between the triangles in this example and everything described as "emergent" is the degree of complexity.
The "proper way" Emergent Property is supposed to be used is when you have a universe of discourse that is random and note that "small sections of apparent order" occur. These properties are then "emergent" as they are not connected with any law that the universe itself possesses. The problem is that any application of this to a level of reality short of applying it to reality itself is entirely inconsistent with all non-paraconsistent logics.
A truly random "thing" does not exist. Such a "thing" could not interact with any thing we are aware of, since any interaction involves a two-way quality exchange/copy/removal,etc. Does anyone here have any idea what "mono-interactive interaction" looks like? The only way to achieve that is to have a "nothing" do something.
The very definition of "Nothing" is that it lacks any and all qualities. No qualities means no abilities. No abilities means "doing" is impossible. Nothing does not result in a change in anything.
Just because you are not aware of all the causal factors DOES NOT mean that there do not exist any causal factors. You can use this for any possible "emergent phenomena." Just because you are not aware of selection mechanisms for quantum physics does not mean that there are no selection mechanisms. Repeat ad nauseum.
MTF
Our best understanding of the nature of the "simulation" we call reality has this concept we call "cause and effect" in place. So when something happens it has non-zero (though nigh infinitely small) effects on everything else in existence (progressively smaller effect with each degree of separation).
The effect that affecting 3^^^3 things (regardless of type or classification) has on other things (even if the individual effects of affecting one thing would be extremely small) would be non-trivial (enormously large even after a positively ludicrous degree of separations).
Once you consider the level of effect that this would have on the whole "simulation" you are forced to consider basically all possible futures. You have nigh-infinite good (when these things are removed/effected you end up with utopia and a range of all possible net benefits for the whole of the simulation) and nigh-infinite penalty (when these things are removed/effected you end up with hell and a range of all possible net losses for the whole of the simulation). I cannot foresee how an AI can possibly have enough processing power to overcome the vagary being unable to predict all possible futures following the event.
Moreover, I personally balk at the assumption of that level of responsibility. It is for the same reason that I balk at time travel scenarios. I refuse to be responsible for whatever changes are wrought across all of reality (which in sum become quite large when you consider a Vast possibly infinite universe regardless of how "small" the initial event seems).
Also does the probability assignment take into account the likelihood of the actor in question approaching you? Assuming there are 3^^^3 people (minds), then surely the probability assignment of approaching you specifically must be adjusted accordingly. I understand that "somebody has to be approached," but surely no one here is willing to contend that any of us have traits which are so exceptional that they cannot be found inside of a population which is 3^^^3 in size?