The reason he does not like the term is that, as pointed out before, "emergence" is not an explanation of anything. However, it is an observational phenomenon: when you get a lot of simple things together, they combine in ways one could not foresee and the resulting entities behave by the rules not constructable from (but reducible to) those of the simple constituents. When you combine a lot of simple molecules, you get a solid, a liquid or a gas with the properties you generally cannot infer without observing them first. When you get a group of people together, they start interacting in apriori unpredictable ways as they form a group. Once you observe the group behavior, you can often reduce it to that of its constituents, but a useful description is generally not in terms of the constituents, but in terms of the collective. For example, in thermodynamics people use gas laws and other macroscopic laws instead of the Newton's laws.

I am guessing that one reason that the (friendly) machine intelligence problem is so hard is that intelligence is an emergent property: once you understand it, you can reduce it to interactions between neurons, but you cannot infer it from such interactions. And what's more, it's several layers above, given that intelligence evolved long after simpler neural processes got established.

Thus what MIRI is doing is studying the laws of an emergent structure (AI) without being able to observe the structure first, since it does not exist, yet. This is like trying to deduce the behavior of a bee hive by studying single cells. Even if you come up with some new "emergent" laws, it may well end up being more like a tree than a hive.