Is there a proof that it's possible to prove Friendliness?

I wonder what SI would do next if they could prove that friendly AI was not possible. For example if it could be shown that value drift was inevitable and that utility-functions are unstable under recursive self-improvement.

No. There's also no proof that it's possible to prove that P!=NP, and for the Friendliness problem it's much, much less clear what the problem even means. You aren't entitled to that particular proof, it's not expected to be available until it's not needed anymore. (Many difficult problems get solved or almost solved without a proof of them being solvable appearing in the interim.)

So in Eliezer's meta-ethics he talks about the abstract computation called "right", whereas in e.g. CEV he talks about stuff like reflective endorsement. So in other words in one place he's talking about goodness as a formal cause and in another he's talking about goodness as a final cause. Does he argue anywhere that these should be expected to be the same thing?

Not explicitly. He does in various places talk about why alternative considerations of abstract 'rightness' - some sort of objective morality or something - are absurd. He does give some details on his reductionist moral realism about the place but I don't recall where.

Incidentally I haven't seen Eliezer talk about formal or final causes about anything, ever. (And they don't seem to be especially useful concepts to me.)

Short answer is that they're nice people, and they understand that power corrupts, so they can't even rationalize wanting to be king of the universe for altruistic reasons.

Also, a post-Singularity future will probably (hopefully) be absolutely fantastic for everyone, so it doesn't matter whether you selfishly get the AI to prefer you or not.

Game theory. If different groups compete in building a "friendly" AI that respects only their personal extrapolated coherent violation (extrapolated sensible desires) then cooperation is no longer an option because the other teams have become "the enemy". I have a value system that is substantially different from Eliezer's. I don't want a friendly AI that is created in some researcher's personal image (except, of course, if it's created based on my ideals). This means that we have to sabotage each other's work to prevent the other researchers to get to friendly AI first. This is because the moment somebody reaches "friendly" AI the game is over and all parties except for one lose. And if we get uFAI everybody loses.

That's a real problem though. If different fractions in friendly AI research have to destructively compete with each other, then the probability of unfriendly AI will increase. That's real bad. From a game theory perspective all FAI researchers agree that any version of FAI is preferable to uFAI, and yet they're working towards a future where uFAI is becoming more and more likely! Luckily, if the FAI researchers take the coherent extrapolated violation of all of humanity the problem disappears. All FAI researchers can work to a common goal that will fairly represent all of humanity, not some specific researcher's version of "FAI". It also removes the problem of different morals/values. Some people believe that we should look at total utility, other people believe we should consider only average utility. Some people believe abstract values matter, some people believe consequences of actions matter most. Here too the solution of an AI that looks at a representative set of all human values is the solution that all people can agree on as most "fair". Cooperation beats defection.

If Luke were to attempt to create a LukeFriendlyAI he knows he's defecting from the game theoretical optimal strategy and thereby increasing the probability of a world with uFAI. If Luke is aware of this and chooses to continue on that course anyway then he's just become another uFAI researcher who actively participates in the destruction of the human species (to put it dramatically).

We can't force all AI programmers to focus on the FAI route. We can try to raise the sanity waterline and try to explain to AI researchers that the optimal (game theoretically speaking) strategy is the one we ought to pursue because it's most likely to lead to a fair FAI based on all of our human values. We just have to cooperate, despite differences in beliefs and moral values. CEV is the way to accomplish that because it doesn't privilege the AI researchers who write the code.

The good guys do not write an AI which values a bag of things that the programmers think are good ideas, like libertarianism or socialism or making people happy or whatever. There were multiple Overcoming Bias sequences about this one point, like the Fake Utility Function sequence and the sequence on metaethics. It is dealt with at length in the document Coherent Extrapolated Volition. It is the first thing, the last thing, and the middle thing that I say about Friendly AI.

...

The good guys do not directly impress their personal values onto a Friendly AI.

http://lesswrong.com/lw/wp/what_i_think_if_not_why/

The rest of your question has the same answer as "why is anyone altruist to begin with", I think.

I for one welcome our new singularitarian overlords!

I think it would be significantly easier to make FAI than LukeFreindly AI: for the latter, you need to do most of the work involved in the former, but also work out how to get the AI to find you (and not accidentally be freindly to someone else).

If it turns out that there's a lot of coherance in human values, FAI will resemble LukeFreindlyAI quite closely anyway.

Right now, and for the foreseeable future, SIAI doesn't have the funds to actually create FAI. All they're doing is creating a theory for friendliness, which can be used when someone else has the technology to create AI. And of course, nobody else is going to use the code if it focuses on SIAI.

Lots of incorrect answers in other replies to this one. The real answer is that, from Luke's perspective, creating Luke-friendly AI and becoming king of the universe isn't much better than creating regular friendly AI and getting the same share of the universe as any other human. Because it turns out, after the first thousand galaxies worth of resources and trillion trillion millenia of lifespan, you hit such diminishing returns that having another seven-billion times as many resources isn't a big deal.

This isn't true for every value - he might assign value to certain things not existing, like powerful people besides him, which other people want to exist. And that last factor of seven billion is worth something. But these are tiny differences in value, utterly dwarfed by the reduced AI-creation success-rate that would happen if the programmers got into a flamewar over who should be king.

What's stopping LukeProg from appointing himself king of the universe?

Personal abhorrence at the thought, and lack of AI programming abilities. :)

(But, your question deserves a more serious answer than this.)

Not an answer, but a solution:

You know what they say the modern version of Pascal's Wager is? Sucking up to as many Transhumanists as possible, just in case one of them turns into God. -- Julie from Crystal Nights by Greg Egan

:-p

I should hope most intelligent people realize they just want to be king of their own sensory inputs.

Now that I understand your question better, here's my answer:

Let's say the engineers decide to make the AI respect only their values. But if they were the sort of people who were likely to do that, no one would donate money to them. They could offer to make the AI respect the values of themselves and their donors, but that would alienate everyone else and make the lives of themselves and their donors difficult. The species boundary between humans and other living beings is a natural place to stop expanding the circle of enfranchised agents.

I've been here less than thirty days. Why does my total karma sometimes but not always show a different number from my karma from the last 30 days?

One of the numbers updates a couple minutes before the other one does. I forget which.

I've been here less than thirty days. Why does my total karma sometimes but not always show a different number from my karma from the last 30 days?

Presumably because the respective caches are recalculated at different intervals.

I think one possible "just so" explanation is:

Humans find symmetry more beautiful than asymmetry. Flowers are symmetrical.

Standard caveats: There are more details, and that's not a complete explanation, but it might prove a good starting point to look into if you're curious about the explanation.

Many flowers are optimized for being easily found by insects, who don't have particularly good eyesight. To stick out from their surroundings, they can use bright unnatural colors (i.e. not green or brown), unusual patterns (concentric circles is a popular one), have a large surface, etc.

Also, flowers are often quite short-lived, and thus mostly undamaged; we find smoothness and symmetry attractive (for evolutionary reasons - they're signs of health in a human).

In addition, humans select flowers that they themselves find pretty to place in gardens and the like, so when you think of "flowers", the pretty varieties are more likely to come to mind than the less attractive ones (like say that of the plane tree, or of many species of grass - many flowers are also prettier if you look at them in the UltraViolet.). If you take a walk in the woods, most plants you encounter won't have flowers you'll find that pretty; ugly or unremarkable flowers may not even register in your mind as "flowers".

Is it flowers in particular that puzzle you? Or is it more generally the fact that humans are wired so as to find anything at all beautiful?

Check out this page. Additionally, at the end of each post, there is a link labeled "Article Navigation". Click that, and it will open links to the previous and next post by the author.

There is a draft of a suggested reading order.

As I understand it, the sequences of LessWrong more or less grew out of prior writings by Eliezer, especially out of his posts at Overcoming Bias, so, there isn't a definitive first post.

Devise and execute a highly precise ritual wherein you invoke the optimal decision theory. That's my stopgap solution.

Strictly speaking, this question may be a bit tangential to LessWrong, but this was never supposed to be a exclusive thread.

The answer will depend on a lot of things, mostly specific to you personally. Bodies differ. Even your own single body changes over the course of time. You have certain specific goals and values, and certain constraints.

Maybe what you're really looking for is recommendations for a proper physical fitness forum, which is relatively free of pseudoscience and what they call "woo." I can't advise, myself, but I'm sure that some LessWrongians can.

Presumably once AGI becomes smarter than humans, it will develop goals of some kind, whether we want it or not. Might as well try to influence them.

In addition to the post Vladimir linked, see also this paper.

This idea is called "Oracle AI"; see this post and its dependencies for some reasons why it's probably a bad idea.

what do total utilitarians and average utilitarians actually mean when they're talking about the sum or the average of several utility functions?

They don't know. In most cases, they just sort of wave their hands. You can combine utility functions, but "sum" and "average" do not uniquely identify methods for doing so, and no method identified so far has seemed uniquely compelling.

If two possible futures have different numbers of people, those will be subject to different affine transforms, so the utility function as a whole will have been transformed in a non-affine way. See repugnant conclusion for a concrete example.

See here.

This is actually an open problem in utilitarianism; there were some posts recently looking to bargaining between agents as a solution, but I can't find them at the moment, and in any case that's not a mainstream LW conclusion.

There isn't a right answer (I think), but some ways of comparing are better than others. Stuart Armstrong is working on some of this stuff, as he mentions here.

Do you actually do this - "Oh, not P! I must be the pope." - or do you just notice this - "Not P, so everything's true. Where do I go from here?".

If you want to know why you shouldn't do this it's because you never really learn not P, you just learn evidence against P which you should update with Bayes' rule. If you want to understand this process more intuitively (and you've already read the sequences and are still confused), I would recommend this short tutorial or studying belief propagation in Bayesian networks, for which I don't know a great source for the intuitions behind, but units 3 and 4 of the online Stanford AI class might help.

To the (mostly social) extent that concepts were useful to your ancestors, one is going to lead to better decisions than the other, and so you should expect to have evolved the latter intuition. (You trust two friends, and then one of them tells you the other is lying- you feel some consternation of the first kind, but then you start trying to figure out which one is trustworthy.)

The reason is that you don't believe anything with logical conviction, if your "axioms" imply absurdity, you discard the "axioms" as untrustworthy, thus refuting the arguments for their usefulness (that always precede any beliefs, if you look for them). Why do I believe this? My brain tells me so, and its reasoning is potentially suspect.

Stupid question: Does everyone agree that algorithmic probability is irrelevant to human epistemic practices?

http://www.qwantz.com/index.php?comic=767&mobile=2

Solomonoff's universal prior assigns a probability to every individual Turing machine. Usually the interesting statements or hypotheses about which machine we are dealing with are more like "the 10th output bit is 1" than "the machine has the number 643653". The first statement describes an infinite number of different machines, and its probability is the sum of the probabilities of those Turing machines that produce 1 as their 10th output bit (as the probabilities of mutually exclusive hypotheses can be summed). This probability is not directly related to the K-complexity of the statement "the 10th output bit is 1" in any obvious way. The second statement, on the other hand, has probability exactly equal to the probability assigned to the Turing machine number 643653, and its K-complexity is essentially (that is, up to an additive constant) equal to the K-complexity of the number 643653.

So the point is that generic statements usually describe a huge number of different specific individual hypotheses, and that the complexity of a statement needed to delineate a set of Turing machines is not (necessarily) directly related to the complexities of the individual Turing machines in the set.

I don't think there's very much conflict. The basic idea of cousin-it's post is that the probabilities of generic statements are not described by a simplicity prior. Eliezer's post is about the reasons why the probabilities of every mutually exclusive explanation for your data should look like a simplicity prior (an explanation is a sort of statement, but in order for the arguments to work, you can't assign probabilities to any old explanations - they need to have this specific sort of structure).

You got me curious, so I did some searching. This paper gives fairly tight bounds in the case where the payoffs are adaptive (i.e. can change in response to your previous actions) but bounded. The algorithm is on page 5.

You should probably be prepared to change how much you plan to spend on exploring based on the initial information recieved.

Sure. For example, if your environment is such that the process of exploitation can alter your environment in such a way that your earlier judgment of "the most valuable thing" is no longer reliable, then an iterative cycle of explore-exploit-explore can potentially get you better results.

Of course, you can treat each loop of that cycle as a separate optimization problem and use the abovementioned strategy.

"Magical gods" in the conventional supernatural sense generally don't exist in any universes, insofar as a lot of the properties conventionally ascribed to them are logically impossible or ill-defined, but entities we'd recognize as gods of various sorts do in fact exist in a wide variety of mathematically-describable universes. Whether all mathematically-describable universes have the same ontological status as this one is an open question, to the extent that that question makes sense.

(Some would disagree with referring to any such beings as "gods", e.g. Damien Broderick who said "Gods are ontologically distinct from creatures, or they're not worth the paper they're written on", but this is a semantic argument and I'm not sure how important it is. As long as we're clear that it's probably possible to coherently describe a wide variety of godlike beings but that none of them will have properties like omniscience, omnipotence, etc. in the strongest forms theologians have come up with.)

Our world doesn't have gods; but if all possible worlds exist (which is an attractive belief for various reasons) then some of those have gods. However, they're irrelivant to us.

I think Eliezer's reply (point '(B)') to this comment by Wei Dai provides some explanation, as to what the decision theory is doing here.

From the reply (concerning UDT):

I still think [an AI ought to be able to come up with these ideas by itself], BTW. We should devote some time and resources to thinking about how we are solving these problems (and coming up with questions in the first place). Finding that algorithm is perhaps more important than finding a reflectively consistent decision algorithm, if we don't want an AI to be stuck with whatever mistakes we might make.

And yet you found a reflectively consistent decision algorithm long before you found a decision-system-algorithm-finding algorithm. That's not coincidence. The latter problem is much harder. I suspect that even an informal understanding of parts of it would mean that you could find timeless decision theory as easily as falling backward off a tree - you just run the algorithm in your own head. So with vey high probability you are going to start seeing through the object-level problems before you see through the meta ones. Conversely I am EXTREMELY skeptical of people who claim they have an algorithm to solve meta problems but who still seem confused about object problems. Take metaethics, a solved problem: what are the odds that someone who still thought metaethics was a Deep Mystery could write an AI algorithm that could come up with a correct metaethics? I tried that, you know, and in retrospect it didn't work.

The meta algorithms are important but by their very nature, knowing even a little about the meta-problem tends to make the object problem much less confusing, and you will progress on the object problem faster than on the meta problem. Again, that's not saying the meta problem is important. It's just saying that it's really hard to end up in a state where meta has really truly run ahead of object, though it's easy to get illusions of having done so.

I have gotten the notion that it's important for Value-Preserving Self-Modification in a potential AI agent, but I'm confused because it all sounds too much like game theory - there all all these other-agents it deals with

My impression is that, with self-modification and time, continuity of identity becomes a sticky issue. If I can become an entirely different person tomorrow, how I structure my life is not the weak game theory of "how do I bargain with another me?" but the strong game theory of "how do I bargain with someone else?"

Other agents are complicated regularities in the world (or a more general decision problem setting). Finding problems with understanding what's going on when we try to optimize in other agents' presence is a good heuristic for spotting gaps in our understanding of the idea of optimization.

I think the main reason is simple. It's hard to create a transparent/reliable agent without decision theory. Also, since we're talking about a super-power agent, you don't want to mess this up. CDT and EDT are known to mess up, so it would be very helpful to find a "correct" decision theory. Though you may somehow be able to get around it by letting an AI self-improve, it would be nice to have one less thing to worry about, especially because how the AI improves is itself a decision.

What exactly is the difference in meaning of "intelligence", "rationality", and "optimization power" as used on this site?

"Intelligence" is often defined as being the "g-factor" of humans - which is a pretty sucky definition of "rationality".

Go to definitions of "intelligence" used by machine intelligence researchers and it's much closer to "rationality".

Optimization power is a processes' capacity for reshaping the world according to its preferences.

Intelligence is optimization power divided by the resources used.

"Intelligence" is also sometimes used to talk about whatever is being measured by popular tests of "intelligence," like IQ tests.

Rationality refers to both epistemic and instrumental rationality: the craft of obtaining true beliefs and of achieving one's goals. Also known as systematized winning.

What Intelligence Tests Miss is a book about the difference between intelligence and rationality. The linked LW-article about the book should answer your questions about the difference between the two.

A short answer would be that intelligence describes how well you think, but not some important traits and knowledge like: Do you use your intelligence (are you a reflective person), do you have a strong need for closure, can you override your intuitions, do you know Bayes-theorem, probability theory, or logic?

Granted, any AGI we create is likely to "escape" and eventually gain access to its own software. This doesn't have to happen before the AGI matures.

Maturing isn't a magical process. It happens because of good modifications made to source code.

But why can't we just make the FAI incapable of changing itself?

Because it would be weak as piss and incapable of doing most things that we want it to do.

The majority of Friendly AI's ability to do good comes from its ability to modify its own code. Recursive self improvement is key to gaining intelligence and ability swiftly. An AI that is about as powerful as a human is only about as useful as a human.

"Safety" of own source code is actually a weak form of the problem. An AI has to keep the external world sufficiently "safe" as well, because the external world might itself host AIs or other dangers (to the external world, but also to AI's own safety), that must either remain weak, or share AI's values, to keep AI's internal "safety" relevant.

In addition to these other answers, I read a paper, I think by Eliezer, which argued that it was almost impossible to stop an AI from modifying its own source code, because it would figure out that it would gain a massive efficiency boost from doing so.

Also, remember that the AI is a computer program. If it is allowed to write other algorithms and execute them, which it has to be to be even vaguely intelligent, then it can simply write a copy of its source code somewhere else, edit it as desired, and run that copy.

I seem to recall the argument being something like the "Beware Seemingly Simple Wishes" one. "Don't modify yourself" sounds like a simple instruction for a human, but isn't as obvious when you look at it more carefully.

However, remember that a competent AI will keep its utility function or goal system constant under self modification. The classic analogy is that Gandhi doesn't want to kill people, so he also doesn't want to take a pill that makes him want to kill people.

I wish I could remember where that paper was where I read about this.

How do we know that an artificial intelligence is even possible? I understand that, in theory, assuming that consciousness is completely naturalistic (which seems reasonable), it should be possible to make a computer do the things neurons do to be conscious and thus be conscious. But neurons work differently than computers do: how do we know that it won't take an unfeasibly high amount of computer-form computing power to do what brain-form computing power does?

As far as we know, it easily could require an insanely high amount of computing power. The thing is, there are things out there that have as much computing power as human brains—namely, human brains themselves. So if we ever become capable of building computers out of the same sort of stuff that human brains are built out of (namely, really tiny machines that use chemicals and stuff), we'll certainly be able to create computers with the same amount of raw power as the human brain.

How hard will it be to create intelligent software to run on these machines? Well, creating intelligent beings is hard enough that humans haven't managed to do it in a few decades of trying, but easy enough that evolution has done it in three billion years. I don't think we know much else about how hard it is.

I've seen some mentions of an AI "bootstrapping" itself up to super-intelligence. What does that mean, exactly? Something about altering its own source code, right?

Well, "bootstrapping" is the idea of AI "pulling itself up by its own bootstraps", or, in this case, "making itself more intelligent using its own intelligence". The idea is that every time the AI makes itself more intelligent, it will be able to use its newfound intelligence to find even more ways to make itself more intelligent.

Is it possible that the AI will eventually "hit a wall", and stop finding ways to improve itself? In a word, yes.

How does it know what bits to change to make itself more intelligent?

There's no easy way. If it knows the purpose of each of its parts, then it might be able to look at a part, and come up with a new part that does the same thing better. Maybe it could look at the reasoning that went into designing itself, and think to itself something like, "What they thought here was adequate, but the system would work better if they had known this fact." Then it could change the design, and so change itself.

How do we know that an artificial intelligence is even possible? I understand that, in theory, assuming that consciousness is completely naturalistic (which seems reasonable), it should be possible to make a computer do the things neurons do to be conscious and thus be conscious. But neurons work differently than computers do: how do we know that it won't take an unfeasibly high amount of computer-form computing power to do what brain-form computing power does?

What prevents you from making a meat-based AI?

A couple of things come to mind, but I've only been studying the surrounding material for around eight months so I can't guarantee a wholly accurate overview of this. Also, even if accurate, I can't guarantee that you'll take to my explanation.

Anyway, the first thing is that brain form computing probably isn't a necessary or likely approach to artificial general intelligence (AGI) unless the first AGI is an upload. There doesn't seem to be good reason to build an AGI in a manner similar to a human brain and in fact, doing so seems like a terrible idea. The issues with opacity of the code would be nightmarish (I can't just look at a massive network of trained neural networks and point to the problem when the code doesn't do what I thought it would).

The second is that consciousness is not necessarily even related to the issue of AGI, the AGI certainly doesn't need any code that tries to mimick human thought. As far as I can tell, all it really needs (and really this might be putting more constraints than are necessary) is code that allows it to adapt to general environments (transferability) that have nice computable approximations it can build by using the data it gets through it's sensory modalities (these can be anything from something familiar, like a pair of cameras, or something less so like a geiger counter or some kind of direct feed from thousands of sources at once).

Also, a utility function that encodes certain input patterns with certain utilities, some [black box] statistical hierarchical feature extraction [/black box] so it can sort out useful/important features in its environment that it can exploit. Researchers in the areas of machine learning and reinforcement learning are working on all of this sort of stuff, it's fairly mainstream.

As far as computing power - the computing power of the human brain is definitely measurable so we can do a pretty straightforward analysis of how much more is possible. As far as raw computing power, I think we're actually getting quite close to the level of the human brain, but I can't seem to find a nice source for this. There are also interesting "neuromorphic" technologies geared to stepping up the massively parallel processing (many things being processed at once) and scale down hardware size by a pretty nice factor (I can't recall if it was 10 or 100), such as the SyNAPSE project. In addition, with things like cloud/distributed computing, I don't think that getting enough computing power together is likely to be much of an issue.

Bootstrapping is a metaphor referring to the ability of a process to proceed on its own. So a bootstrapping AI is one that is able to self-improve along a stable gradient until it reaches superintelligence. As far as "how does it know what bits to change", I'm going to interpret that as "How does it know how to improve itself". That's tough :) . We have to program it to improve automatically by using the utility function as a guide. In limited domains, this is easy and has already been done. It's called reinforcement learning. The machine reads off its environment after taking an action an updates its "policy" (the function it uses to pick its actions) after getting feedback (positive or negative or no utility).

The tricky part is having a machine that can self-improve not just by reinforcement in a single domain, but in general, both by learning and by adjusting its own code to be more efficient, all while keeping its utility function intact - so it doesn't start behaving dangerously.

As far as SIAI, I would say that Friendliness is the driving factor. Not because they're concerned about friendliness, but because (as far as I know) they're the first group to be seriously concerned with friendliness and one of the only groups (the other two being headed by Nick Bostrom and having ties to SIAI) concerned with Friendly AI.

Of course the issue is that we're concerned that developing a generally intelligent machine is probable, and if it happens to be able to self improve to a sufficient level it will be incredibly dangerous if no one put in some serious, serious effort into thinking about how it could go wrong and solving all of the problems necessary to safeguard against that. If you think about it, the more powerful the AGI is, the more needs to be considered. An AGI that has access to massive computing power, can self improve and can get as much information (from the internet and other sources) as it wants, could easily be a global threat. This is, effectively, because the utility function has to take into account everything the machine can affect in order to guarantee we avoid catastrophe. An AGI that can affect things at a global scale needs to take everyone into consideration, otherwise it might, say, drain all electricity from the Eastern seaboard (including hospitals and emergency facilities) in order to solve a math problem. It won't "know" not to do that, unless it's programed to (by properly defining its utility function to make it take those things into consideration). Otherwise it will just do everything it can to solve the math problem and pay no attention to anything else. This is why keeping the utility function intact is extremely important. Since only a few groups, SIAI, Oxford's FHI and the Oxford Martin Programme on the Impacts of Future Technologies, seem to be working on this, and it's an incredibly difficult problem, I would much rather have SIAI develop the first AGI than anywhere else I can think of.

Hopefully that helps without getting too mired in details :)

Consciousness isn't the point. A machine need not be conscious, or "alive", or "sentient," or have "real understanding" to destroy the world. The point is efficient cross-domain optimization. It seems bizarre to think that meat is the only substrate capable of efficient cross-domain optimization. Computers already surpass our abilities in many narrow domains; why not technology design or general reasoning, too?

Neurons work differently than computers only at certain levels of organization, which is true for every two systems you might compare. You can write a computer program that functionally reproduces what happens when neurons fire, as long as you include enough of the details of what neurons do when they fire. But I doubt that replicating neural computation is the easiest way to build a machine with a human-level capacity for efficient cross-domain optimization.

How does it know what bits to change to make itself more intelligent?

There is an entire field called "metaheuristics" devoted to this, but nothing like improving general abilities at efficient cross-domain optimization. I won't say more about this at the moment because I'm writing some articles about it, but Chalmers' article analyzes the logical structure of intelligence explosion in some detail.

Finally, why is SIAI the best place for artificial intelligence? What exactly is it doing differently than other places trying to develop AI?

The emphasis on Friendliness is the key thing that distinguishes SIAI and FHI from other AI-interested organizations, and is really the whole point. To develop full-blown AI without Friendliness is to develop world-destroying unfriendly AI.

How do we know that an artificial intelligence is even possible? I understand that, in theory, assuming that consciousness is completely naturalistic (which seems reasonable), it should be possible to make a computer do the things neurons do to be conscious and thus be conscious. But neurons work differently than computers do

The highlighted portion of your sentence is not obvious. What exactly do you mean by work differently? There's a thought experiment (that you've probably heard before) about replacing your neurons, one by one, with circuits that behave identically to each replaced neuron. The point of the hypo is to ask when, if ever, you draw the line and say that it isn't you anymore. Justifying any particular answer is hard (since it is axiomatically true that the circuit reacts the way that the neuron would).
I'm not sure that circuit-neuron replacement is possible, but I certainly couldn't begin to justify (in physics terms) why I think that. That is, the counter-argument to my position is that neurons are physical things and thus should obey the laws of physics. If the neuron was build once (and it was, since it exists in your brain), what law of physics says that it is impossible to build a duplicate?

how do we know that it won't take an unfeasibly high amount of computer-form computing power to do what brain-form computing power does?

I'm not physicist, but I don't know that it is feasible (or understand the science well enough to have an intelligent answer). That said, it is clearly feasible with biological parts (again, neurons actually exist).

I've seen some mentions of an AI "bootstrapping" itself up to super-intelligence. What does that mean, exactly? Something about altering its own source code, right? How does it know what bits to change to make itself more intelligent? (I get the feeling this is a tremendously stupid question, along the lines of "if people evolved from apes then why are there still apes?")

By hypothesis, the AI is running a deterministic process to make decisions. Let's say that the module responsible for deciding Newcomb problems is originally coded to two-box. Further, some other part of the AI decides that this isn't the best choice for achieving AI goals. So, the Newcomb module is changed so that it decides to one-box. Presumably, doing this type of improvement repeatedly to will make the AI better and better at achieving its goals. Especially if the self-improvement checker can itself by improved somehow.

It's not obvious to me that this leads to super intelligence (i.e. Straumli-perversion level intelligence, if you've read [EDIT] A Fire on the Deep), even with massively faster thinking. But that's what the community seems to mean by "recursive self-improvement."

Only preliminary research into the potential EV algorithms have been explored. See these citations...

Brandt 1979; Railton 1986; Lewis 1989; Sobel 1994; Zimmerman 2003; Tanyi 2006

...from The Singularity and Machine Ethics.

Is it really so strange that people are still arguing over "interpretations of quantum mechanics" when the question of whether atoms existed wasn't settled until one hundred years after John Dalton published his work?

Here's some evidence for macroscopic decoherence.

As a step toward this goal, I would really appreciate someone rewriting the post you mentioned to sound more like science and less like advocacy. I tried to do that, but got lost in the forceful emotional assertions about how collapse is a gross violation of Bayes, and how "The discussion should simply discard those particular arguments and move on."

From the Wikipedia fined-tuned universe page

Mathematician Michael Ikeda and astronomer William H. Jefferys have argued that [, upon pre-supposing MWI,] the anthropic principle resolves the entire issue of fine-tuning, as does philosopher of science Elliott Sober. Philosopher and theologian Richard Swinburne reaches the opposite conclusion using Bayesian probability.

(Ikeda & Jeffrey are linked at note 21.)

In a nutshell, MWI provides a mechanism whereby a spectrum of universes are produced, some life-friendly and some life-unfriendly. Consistent with the weak anthropic principle, life can only exist in the life-friendly (hence fine-tuned) universes. So, MWI provides an explanation of observed fine-tuning, whereas the standard QM interpretation does not.

Quantum mechanics can be described by a set of postulates. (Sometimes five, sometimes four. It depends how you write them.)

In the "standard" Interpretation, one of these postulates invokes something called "state collapse".

MWI can be described by the same set of postulates without doing that.

When you have two theories that describe the same data, the simpler one is usually the right one.