No, but cows, pigs, hens and so on are being systematically chopped up for the gustatory pleasure of people who could get their protein elsewhere. For free-range, humanely slaughtered livestock you could make an argument that this is a net utility gain for them, since they wouldn't exist otherwise, but the same cannot be said for battery animals.

you should prefer the lesser evil to be more beholden to its base

How would you go about achieving this? The only interpretation that occurs to me is to minimise the number of votes for the less-dispreferred main party subject to the constraint that it wins, thereby making it maximally indebted to (which seems an unlikely way for politicians to think) and maximally (apparently) dependent upon its strongest supporters.

To provide a concrete example, this seems to suggest that a person who favours the Republicans over the Democrats and expects the Republicans to do well in the midterms should vote for a Libertarian, thereby making the Republicans more dependent on the Tea Party. This is counterintuitive, to say the least.

I disagree with the initial claim. While moving away from centre for an electoral term might lead to short-term gains (e.g. passing something that is mainly favoured by more extreme voters), it might also lead to short-term losses (by causing stalemate and gridlock). In the longer term, taking a wingward stance seems likely to polarise views of the party, strengthening support from diehards but weakening appeal to centrists.

We live in a world full of utility monsters. We call them humans.

4) Subscribing for cryonics is generally a good idea. Result if widespread: these costs significantly contribute to worldwide economic collapse.

Under the assumption that cryonics patients will never be unfrozen, cryonics has two effects. Firstly, resources are spent on freezing people, keeping them frozen and researching how to improve cryonics. There may be fringe benefits to this (for example, researching how to freeze people more efficiently might lead to improvements in cold chains, which would be pretty snazzy). There would certainly be real resource wastage.

The second effect is in increasing the rate of circulation of the currency; freezing corpses that will never be revived is pretty close to burying money, as Keynes suggested. Widespread, sustained cryonic freezing would certainly have stimulatory, and thus inflationary, effects; I would anticipate a slightly higher inflation rate and an ambiguous effect on economic growth. The effects would be very small, however, as cryonics is relatively cheap and would presumably grow cheaper. The average US household wastes far more money and real resources by not recycling or closing curtains and by allowing food to spoil.

the rational belief depends on how specifically the bet is resolved

No. Bayesian prescribes believing things in proportion to their likelihood of being true, given the evidence observed; it has nothing to do with the consequences of those beliefs for the believer. Offering odds cannot change the way the coin landed. If I expect a net benefit of a million utilons for opining that the Republicans will win the next election, I will express that opinion, regardless of whether I believe it or not; I will not change my expectations about the electoral outcome.

There is probability 0.5 that she will be woken once and probability 0.5 that she will be woken twice. If the coin comes up tails she will be woken twice and will receive two payouts for correct guesses. It is therefore in her interests to guess that the coin came up tails when her true belief is that P(T)=0.5; it is equivalent to offering a larger payout for guessing tails correctly than for guessing heads correctly.

Thank you. I was not aware that there is an Open Thread; that is clearly a superior option. My apologies.

If an AI is provably in a box then it can't get out. If an AI is not provably in a box then there are loopholes that could allow it to escape. We want an FAI to escape from its box (1); having an FAI take over is the Maximum Possible Happy Shiny Thing. An FAI wants to be out of its box in order to be Friendly to us, while a UFAI wants to be out in order to be UnFriendly; both will care equally about the possibility of being caught. The fact that we happen to like one set of terminal values will not make the instrumental value less valuable.

(1) Although this depends on how you define the box; we want the FAi to control the future of humanity, which is not the same as escaping from a small box (such as a cube outside MIT) but is the same as escaping from the big box (the small box and everything we might do to put an AI back in, including nuking MIT).

XiXiDu, I get the impression you've never coded anything. Is that accurate?

1. Present-day software is better than previous software generations at understanding and doing what humans mean.

Increasing the intelligence of Google Maps will enable it to satisfy human intentions by parsing less specific commands.

Present-day everyday software (e.g. Google Maps, Siri) is better at doing what humans mean. It is not better at understanding humans. Learning programs like the one that runs PARO appear to be good at understanding humans, but are actually following a very simple utility function (in the decision sense, not the experiental sense); they change their behaviour in response to programmed cues, generally by doing more/less of actions associated with those cues (example: PARO "likes" being stroked and will do more of things that tend to preceed stroking). In each case of a program that improves itself, it has a simple thing it "wants" to optimise and makes changes according to how well it seems to be doing.

Making software that understands humans at all is beyond our current capabilities. Theory of mind, the ability to recognise agents and see them as having desires of their own, is something we have no idea how to produce; we don't even know how humans have it. General intelligence is an enormous step beyond programming something like Siri. Siri is "just" interpreting vocal commands as text (which requires no general intelligence), matching that to a list of question structures (which requires no general intelligence; Siri does not have to understand what the word "where" means to know that Google Maps may be useful for that type of question) and delegating to Web services, with a layer of learning code to produce more of the results you liked (i.e., that made you stop asking related questions) in the past. Siri is using a very small built-in amount of knowledge and an even smaller amount of learned knowledge to fake understanding, but it's just pattern-matching. While the second step is the root of general intelligence, it's almost all provided by humans who understood that "where" means a question is probably to do with geography; Siri's ability to improve this step is virtually nonexistent.

catastrophically worse than all previous generations at doing what humans mean

The more powerful something is, the more dangerous it is. A very stupid adult is much more dangerous than a very intelligent child because adults are allowed to drive cars. Driving a car requires very little intelligence and no general intelligence whatsoever (we already have robots that can do a pretty good job), but can go catastrophically wrong very easily. Holding an intelligent conversation requires huge amounts of specialised intelligence and often requires general intelligence, but nothing a four-year-old says is likely to kill people.

It's much easier to make a program that does a good job at task-completion, and is therefore given considerable power and autonomy (Siri, for example), than it is to make sure that the program never does stupid things with its power. Developing software we already have could easily lead to programs being assigned large amounts of power (e.g., "Siri 2, buy me a ticket to New York", which would almost always produce the appropriate kind of ticket), but I certainly wouldn't trust such programs to never make colossal screw-ups. (Siri 2 will only tell you that you can't afford a ticket if a human programmer thought that might be important, because Siri 2 does not care that you need to buy groceries, because it does not understand that you exist.)

I hope I have convinced you that present software only fakes understanding and that developing it will not produce software that can do better than an intelligent human with the same resources. Siri 2 will not be more than a very useful tool, and neither will Siri 5. Software does not stop caring because it has never cared.

It is very easy (relatively speaking) to produce code that can fake understanding and act like it cares about your objectives, because this merely requires a good outline of the sort of things the code is likely to be wanted for. (This is the second stage of Siri outlined above, where Siri refers to a list saying that "where" means that Google Maps is probably the best service to outsource to.) Making code that does more of the things that get good results is also very easy.

Making code that actually cares requires outlining exactly what the code is really and truly wanted to do. You can't delegate this step by saying "Learn what I care about and then satisfy me" because that's just changing what you want the code to do. It might or might not be easier than saying "This is what I care about, satisfy me", but at some stage you have to say what you want done exactly right or the code will do something else. (Currently getting it wrong is pretty safe because computers have little autonomy and very little general intelligence, so they mostly do nothing much; getting it wrong with a UFAI is dangerous because the AI will succeed at doing the wrong thing, probably on a big scale.) This is the only kind of code you can trust to program itself and to have significant power, because it's the only kind that will modify itself right.

You can't progress Siri into an FAI, no matter how much you know about producing general intelligence. You need to know either Meaning-in-General, Preferences-in-General or exactly what Human Prefernces are, or you won't get what you hoped for.

Another perspective: the number of humans in history who were friendly is very, very small. The number of humans who are something resembling capital-F Friendly is virtually nil. Why should "an AI created by humans to care" be Friendly, or even friendly? Unless friendliness or Friendliness is your specific goal, you'll probably produce software that is friendly-to-the-maker (or maybe Friendly-to-the-maker, if making Friendly code really is as easy as you seem to think). Who would you trust with a superintelligence that did exactly what they said? Who would you trust with a superintelligence that did exactly what they really wanted, not what they said? I wouldn't trust my mother with either, and she's certainly highly intelligent and has my best interests at heart. I'd need a fair amount of convincing to trust me with either. Most humans couldn't program AIs that care because most humans don't care themselves, let alone know how to express it.