If I take a 1 kg computer and put a 1 kg rock on top of it, do I now have 2 kg computer? Are you only counting the "essential" weight, and if so, how do you define "essential"? What if I have a 100 kg computer, of which 1 kg is running a sentient program, and 99 kg is playing Solitaire? How do you decide how much of the computations are part of the sentience?

What if we run a computer, record its state at each clock cycle, and broadcast those states to a billion TV screens? Do we now weight the computer nine orders of magnitude more than we would otherwise?

I think you're getting downvoted for your TL;DR, which is extremely difficult to parse. May I suggest:

TL;DR: Treating "computers running minds" as discrete objects might cause a paradox in probability calculations that involve self-location.

To the contrary, if the maximizer cares about the experienced number of paperclips, and each of the maximizers only learns about the paperclips generated by it's own choice regarding the given options

Right, that's why I split them up into different worlds, so that they don't get any utility from paperclips created by the other paperclip maximizer.

how much value to give each of the computer's votes is decided by the operators of the experiment, not the computers

Not true - see the Sleeping Beauty problem.

Suppose there's a paperclip maximizer that could either be running on a 1 kg computer or a 2 kg computer - say the humans flipped a coin when picking which computer to run it on.

Since the computations are the same, the paperclip maximizer doesn't know whether it's 1 kg or 2 kg until I tell it. But before I tell it, I offer the paperclip maximizer a choice between options A and B: A results in 5 paperclips if it's 1 kg and 0 otherwise, B results in 4 paperclips if it's 2 kg and 0 otherwise.

It seems like the paperclip-maximizing strategy is to give equal weight (ha) to being 1 kg and 2 kg, and pick A.

What happens if you're using this method and you're offered a gamble where you have a 49% chance of gaining 1000000utils and a 51% chance of losing 5utils (if you don't take the deal you gain and lose nothing). Isn't the "typical outcome" here a loss, even though we might really really want to take the gamble? Or have I misunderstood what you propose?

A bounded utility function, on which increasing years of happy life (or money, or whatever) give only finite utility in the infinite limit, does not favor taking vanishing probabilities of immense payoffs. It also preserves normal expected utility calculations so that you can think about 90th percentile and 10th percentile, and lets you prefer higher payoffs in probable cases.

Basically, this "median outcome" heuristic looks like just a lossy compression of a bounded utility function's choice outputs, subject to new objections like APMason's. Why not just go with the bounded utility function?

Ok, so who's the other one living in Berlin?

The problem seems to vanish if you don't ask "What is the expectation value of utility for this decision, if I do X", but rather "If I changed my mental algorithms so that they do X in situations like this all the time, what utility would I plausibly accumulate over the course of my entire life?" ("How much utility do I get at the 50th percentile of the utility probability distribution?") This would have the following results:

• For the limit case of decisions where all possible outcomes happen infinitely often during your lifetime, you would decide exactly as if you wanted to maximize expectation value in an individual case.

• You would not decide to give money to Pascals' mugger, if you don't expect that there are many fundamentally different scenarios which a mugger could tell you about: If you give a 5 % chance to the scenario described by Pascals mugger and believe that this is the only scenario which, if true, would make you give 5 $ to some person, you would not give the money away.

• In contrast, if you believe that there are 50 different mugging scenarios which people will tell you during your life to pascal-mug you, and you assign an independent 5 % chance to all of them, you would give money to a mugger (and expect this to pay off occasionally).

Both seem to be interesting questions. Answer both?

In my last post, I showed that the brain does not encode concepts in terms of necessary and sufficient conditions. So, any philosophical practice which assumes this — as much of 20th century conceptual analysis seems to do — is misguided.

This argument must be missing something crucial, because it fails to account for why the necessary-and-sufficient approach is so fantastically useful in mathematics. Mathematics deals with human concepts. Many of these concepts are very likely not stored in the brain as necessary and sufficient conditions. (Concepts learned in a formal setting might be stored that way, but there's little reason to think that a common concept like "triangle" is for most people.) And yet it proved incredibly fruitful to recast these concepts in terms of necessary and sufficient conditions.

In the case of mathematics, it turns out to be worthwhile to think about concepts in the decidedly unnatural mode of necessary and sufficient conditions. One might reasonably have hoped that the same admittedly unnatural mode would prove similarly worthwhile for concepts like "democracy". After all, unnatural doesn't necessarily mean worse. Now, for concepts like "democracy", the unnatural approach does prove to be worse. But it can't be simply because the approach was unnatural.