But then you're just plain old maximizing the wrong thing.

Er, yes? But we don't exactly have the right thing lying around, unless I've missed some really exciting FAI news...

Consider places like dropbox; they represent a huge demand for cheap storage. People probably (?) won't want huge(er) drives in their home computers going forward, but they are quite likely to want cloud storage if it comes down another order of magnitude in price. Just because people don't necessarily directly consume hard drives doesn't mean there isn't a large demand.

This is a good point that I didn't address in the post. I'd thought about it a while back but I omitted discussing it in the post.

A few counterpoints:

• Dropbox is all about backing up data that you already have. Even if everybody used Dropbox for all their content, that would still only double the need for storage space (if Dropbox stores everything at 3 locations, then it would 4X the need for storage space). This doesn't create huge incentives for improvement.
• In practice, Dropbox and cloud services wouldn't multiply storage space needs by that much, because a lot of content shared on these would be shared across devices (for instance, Amazon's Cloud Music Service doesn't store a different copy of each track for each buyer, it just stores one, or a few, copies per track). And many people won't even keep local copies. This would reduce rather than increase local storage needs. Even today, many people don't store movies on their hard drives or in DVDs but simply rely on online streaming and/or temporary online downloading.

I should note that I'm somewhat exceptional: I like having local copies of things to a much greater extent than most people (I download Wikipedia every month so I can have access to it offline, and I have a large number of movies and music stored on my hard drive). But to the extent that the Internet and improved connectivity has an effect, I suspect it would be ranging from something like multiplying demand by 4X (high-end) to actually reducing demand.

The point about camera, still, and video is good, and I do see applications in principle that could be used to fill up a lot of disk space. I don't think there is a lot of demand for these applications at the current margin. How many people who aren't photographers (by profession or hobby) even think about the storage space of their photos on their hard drives? How many people shoot videos and store them on their hard drives to a level that they actually have to start thinking about disk space considerations? I suspect the numbers involved here would be negligible. But I could be mistaken.

I think your expanded point #6 fails to consider alternative pressures for hard drive & flash memory. Consider places like dropbox; they represent a huge demand for cheap storage. People probably (?) won't want huge(er) drives in their home computers going forward, but they are quite likely to want cloud storage if it comes down another order of magnitude in price. Just because people don't necessarily directly consume hard drives doesn't mean there isn't a large demand.

Consider also that many people have high MP digital cameras, still and video. Those files add up quickly.

Counter-intuitive to say the least.

Why? This agrees with my intuition, ask for too much and you wind up with nothing.

Hypocrisy is only a vice for people with correct views. Consistently doing the Wrong Thing is not praiseworthy.

Unfortunately, it's much easier to demonstrate inconsistency than incorrectness.

The more I think about this, the more I'm confused. I don't see how this adds any substance to the claim that we don't know how to write down our values.

This proposes a way to get an OK result even if we don't quite write down our values correctly.

The IC correspond roughly with what we want to value, but differs from it in subtle ways, enough that optimising for one could be disastrous for the other. If we didn't optimise, this wouldn't be a problem. Suppose we defined an acceptable world as one that we would judge "yeah, that's pretty cool" or even "yeah, that's really great". Then assume we selected randomly among the acceptable worlds. This would probably result in a world of positive value: siren worlds and marketing worlds are rare, because they fulfil very specific criteria. They triumph because they score so high on the IC scale, but they are outnumbered by the many more worlds that are simply acceptable.

Implication: the higher you set your threshold of acceptability, the more likely you are to get a horrific world. Counter-intuitive to say the least.

TL;DR: Worlds which meet our specified criteria but fail to meet some unspecified but vital criteria outnumber (vastly?) worlds that meet both our specified and unspecified criteria.

Is that an accurate recap? If so, I think there's two things that need to be proven:

1. There will with high probability be important unspecified criteria in any given predicate.

2. The nature of the unspecified criteria is such that it is unfulfilled in a large majority of worlds which fulfill the specified criteria.

(1) is commonly accepted here (rightly so, IMO). But (2) seems to greatly depend on the exact nature of the stuff that you fail to specify and I'm not sure how it can be true in the general case.

EDIT: The more I think about this, the more I'm confused. I don't see how this adds any substance to the claim that we don't know how to write down our values.

EDIT2: If we get to the stage where this is feasible, we can measure the size of the problem by only providing half of our actual constraints to the oracle AI and measuring the frequency with which the hidden half happen to get fulfilled.

Possibly of interest: Help Teach 1000 Kids That Death is Wrong. http://www.indiegogo.com/projects/help-teach-1000-kids-that-death-is-wrong

(have not actually looked in detail, have no opinion yet)

I think you're getting downvoted for your TL;DR, which is extremely difficult to parse. May I suggest:

TL;DR: Treating "computers running minds" as discrete objects might cause a paradox in probability calculations that involve self-location.