Even so, I highly doubt the best human traders are anywhere close to optimal. It'd be interesting to see how much better a machine-learning approach would fare.

This appears to be a general property of the Monte Carlo search algorithm, which AlphaGo employs.

Most of the arguments I've seen against AI risk I've seen (in popular media, that is) take the form of arguments against AGI, full-stop. Naturally there exist more nuanced arguments (though personally I've yet to see any I find convincing), but I was referring to the arguments made by a specific part of the population, i.e. "people who engage in such goalpost-moving"--and in my (admittedly limited) experience, those sorts of people don't usually put forth very deep arguments.

I don't believe Houshalter was referring to proving Friendliness (or something along those lines); my impression is that he was talking about implementing an AI, in which case neural networks, while "scruffy", should be considered a legitimate approach. (Of course, the "scruffiness" of NN's could very well affect certain aspects of Friendliness research; my relatively uninformed impression is that it's very difficult to prove results about NN's.)

IMHO, AI safety is a thing now because AI is a thing now and when people see AI breakthroughs they tend to think of the Terminator.

Under that hypothesis, shouldn't AI safety have become a "thing" (by which I assume you mean "gain mainstream recognition") back when Deep Blue beat Kasparov?

People who engage in such goalpost-moving have already written down their bottom line, most likely because AI risk pattern-matches to the literary genre of science fiction. I wouldn't expect such people to be swayed by any sort empirical evidence short of the development of strong AGI itself. Any arguments they offer against strong AGI amount to little more than rationalization. (Of course, that says nothing about the strengths of the arguments themselves, which must be evaluated on their own merits.)

Same question and they're not copies of me? Same answer.

As I'm sure you're aware, the purpose of these thought experiments is to investigate what exactly your view of consciousness entails from a decision-making perspective. The fact that you would have given the same answer even if the virtual instances weren't copies of you shows that your reason for saying "no" has nothing to do with the purpose of the question. In particular, telling me that "it's a dick move" does not help elucidate your view of consciousness and self, and thus does not advance the conversation. But since you insist, I will rephrase my question:

Would someone who shares your views on consciousness but doesn't give a crap about other people say "yes" or "no" to my deal?

Sure, I'm the copy.

In other words, I could make you believe that you were either the original or the copy simply by telling you you were the original/the copy. This means that before I told you which one you were, you would have been equally comfortable with the prospect of being either one (here I'm using "comfortable" in an epistemic sense--you don't feel as though one possibility is "privileged" over the other). I could have even made you waffle back and forth by repeatedly telling you that I lied. What a strange situation to find yourself in--every possible piece of information about your internal experience is available to you, yet you seem unable to make up your mind about a very simple fact!

The pattern theorists answer this by denying this so-called "simple" fact's existence: the one says, "There is no fact of the matter as to which one I am, because until our experiences diverge, I am both." You, on the other hand, have no such recourse, because you claim there is a fact of the matter. Why, then, is the information necessary to determine this fact seemingly unavailable to you and available to me, even though it's a fact about your consciousness, not mine?