I am stipulating that P really truly wants the gold to exist (in the same way that you would want there not to exist a bunch of people who are being tortured, ceteris paribus). Whether P should be trusting the oracle is besides the point. The difference between these scenarios is that you are correct in believing that the people being tortured is morally bad. However, your well-being would not be affected by whether the people are being tortured, only by your belief of how likely this is. Of course, you would still try to stop the torture if you could, even if you knew that you would never know whether you were successful, but this is mainly an act of altruism.
My main point is probably better expressed as "Beings with identical mental states must be equally well off". Disagreeing with this seems absurd to me, but apparently a lot of people do not share this intuition.
Also, you could easily eliminate the oracle in the example by just stating that P spontaneously comes to believe the cube exists for no reason. Or we could imagine that P has a perfectly realistic hallucination of the oracle. The fact that P's belief is unjustified does not matter. According to S, the reasons for P's mental state are irrelevant.
Whether P should be trusting the oracle is besides the point.
No, it isn't. You are claiming that P "really" wants the gold to exist, but you are also claiming that P thinks that at least one of the definitions of "the gold exists" is "the oracle said the gold exists." You are flummoxed by the paradox of P feeling just as happy due to a false belief in gold as he would based on a true belief in gold, and you are ignoring the thing that ACTUALLY made him happy: which was the oracle telling him the gold was real.
How surpri...
I figure morality as a topic is popular enough and important enough and related-to-rationality enough to deserve its own thread.
Questions, comments, rants, links, whatever are all welcome. If you're like me you've probably been aching to share your ten paragraph take on meta-ethics or whatever for about three uncountable eons now. Here's your chance.
I recommend reading Wikipedia's article on meta-ethics before jumping into the fray, if only to get familiar with the standard terminology. The standard terminology is often abused. This makes some people sad. Please don't make those people sad.