Isn't it more likely that someone realizing that they have been subverted by a selfish meme would be trying to self-modify?

What does "subverted" mean in this context? For example I devote a lot of resources into thinking about philosophical problems which does not seem to contribute to my genetic fitness. Have I been "subverted" by a selfish meme (i.e., the one that says "the unexamined life is not worth living")? If so, I don't feel any urge to try to self-modify away from this. Couldn't a utilitarian feel the same?

There is no 'open question' issue here - "mistake", like "exothermic", does not have any prior metaphysical meaning. We are free to define it as we wish, naturalistically.

I'm having trouble with the word "metaphysical". In order for me to make sense of the claim that "mistake" and "exothermic" do not have prior metaphysical meanings, I would like to see some examples of words that do have prior metaphysical meanings, so that I can try to figure out from contrasting examples of having and not having prior metaphysical meanings what it means to have a prior metaphysical meaning. Because at the moment I don't know what you're talking about.

With apologies to Ludwig Wittgenstein, if we can't talk about the singularity, maybe we should just remain silent. :)

I happen to agree with you that the SIAI mission will never be popular. But a part of the purpose of this website is to create more people willing and capable to work (directly or indirectly) on that mission. So, not mentioning FAI would be a bit counterproductive - at least at this stage.

If I bet higher than 1/6th on a fair die's rolling 6 because in the last ten rolls 6 hasn't come up -meaning it's now 'due'- I make a mistake. I commit an error of reasoning; I do something wrong; I act in a manner I ought not to.

What about the virus particle which, in the course of sloshing about in an appropriate medium, participates in the coming into existence of a particle composed of RNA which, as it happens, is mostly identical but differs from itself in a few places. Are you saying that this particle makes a mistake in the same sense of 'mistake' as I do in making my bet?

Option (1): The sense is precisely the same (and it is unproblematically naturalistic). In this case I have to ask what the principles are by which one infers to conclusions about a virus's mistakes from facts about replication. What are the physical laws, how are their consequences (the consequences, again, being claims about what a virus ought to do) measured or verified, and so on?

Option (2): The senses are different. This was the point of calling the RNA mistake metaphorical. It was to convey that the sense is importantly different than it is in the betting case. The idea is that the sense, if any, in which a virus makes a 'mistake' in giving rise to a non-exact replica of itself is not enough to sustain the kind of norms required for rationality. It is not enough to sustain the conclusions about my betting behaviour. Is this fair?

An agent morally ought not to do something iff it tends to generate consequences contrary to the agent's interests, those negative consequences arising from the reactions of disapproval coming from other agents.

That doesn't work. It would mean conformists are always in the right, irrespective of what they are conforming to.

I agree with you that behaviorism and PCT are different, which is why I don't understand why you're interpreting the robot as PCT and not behaviorist. From the program, it seems pretty clearly (STIMULUS: see blue -> RESPONSE: fire laser) to me.

Well, your robot example was an intuition pump constructed so as to be as close as possible to stimulus-response nature. If you consider something only slightly more complicated the distinction may become clearer: a room thermostat. Physically ripped out of its context, you can see it as a stimulus-response device. Temperature at sensor goes above threshold --> close a switch, temperature falls below threshold --> open the switch. You can set the temperature of the sensor to anything you like, and observe the resulting behaviour of the switch. Pure S-R.

In context, though, the thermostat has the effect of keeping the room temperature constant. You can no longer set the temperature of the sensor to anything you like. Put a candle near it, and the temperature of the rest of the room will fall while the sensor remains at a constant temperature. Use a strong enough heat source or cold source, and you will be able to overwhelm the control system's efforts to maintain a constant temperature, but this fails to tell you anything about how the control system works normally. Do the analogous thing to a living organism and you either kill it or put it under such stress that whatever you observe is unlikely to tell you much about its normal operation -- and biology and psychology should be about how organisms work, not how they fail under torture.

Did you know that lab rats are normally starved until they have lost 20% of their free-feeding weight, before using them in behavioural experiments?

Here's a general block diagram of a control system. The controller is the part above the dotted line and its environment the part below (what would be called the plant in an industrial context). R = reference, P = perception, O = output, D = disturbance (everything in the environment besides O that affects the perception). I have deliberately drawn this to look symmetrical, but the contents of those two boxes makes its functioning asymmetrical. P remains close to R, but O and D need have no visible relationship at all.

 R |
|
V
+-------+
| |
+--->| |----+
| | | |
^ +-------+ v
| |
....... P | ............... | O .......
| |
^ +-------+ v
| | | |
+----| |<---+
| |
+-------+
^
|
D |

When you are dealing with a living organism, R is somewhere inside it. You probably cannot measure it even if you know it exist. (E.g. just what and where, physically, is the set point for deep body temperature in a mammal? Not an easy question to answer.) You may or may not know what P is -- what the organism is actually sensing. It is important to realise that when you perform an experiment on an animal, you have no way of setting P. All you can do is create a disturbance D that may influence P. D, from a behavioural point of view, is the "stimulus" and O, the creature's action on its environment, is the "response". the behaviourist description of the situation is this:

 +-------+
D | | O
----->| |----->
| |
+-------+

This is simply wrong. The system does not work like that and cannot be understood like that. It may look as if D causes O, but that is like thinking that a candle put in a certain place chills the room, a fact that will seem mysterious and paradoxical when you do not know that the thermostat is present, and will only be explained by discovering the actual mechanism, discarding the second diagram in favour of the first. No amount of data collection will help until one has made that change. This is why correlations are so lamentably low in psychological experiments.

Do you have GChat or any kind of instant messenger?

No, I've never used any of those systems. I prefer a medium in which I can take my time to work out exactly what I want to say.

Consider then a virus particle ... Surely there is nothing in biochemistry, genetics or other science which implies there is anything our very particle ought to do. It's true that we may think of it as having the goal to replicate itself, and consider it to have made a mistake if it replicates itself inaccurately, but these conceptions do not issue from science. Any sense in which it ought to do something, or is wrong or mistaken in acting in a given way, is surely purely metaphorical (no?).

No. The distinction between those viral behaviors that tend to contribute to the virus replicating and those viral behaviors that do not contribute does issue from science. It is not a metaphor to call actions that detract from reproduction "mistakes" on the part of the virus, any more than it is a metaphor to call certain kinds of chemical reactions "exothermic". There is no 'open question' issue here - "mistake", like "exothermic", does not have any prior metaphysical meaning. We are free to define it as we wish, naturalistically.

So much for the practical ought, the version of ought for which ought not is called a mistake because it generates consequences contrary to the agent's interests. What about the moral ought, the version of ought for which ought not is called wrong? Can we also define this kind of ought naturalistically? I think that we can, because once again I deny that "wrong" has any prior metaphysical meaning. The trick is to make the new (by definition) meaning not clash too harshly with the existing metaphysical connotations.

How is this for a first attempt at a naturalistic definition of the moral ought as a subset of the practical ought? An agent morally ought not to do something iff it tends to generate consequences contrary to the agent's interests, those negative consequences arising from the reactions of disapproval coming from other agents.

In general, it is not difficult at all to define either kind of ought naturalistically, so long as one is not already metaphysically committed to the notion that the word 'ought' has a prior metaphysical meaning.

This article lists the top Google+ users by # of followers. Worth a chuckle.

ETA: in general, bare links are usually not appreciated here. Still, here are two more links to interesting articles in the tech blogosphere discussing Google+.

I'm not so sure. Using the analogy of a computer program, we could think of thoughts either as like the lines of code in the program (in which case they're at one with, or in control of, the processes generating behavior, depending on how you want to look at it) or you could think of thoughts as like the status messages that print "Reticulating splines" or "50% complete" to the screen, in which case they're byproducts of those processes (very specific, unnatural byproducts, to boot).

My view is closer to the latter; they're a way of allowing the brain to make inferences about its own behavior and to communicate those inferences. Opaque processes decide to go to Subway tonight because they've heard it's low calorie, then they produce the verbal sentence "I should go to Subway tonight because it's low calorie", and then when your friend asks you why you went to Subway, you say "Because it's low calorie").

The tendency of thoughts to appear in a conversational phrasing ("I think I'll go to Subway tonight") rather than something like "Dear Broca's Area - Please be informed that we are going to Subway tonight, and adjust your verbal behavior accordingly - yours sincerely, the prefrontal cortex" is a byproduct of their use in conversation, not their internal function.

Right now I'm just asserting that this is a possibility and that it's distinct from thoughts being part of the decision-making structure. I'll try to give some evidence for it later.

I'm going to be pushing the view that people's thoughts and words are a byproduct of the processes that determine behavior rather than directly controlling them.

But then again...

[An old man asked:] '...one of my students asked me whether the enlightened man is subject to the law of causation. I answered him: "The enlightened man is not subject to the law of causation." For this answer evidencing a clinging to absoluteness I became a fox for five hundred rebirths, and I am still a fox. Will you save me from this condition with your Zen words and let me get out of a fox's body? Now may I ask you: Is the enlightened man subject to the law of causation?'''

Hyakujo said: `The enlightened man is one with the law of causation.'

See also.