Being able to be predicted is a great weakness.

Being able to be predicted is absolutely vital for making credible threats and promises. And, along with being able to accurately predict, allows for cooperation with other rational agents.

There appears to be a lot of logic here that is happening implicitly because I'm not following you.

You wrote:

An AI could not predict its own actions, because any intelligent agent is quite capable of implementing the algorithm: "Take the predictor's predicted action. Do the opposite."

Now, this seems like a very narrow sort of AI that would go and then do something else against what was predicted.

For example, it is a logical contradiction for someone to predict my actions in advance (and tell me about it), because my "programming" will lead me to do something else, much like the above algorithm.

You seem to be using "logical contradiction" in a non-standard fashion. Do you mean it won't happen given how your mind operates? In that case, permit me to make a few predictions about your actions over the next 48 hours (that you could probably predict also): 1) You will sleep at some point in that time period. 2) You will eat at some point in that time period. I make both of those with probability around .98 each. If we extend to one month I'm willing to make a similar confidence prediction that you will make a phonecall or check your email within in that time. I'm pretty sure you are not going to go out of your way as a result of these predictions to try to go do something else.

You also seem to be missing the point about what an AI would actually need to improve. Say for example that the AI has a subroutine for factoring integers. If it comes up with a better algorithm for factoring integers, it can replace the subroutine with the new one. It doesn't need to think deeply about how this will alter behavior.

I don't find predictability a weakness. If someone says to me, "Hey, Alicorn, I predict you're going to eat that sandwich you're holding," I'm going to say, "Yes. You are exactly right. And I'm glad you are! If you were wrong, then I wouldn't get to eat this delicious sandwich, which I want (that being why I made it and picked it up)."

Did you have some other, less general sort of predictability in mind when you made the claim that it's a weakness?

For example, it is a logical contradiction for someone to predict my actions in advance (and tell me about it),

Again, this is not a logical contradiction. You do not have a clear understanding of what the concept entails.It doesn't mean 'sometimes impractical' or 'often people adapt to avoid it'.

But an AI with that programming is predictable, and, much worse, manipulable! In order to get it to do anything, you need only inform it that you predicted that it will not do that thing*. It's just a question of how long it takes people to realize that it has this behavior. It is far weaker than an AI that sometimes behaves as predicted and sometimes does not. Consider e.g. Alicorn's sandwich example; if we imagine an AI that needed to eat (a silly idea but demonstrates the point), you don't want it to refuse to do so simply because someone predicted it will (which anyone easily could).

*This raises the question of whether the AI will realize that in fact you are secretly predicting that it will do the opposite. But once you consider that then the AI has to keep track of probabilities of what people's true (rather than just claimed) predictions are, I think it becomes clear that this is just a silly thing to be implementing in the first place. Especially because even if people didn't go up to it and say "I bet you're going to try to keep yourself alive", they would still be implicitly predicting it by expecting it.