Hmm, maybe I'm interpreting the statement to mean something weaker and more handwavy than you are. I agree with claims like "with current technology, it can be hard to make an AI pursue some goals as competently as other goals" and "if a goal is hard to specify given available training data, then it's harder to make an AI pursue it".

However, I think how competently an AI pursues a goal is somewhat different than whether an AI tries to pursues a goal at all.(Which is what I think the strong version of the thesis is still getting at.) I was trying to get at the "hard to specify" thing with the simplicity caveat. There are also many other caveats because goals and other concepts are quite handwavy.

Doesn't seem important to discuss further.

I think I agree with everything you said. (Except for the psychologising about Eliezer on which I have no particular opinion.)

Could you give an example of knowledge and skills not being value neutral?

(No need to do so if you're just talking about the value of information depending on the values one has, which is unsurprising. But it sounds like you might be making a more substantial point?)

I mean, the most straightforward reading of Chapters 7 and 8 of Superintelligence is just a possibility-therefore-probability fallacy in my opinion.

The most relevant quote from Superintelligence (that I could find) is:

Second, the orthogonality thesis suggests that we cannot blithely assume that a superintelligence will necessarily share any of the final values stereotypically associated with wisdom and intellectual development in humans—scientific curiosity, benevolent concern for others, spiritual enlightenment and contemplation, renunciation of material acquisitiveness, a taste for refined culture or for the simple pleasures in life, humility and selflessness, and so forth. We will consider later whether it might be possible through deliberate effort to construct a superintelligence that values such things, or to build one that values human welfare, moral goodness, or any other complex purpose its designers might want it to serve. But it is no less possible— and in fact technically a lot easier—to build a superintelligence that places final value on nothing but calculating the decimal expansion of pi. This suggests that—absent a special effort—the first superintelligence may have some such random or reductionistic final goal.

My interpretation is that Bostrom is trying to be reasonably precise here and trying to do something like:

1. You might have "blithely assumed" that things would necessarily be fine, but orthogonality. (Again, extremely obvious.)
2. Also, it (separately) seems to me (Bostrom) to be technically easier to get your AI to have a simple goal, which implies that random goals might be more likely.

I think you disagree with point (2) here (and I disagree with point 2 as well), but this seems different from the claim you made. (I didn't bother looking for Bostrom's arguments for (2), but I expect them to be weak and easily defeated, at least ex-post.)

TBC, I can see where you're coming from, but I think Bostrom tries to avoid this fallacy. It would be considerably better if he explicitly called out this fallacy and disclaimed it. So, I think he should be partially blamed for likely misinterpretations.

Intuitions about simplicity in regimes where speed is unimportant (e.g. turing machines with minimal speed bound) != intuitions from the solomonoff prior being malign due to the emergence of life within these turing machines.

It seems important to not equivocate between these.

(Sorry for the terse response, hopefully this makes sense.)