"There are plenty of conceivable architectures for which this meta level thinking is incapable of happening, yet nevertheless are capable of producing arbitrarily complex intelligent behavior."
Maybe, but that's exactly like the orthogonality thesis. The fact that something is possible in principle doesn't mean there's any easy way to do it in practice. The easy way to produce arbitrarily complex intelligent behavior in practice is to produce something that can abstract to an arbitrary degree of generality, and that means recognizing abstractions like "goal", "good," and so on.
The reason why a human baby becomes intelligent over time is that right from the beginning it has the ability to generalize to pretty much any degree necessary. So I don't see how that argues against my position. I would expect AIs also to require a process of "growing up" although you might be able to speed that process up so that it takes months rather than years. That is still another reason why the orthogonality thesis is false in practice. AIs that grow up among human beings will grow up with relatively humanlike values (although not exactly human), and the fact that arbitrary values are possible in principle will not make them actual.
The fact that something is possible in principle doesn't mean there's any easy way to do it in practice. The easy way to produce arbitrarily complex intelligent behavior in practice is to produce something that can abstract to an arbitrary degree of generality, and that means recognizing abstractions like "goal", "good," and so on.
I actually had specific examples in mind, basically all GOFAI approaches to general AI. But in any case this logic doesn't seem to hold up. You could argue that something needs to HAVE goals in order to be ...