@Julian: - But are you saying there are no terminal values detached from our evolutionary history - no reason beyond "that's the way we're put together" for us to eg: prefer mind to automaton?

Are you asking me about causal explanations or moral justifications? In terms of causal explanation, "The way you're put together" will always be lawfully responsible your mind's computation of any judgment you make. In terms of moral justification, evolution is not a justification for anything.

Does this imply an alien with a different history could share no terminal values at all with us?

Yes, though you're jumping the gun quite a bit on this.

@Caledonian: The first network makes assumptions too; it is not capable of representing a general joint probability distribution over properties. But this discussion should be moved to the relevant post.