I think it might be helpful to have a variant of 3a that likewise says the orthogonality thesis is false, but is not quite so optimistic as to say the alternative is that AI will be "benevolent by default". One way the orthogonality thesis could be false would be that an AI capable of human-like behavior (and which could be built using near-future computing power, say less than or equal to the computing power needed for mind uploading) would have to be significantly more similar to biological brains than current AI approaches, and in particular would have ...
Perhaps one can think of a sort of continuum where on one end you have a full understanding that it's a characteristic of language that "everything has a name" as in the Anne Sullivan quote, and on the other end, an individual knows certain gestures are associated with getting another person to exhibit certain behaviors like bringing desired objects to them, but no intuition that there's a whole system of gestures that they mostly haven't learned yet (as an example, a cat might know that rattling its food bowl will cause its owner to come over and refill i...
I don't think it's quite right to say the idea of the universe being in some sense mathematical is purely a carry-over of Judeo-Christian heritage--what about the Greek atomists like Leucippus and Democritus for example? Most of their writings have been lost but we do know that Democritus made a distinction similar to the later notion of primary (quantitative) vs. secondary (qualitative) properties discussed at https://plato.stanford.edu/entries/qualities-prim-sec/ with his comment about qualitative sensations being matters of human convention: "By convent... (read more)