On AI: are we sure we are not influenced by meta-religious ideas of sci-fi writers who write about sufficiently advanced computers just "waking up into consciousness" i.e. create a hard, almost soul-like, barrier between conscious and not conscious, which carries an assumption that consciousness is a typically human-like feature? It is meta-religious as it is based on the unique specialness of the human soul.

I mean I think the potential variation space of intelligent, conscious agents is very, very large and a randomly selected AI will not be human-like in way we would recognize it to. We will not recognize its consciousness, we will not recognize its intelligence, even its agency, all we would see it does mysterious complicated stuff we don't understand. It may almost look random. It does stuff, maybe it communicates with us although the human-language words it uses will not reflects its thought processes, but it will be profoundly alien.