One question I had reading this is: What does it matter if our model of human consciousness is wrong? If we create FAI that has all of the outward functionality of consciousness I still would consider that a win. Not all eyes that have evolved are human eyes; the same could happen with consciousness. If we manufactured some mechanical "eye" that didn't model exactly the interior bits of a human eye but was still able to do what eyes do, shouldn't we still consider this an eye? It would seem nonsensical to me to question whether this mechanical eye "really" sees because the act of seeing is a subjective experience that can't be truly modeled or say that it's not "real" seeing because it's computational seeing.