I think we have a disconnect on how much we believe proper scary AIs will be like previous computer programs.

My conception of current computer programs is that they are crystallised thoughts plucked from our own minds and easily controllable and unchangeable. When we get interesting AI the programs will morphing and be far less controllable without a good theory of how to control the change.

I shudder every time people say the "AI's source code" as if it is some unchangeable and informative thing about the AI's behaviour after the first few days of the AI's existence.

I'm not sure how to resolve that difference.