Hmm, interesting that this has -5 karma 5min after posting. That is not enough time to read the post. Can those downvoting explain? Thank you.
Responding to your last sentence: one thing I see as a cornerstone of biomimetic AI architectures I propose is the non-fungibility of digital minds. By being hardware-bound, humans could have an array of fail-safes to actually shut such systems down (in addition to other very important benefits like reduced copy-ability and recursive self-improvement).
In one way, of course this will not prevent covert influence and power accumulation etc. but one can argue such things are already quite prevalent in human society. So if the human-AI equilibrium stabilizes in AIs being extremely influential yet "overthrowable" if they obviously overstep, then I think this could be acceptable.
Hmm, so it is even more troubling, when eventually it does not end well, but initially it may seem like everything is fine.
To me that gives one more reason to why we should start experimenting with autonomous, unpredictable intelligent entities as soon as possible, and see if arrangements other than master-slave are possible.
Thank you. What a coincidence, huh?
I think you miss the point where gradual disempowerment from AI happens as AI is more economically and otherwise performant option that systems can and will select instead of humans. Less reliance on human involvement leads to less bargaining power for humans.
But I mean we already have examples like molochian corporate structures that kind of lost the need to value individual humans as they can afford high churn rate and there are always other people to get a decently paid corporate job even if the conditions are ... suboptimal.
If the current trajectory continues, it's not the case that the AI you have is a faithful representative of you, personally, run in your garage. Rather it seems there is a complex socio-economic process leading to the creation of the AIs, and the smarter they are, the more likely it is they were created by a powerful company or a government.
This process itself shapes what the AIs are "aligned" to. Even if we solve some parts of the technical alignment problem we still face the question of what is the sociotechnical process acting as “principal”.
This touches an idea I'd really like to get more attention. The idea that we should build AI fundamentally tethered to human nature so that this drift toward arbitrary form that e.g. "just so happens to sell best" doesn't happen. I call it tetherware - more about in my post here.
Hmm, seems like someone beat me to it. Federico Faggin describes the idea I had in mind with his Quantum Information Panpsychism theory. Check it out here if interested - and I'll appreciate your opinion on plausiblity of the theory.
https://www.essentiafoundation.org/quantum-fields-are-conscious-says-the-inventor-of-the-microprocessor/seeing/
I know this might not be a very satisfying response, but as extraordinary claims require extraordinary arguments, I'm going to need a series of posts to explain - hence the Substack.
Haha, I've noticed you reacted with "I'd bet this is false" - I would be quite willing to present my arguments and contrast them with yours, but ultimately this is philosophical belief and no conclusive evidence can be produced for either side (that we know of). Sorry if my comment was misleading.
Thank you for the feedback. I'll try to address the key points.
1) I actually have looked into EY's metaphysical beliefs and my conclusion is they are inconsistent and opaque at best, and have been criticized for that here. In any case, when I say someone operates from a single metaphysical viewpoint like physicalism, this is not any kind of criticism of their inability to consider something properly or whatnot. It just tries to put things into wider context by explaining that changing the metaphysical assumptions might change their conclusions or predictions.
2) The post in no way says that there is something that would "prevent" the existential risk. It clearly states such risk would not be mitigated. I could have made this more explicit. What the post says is that by introducting a "possibility," no matter how remote, of certain higher coordination or power that would attempt to prevent X-risk because it is not in its interest, then in such a universe the expected p(doom) would be lower. Does that make sense?
3) You say that
My reaction to that is that here your are exactly conflating physicalism with the "descriptive scope of science" which is exactly the category mistake I'm trying to point to! There will always be something unexplainable beyond the descriptive scope of science, and physicalism is filling that with "nothing but more physical-like clockwork things." But that is a belief. It might be the "most logical believe with fewest extra assumptions." But that should not grant it any special treatment among all other metaphysical interpretations.
4) Yes, I used the word "share/transmit information across distance" while describing non-locality. And while you cannot "use entanglement to transmit information," I think it's correct to say that the internal information of an entangled particle transmits information of its internal state to its entangled partner?
5) Please, don't treat this as an "attack on AI safety" - all I'm trying is to open it to a wider metaphysical consideration.