If the sufficiently advanced technology is indistinguishable from magic, the arguments about "sufficiently advanced AI system" in absence of actual definition what it is, are indistinguishable from magical thinking.
Ok, then, so the actual problem is that the people who worry about AIs that behave psychopatically have such a capable definition for AI that you consider them basically speaking nonsense?
The "sufficiently advanced" in their argumentations means "sufficiently advanced in the direction of making my argument true" and nothing more.
If I adopt pragmatic version of "advancedness", then the software (algorithms) that are somehow magically made to* self identify with it's computing substrate, is less advanced, unless it is also friendly or something.
Faith, Hope, and Singularity: Entering the Matrix with New York’s Futurist Set
To my knowledge LessWrong hasn't received a great deal of media coverage. So, I was surprised when I came across an article via a Facebook friend which also appeared on the cover of the New York Observer today. However, I was disappointed upon reading it, as I don't think it is an accurate reflection of the community. It certainly doesn't reflect my experience with the LW communities in Toronto and Waterloo.
I thought it would be interesting to see what the broader LessWrong community thought about this article. I think it would make for a good discussion.
Possible conversation topics:
Edit 1: Added some clarification about my view on the article.
Edit 2: Re-added link using “nofollow” attribute.