Thanks! I'd already read the first link, and remember thinking that it needed to be argued better. Mainly, I still think people conflate tools with agents in a box. It seems obvious (in principle) that you could build an AI that doesn't do anything but Math/CS/Physics, and doesn't even know humans exist.
I'm planning on writing up my disagreements more formally. But first, I'm waiting on getting a copy of Nick Bostrom's new book, so that I can be certain that I'm critiquing the strongest arguments.
I hadn't seen the second link. I'll definitely have to give it a more thorough read-through later.
ErinFlight said:
Thinking about it, I realized that this might be a common concern. There are probably plenty of people who've looked at various more-or-less technical or jargony Less Wrong posts, tried understanding them, and then given up (without posting a comment explaining their confusion).
So I figured that it might be good to have a thread where you can ask for explanations for any Less Wrong post that you didn't understand and would like to, but don't want to directly comment on for any reason (e.g. because you're feeling embarassed, because the post is too old to attract much traffic, etc.). In the spirit of various Stupid Questions threads, you're explicitly encouraged to ask even for the kinds of explanations that you feel you "should" get even yourself, or where you feel like you could get it if you just put in the effort (but then never did).
You can ask to have some specific confusing term or analogy explained, or to get the main content of a post briefly summarized in plain English and without jargon, or anything else. (Of course, there are some posts that simply cannot be explained in non-technical terms, such as the ones in the Quantum Mechanics sequence.) And of course, you're encouraged to provide explanations to others!