Not so much a specific misconception, but understanding the current state of AI research and understanding how mechanical most AI is (even if the mechanisms are impressive) should make you realize that being a "Friendly AI researcher" is a bit like being a unicorn tamer (and I mean that in a nice way - surely some enterprising genetic engineer will someday make unicorns).
Edit: Maybe I was being a little snarky - my meaning is simply this: Given how little we know about what actual Strong AI will look like (And we genuinely know very very little), any FAI effort will face tremendous obstacles in transforming theory into practice - both in the fact that the theory will have been developed without the guidance that real-world constraints and engineering goals provide, and the fact that there is always overhead and R&D involved in applying theoretical research. I think many people here underestimate this vast difference.
both in the fact that the theory will have been developed without the guidance that real-world constraints and engineering goals provide, and the fact that there is always overhead and R&D involved in applying theoretical research. I think many people here underestimate this vast difference.
Some people might underestimate the difficulty. On the other hand even if doing FAI research is immensely difficult that doesn't mean that we shouldn't do FAI research. The stakes are to high to avoid doing the best we can.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Thread started before the end of the last thread to ecourage Monday as first day.