More broadly, I'm skeptical of 'intelligence' in general. It doesn't seem like a useful term.
People here have tried to define intelligence in more strict terms. See Playing Taboo with “Intelligence”. They define 'intelligence' as an agent’s ability to achieve goals in a wide range of environments.
It seems your post seems to be more about free will than intelligence as defined by Muehlhauser in the above article. Free will has been covered quite comprehensibly on LessWrong) so I'm not particularly interested debating about it.
Anyway, if you define intelligence as the ability to achieve goals in a wide range of environments then it doesn't really matter if the AI's actions are just an extension of what it was programmed to do. Even people are just extensions of what they were "programmed to do by evolution". Unless you believe in magical free will, one's actions have to come from some source and in this regard people don't differ from paper clip maximizers.
What would yours be?
I just think there are good optimizers and then there are really good optimizers. Between these there aren't any sudden jumps except when the FOOM happens and possibly from unFriendly to Friendly. There isn't any sudden point when the AI becomes sentient and the question how well the AI resembles humans is just a question of how well the AI can optimize towards this.
Say we bet, you and I, on whether AI will happen in 50 years. What would you want me to accept as evidence that it had done so.
There are already some really good optimizers, like Deep Blue and other chess computers that are far better at playing chess than their makers. But you probably meant when AIs become sentient? I don't know exactly how sentience works, but I think something akin to the Turing test that shows how well the AI can behave like humans is sufficient to show that AI is sentient, at least in one subset of sentient AIs. To reach a FOOM scenario the AI doesn't have to be sentient, just really good at cross-domain optimization.
I'm confused. You are looking for good reasons to believe that AI is not possible, per your post two above, but from your beliefs it would seem that you either consider AI to already exist (optimizers) or be impossible (sentient).
A long blog post explains why the author, a feminist, is not comfortable with the rationalist community despite thinking it is "super cool and interesting". It's directed specifically at Yvain, but it's probably general enough to be of some interest here.
http://apophemi.wordpress.com/2014/01/04/why-im-not-on-the-rationalist-masterlist/
I'm not sure if I can summarize this fairly but the main thrust seems to be that we are overly willing to entertain offensive/taboo/hurtful ideas and this drives off many types of people. Here's a quote:
The author perceives a link between LW type open discourse and danger to minority groups. I'm not sure whether that's true or not. Take race. Many LWers are willing to entertain ideas about the existence and possible importance of average group differences in psychological traits. So, maybe LWers are racists. But they're racists who continually obsess over optimizing their philanthropic contributions to African charities. So, maybe not racists in a dangerous way?
An overly rosy view, perhaps, and I don't want to deny the reality of the blogger's experience. Clearly, the person is intelligent and attracted to some aspects of LW discourse while turned off by other aspects.