I'd just forget the majoritarian argument altogether, it's a distraction.
The second question does seem important to me, I too am skeptical that an AI would "obviously" have the capacity to recursively self-improve.
The counter-argument is summarized here, whereas we humans are stuck with an implementation substrate which was never designed for understandability, an AI could be endowed with both a more manageable internal representation of its own capacities and a specifically designed capacity for self-modification.
It's possible - and I find it intuitively plausible - that there is some inherent general limit to a mind's capacity for self-knowledge, self-understanding and self-modification. But an intuition isn't an argument.
I see Yoreth's version of the majoritarian argument as ahistorical. The US Government did put a lot of money into AI research and became disillusioned. Daniel Crevier wrote a book AI: The tumultuous history of the search for artificial intelligence. It is a history book. It was published in 1993, 17 years ago.
There are two possible responses. One might argue that time has moved on, things are different now, and there are serious reasons to distinguish today's belief that AI is around the corner from yesterday's belief that AI is around the corner. Wrong th...
This thread is for the discussion of Less Wrong topics that have not appeared in recent posts. If a discussion gets unwieldy, celebrate by turning it into a top-level post.