I've never heard of anyone saying "I thought that person was really intelligent, but they turned out not to be"
This is how I feel about most (but not all) of the people I've met from Less Wrong.
I think you're describing a selection effect; it's easy to notice when someone you don't think capable does something well, but it's harder to notice failures of people you think are intelligent. There are usually too many ways to give people the benefit of the doubt, and once you've started thinking someone's intelligent, you'll probably just keep thinking that.
This is how I feel about most (but not all) of the people I've met from Less Wrong.
There's probably a more general effect in play here: people are smarter in writing than they are extemporaneously. IME this is true of almost everyone, but it's especially true of people who are famous for their writing: Paul Graham, Steve Yegge, Eliezer Yudkowsky, people like that.
I've never heard of anyone saying "I thought that person was really intelligent, but they turned out not to be", and when there are scandals about people with fake credentials, they don't seem to come from people with fake credentials making mistakes-- instead, someone checks the history.
It seems to me that you can find out a lot about people's intelligence by talking with them a little, though I've underestimated people who were bright enough but didn't present as intellectual.
The real problems are with identifying conscientiousness, benevolence, and loyalty-- that's where the unpleasant surprises show up.