I often categorize people as 10-points-smarter-than-me, 20-points-smarter-than-me, etc, just naturally as I go about my day, and I'm (currently) fairly confident of my evaluations.
So you don't have any independent way to verify your evaluations. Let's apply the outside view here: Would you trust the assessment of someone of average or below average intelligence about the relative intelligence of people smarter than him. Note that there exist entire cottage industries of quacks, charlatans, and scammers based on convincing people that someone is smarter than they really are.
The outside view is very good to apply, especially in this case where there hasn't been much independent validation and lots of opportunity for confirmation bias. However, I would and do generally trust the assessment someone else makes about the intelligence of someone else. (With the exception of any assessments based on politics or tribe affiliation.) I guess I agree with the OP that intelligence is fairly straightforward to estimate with secondary signals.
I'm not familiar with any charlatans or scammers being successful by pretending to be smarter tha...
I've never heard of anyone saying "I thought that person was really intelligent, but they turned out not to be", and when there are scandals about people with fake credentials, they don't seem to come from people with fake credentials making mistakes-- instead, someone checks the history.
It seems to me that you can find out a lot about people's intelligence by talking with them a little, though I've underestimated people who were bright enough but didn't present as intellectual.
The real problems are with identifying conscientiousness, benevolence, and loyalty-- that's where the unpleasant surprises show up.