I more or less accept your reasoning as far as it goes, but:
our hardware and factory settings are so ridiculously mal-adapted to the epistemic environment of the modern world that this market test is extremely often utterly broken and useless.
If this is true, then why have so much confidence in your own personal appraisal of who to trust and who to write off as deluded? It is of course true that nearly everyone believes what they do for non-truth-tracking reasons, but "nearly everyone" isn't everyone, and there are many people, both theist and atheist, who believe what they do even despite strong memetic pressures to the contrary. Take me, for example; my theism doesn't win me any points with anyone, at least not as many points as it loses. And there are many theists like me. Knowing what you know about how easily humans fall into delusion, how can you be so confident that it's the other side that is deluded, and not your own? To return to the point, can you really be confident enough to disregard Pascal's wager? If so, how did so many at-least-nominally-truth-seeking people, from Plato to Pascal to Kant to me, end up disagreeing with you? How did we fall into such an obvious error?
We seem to be getting into some potentially very important territory, and I would certainly like to continue this discussion, but I'm running out of time for now and may be busy for up to 24 hours.
Before I go though, I should say at least one thing. It's certainly not an obvious error, and I could well be the one who's wrong. The discussions about rationality on Less Wrong are extremely useful for a basic reason: it's an extremely difficult and intricate epistemic journey to compensate for our mal-adapted hardware and software, and LW does it better than a...
Are there any essays anywhere that go in depth about scenarios where AIs become somewhat recursive/general in that they can write functioning code to solve diverse problems, but the AI reflection problem remains unsolved and thus limits the depth of recursion attainable by the AIs? Let's provisionally call such general but reflection-limited AIs semi-general AIs, or SGAIs. SGAIs might be of roughly smart-animal-level intelligence, e.g. have rudimentary communication/negotiation abilities and some level of ability to formulate narrowish plans of the sort that don't leave them susceptible to Pascalian self-destruction or wireheading or the like.
At first blush, this scenario strikes me as Bad; AIs could take over all computers connected to the internet, totally messing stuff up as their goals/subgoals mutate and adapt to circumvent wireheading selection pressures, without being able to reach general intelligence. AIs might or might not cooperate with humans in such a scenario. I imagine any detailed existing literature on this subject would focus on computer security and intelligent computer "viruses"; does such literature exist, anywhere?
I have various questions about this scenario, including: