All of InhalingExhaler's Comments + Replies

It's like repairing the foundations of a building. You can't uproot all of them, but you can uproot any of them, as long as you take care that the building doesn't fall down during renovations.

Welcome to Less Wrong! My short answer to the conundrum is that if the first thing your tool does is destroy itself, the tool is defective. That doesn't make "rationality" defective any more than crashing your first attempt at building a car implies that "The Car" is defective. Designing foundations for human intelligence is rather like designing foundations for artificial (general) intelligence in this respect. (I don't know if you've looked at The Sequences yet, but it has a lot of material on the common fallacies the latter enterprise has often fallen into, fallacies that apply to everyday thinking as well.) That people, on the whole, do not go crazy — at least, not as crazy as the tool that blows itself up as soon as you turn it on — is a proof by example that not going crazy is possible. If your hypothetical system of thought immediately goes crazy, the design is wrong. The idea is to do better at thinking than the general run of what we can see around us. Again, we have a proof by example that this is possible: some people do think better than the general run.

As soon as the Dark Matrix Lords can (and do) directly edit your perceptions, you've lost. (Unless they're complete idiots about it) They'll simply ensure that you cannot perceive any inconsistencies in the world, and then there's no way to tell whether or not your perceptions are, in fact, being edited. The best thing you could do is find a different proof and hope that the Dark Lord's perception-altering abilities only ever affected a single proof. At this point, John has to ask himself - why? Why does it matter what is true and what is not? Is there a simple and straightforward test for truth? As it turns out, there is. A true theory, in the absence of an antagonist who deliberately messes with things, will allow you to make accurate predictions about the world. I assume that John cares about making accurate predictions, because making accurate predictions is a prerequisite to being able to put any sort of plan in motion. Therefore, what I think John should do is come up with a number of alternative ideas on how to predict probabilities - as many as he wants - and test them against Bayesian reasoning. Whichever allows him to make the most accurate predictions will be the most correct method. (John should also take care not to bias his trials in favour of situations - like tossing a coin 100 times - in which Bayesian reasoning might be particularly good as opposed to other methods)