Great post! If this is the beginning of trend to make Less Wrong posts more accessible to a general audience, then I'm definitely a fan. There's a lot of people I'd love to share posts with who give up when they see a wall of text.

There are two key things here I think can be improved. I think they were probably skipped over for mostly narrative purposes and can be fixed with brief mentions or slight rephrasings:

You won't get a direct collision between belief and reality - or between someone else's beliefs and reality - by sitting in your living-room with your eyes closed.

In addition to comparison to external data such as experimental results, there are also critical insights on reality to be gained by armchair examination. For example, armchair examination of our own or others’ beliefs may lead us to realise that they are self-contradictory, and therefore that it is impossible for them to be true. No experimental results needed! This is extraordinarily common in mathematics, and also of great personal value in everyday thinking, since many cognitive mistakes lead directly to some form of internal contradiction.

And yet it seems to me - and I hope to you as well - that the statement "The photon suddenly blinks out of existence as soon as we can't see it, violating Conservation of Energy and behaving unlike all photons we can actually see" is false, while the statement "The photon continues to exist, heading off to nowhere" is true.

It's better to say that the first statement is unsupported by the evidence and purely speculative. Here's one way that it could in fact be true: if our world is a simulation which destroys data points which won’t in any way impact the future observations of intelligent beings/systems. In fact, that’s an excellent optimisation over an entire class of possible simulations of universes. There would be no way for us to know this of course (the question is inherently undecideable) but it could still happen to be true. In fact, we can construct extremely simply toy universes for which this is true. Undecideability in general is a key consideration that seems missing from many Less Wrong articles, especially considering how frequently it pops up within any complex system.

"Reality is that which, when you stop believing in it, doesn't go away. " - Philip K Dick.

Also known as the "people can't remember things without distinctive features" phenomenon. Still interesting to note their behaviours in the situation though.

I understand the point Eliezer's trying to make here. However, you (whoever's reading this) could not convince me that ss0 + ss0 =sss0 in Peano arithmetic (I define the scenario in which my mind is directly manipulated so that I happen to believe this not to constitute "convincing me"). Here's why I believe this position to be rational:

A)In order for me to make this argument, I have to presume communication of it. It's not that I believe the probability of that communication to be 1. Certainly many people might read this comment and not know Peano arithmetic, misunderstand my language, not finish reading, etc. etc. etc. and the probability of this is nontrivial. However, arguments are directed at the possible worlds in which they are understood.

B)Communication of "ss0 + ss0 =" as a statement of Peano arithmetic already fully constrains the answer to be "ssss0" simply by virtue of what these symbols mean. That is to say, that having understood these symbols and Peano arithmetic, no further experience is necessary to know that "sss0" is wrong. Mental flaws at any point in this process or understanding are possible, but they exist only within possible worlds in which communication of these ideas does not actually occur because to think that "ss0 + ss0 = sss0" is to misunderstand Peano arithmetic, and understanding Peano arithmetic is a prerequisite for understanding a claim about it.

Therefore

C)There is no possible world within which I can be convinced of the properly communicated concept "ss0 + ss0 = sss0". Of course, this doesn't mean there's no possible world in which I can be convinced that I am experiencing a neurological fault or being manipulated, or that there are no possible worlds in which I happen to wrongly believe that ss0 + ss0 = sss0. It's just that someone experiencing a neurological fault or being manipulated is not the same thing as someone being convinced.

A similar argument holds for the impossibility of me convincing myself that ss0 + ss0 = sss0. I understand ss0 + ss0 = ssss0 in Peano arithmetic well enough that I can review in a very short period of time why it must be so. Thus you would literally have to make me forget that I know this in order to have me believe otherwise, which hardly counts as "convincing." This does not mean that I am presuming mental errors or Dark Lords of the Matrix to be impossible. For clarification, here's what a runthrough of me experiencing what Eliezer proposes would look like:

I get up one morning, take out two earplugs, and set them down next to two other earplugs on my nighttable, and noticed that there were now three earplugs, without any earplugs having appeared or disappeared—in contrast to my stored memory that 2 + 2 was supposed to equal 4.

Because that stored memory entails an understanding of why, I run through those reasons. If they're incomplete this constitutes me "forgetting that I know this." (It does not mean that I don't know this now. Right now I do.) Therefore I don't have a "stored memory that 2 + 2 was supposed to equal 4." I have an incomplete stored memory which tries to say something about 2, +, =, and 4 (if my personality were intact I would probably try and re-derive the missing parts of it, after calling 911). Either way I identify a cognitive fault. In real life waking up to this my most likely suspect would be that my experience of one earplug disappearing was deleted before I processed it, but there are lots of other possibilities as well. If I repeated the experiment multiple times I would consider either a systematic fault or "being messed with" at a fundamental level.

When I visualize the process in my own mind, it seems that making XX and XX come out to XXXX requires an extra X to appear from nowhere

Still presuming an intact line of reasoning saying why this must not be so, I would again identify a cognitive fault, and a pretty cool one at that. Something this intricate might well leave me suspecting Dark Lords of the Matrix as a nontrivial possibility, provided all other cognitive functions seemed fully intact. Still wouldn't be as likely as a weird brain fault, though. I would definitely have fun investigating this.

I check a pocket calculator, Google, and my copy of 1984 where Winston writes that "Freedom is the freedom to say two plus two equals three."

Dark Lords of the Matrix bump higher, but Psychosis has definitely leapt into the front of the pack.

I could keep going, of course. These last few presume I can still reason out something like Peano arithmetic. If I can't incidentally, then of course they look different but I still don't think it would be accurate to describe any possible outcome as "me being convinced that 2 + 2 = 3." If you run all the way down the list until you literally delete all things that I know and all ways I might obtain them, I would describe that as a possible universe in which "me" has been deleted. The strict lower bound on where I can still stumble across my cognitive fault and/or manipulation is when my reasoning ability is no longer Turing complete. This essentially requires the elimination of all complex thought, though of course making it merely unlikely for me to stumble upon the fault is much easier--just delete everything I know about formal mathematics, for example.

tl;dr

I agree with most of what Eliezer is saying, but wouldn't say that I could be convinced 2 + 2 = 3. Does this make my belief unconditional? Dependant on me understanding what 2 + 2 = 3 means, maybe it does. Maybe an understanding of 2, +, =, 3, and 4 necessitates 2 + 2 = 4 for a rational mind, and any deviation from this, even in internal mental processes, would be identifiable as a fault. After all, you can run a software program to detect flaws in some computer processors.