If A believes h is incoherent, but B is unable to demonstrate h's incoherence, A should regard B's inability to coherently explain h's incoherence as evidence that h is not , in fact, incoherent.

Er, 'A' believes 'h' is coherent in this case.

I have realized, though that my statement was profoundly unclear, even after the edit.

Let me attempt to rephrase yet again, more precisely:

"If a bunch of people on LW tell you your hypothesis is incoherent and you need to dissolve your question, this should not be considered evidence that your hypothesis is sound, merely because nobody directly refuted your incoherence, in terms currently comprehensible by you."

Or, by analogy, if you go to a biology forum and ask about missing links or why there are still apes, and then when you get explanations that dissolve the wrong questions involved, you say, "aha, but you still haven't answered my [wrong] question, so therefore I'm right", this is not sound argument.

The denial that B's failure to coherently explain h's incoherence increases the probability that h is coherent expresses the common failure to regard others' beliefs as evidence of what's true.

In this case, though, the incoherence has actually been quite clearly counterargued by many, and is already thoroughly refuted by the sequences.

Could you edit this? I can't decipher it

Sorry, that was a messed up edit - I was at first writing "doesn't represent evidence for incoherence" and then messed up the edit to "doesn't represent evidence for the incoherent hypothesis".

More colloquially, if somebody can't coherently answer your incoherent question, it doesn't mean that the viewpoint which created the question is therefore sensible or true.

How about, "If I offer a not-even-wrong refutation of your not-even-wrong hypothesis, you can't take the not-even-wrongness of the refutation as evidence for the hypothesis."

I read it to mean that once one has demonstrated a hypothesis to be incoherent, one does not then also need to demonstrate it to be false.

if while you slept somebody made changes to your brain at any smaller level, you wouldn't wake up thinking "I feel weird".

I have no reason to assume that you couldn't replace me entirely, piece by piece. After all, I have different cells now than I did previously, and will have different cells later, and all the while still perceive myself the same.

The only thing weird here, is the idea that I would somehow notice. I mean, if I could notice, it wouldn't be a very good replacement, would it?

(Actually, given my experience with mind hacking, my observation is that it's very difficult to notice certain background characteristics of one's thought processes, such that even if a machine translation did introduce a systematic distortion, it seems unlikely to me that anyone would notice it in themselves, at least easily or at first!)

One can only detect, as opposed to invent, what is already there. Being a NAND gate is not a physical property that is already there,

Actually, "NAND Gate" is a term that we use to label something that is there --a tag we assign to patterns in the physical world that follow similar patterns of behavior to a representation we hold in our minds.

This is a bit like trees falling in the forest. If there is nobody there to label it a NAND gate, then it will still do the exact same thing... but there's no "NAND gate" there.

And, when the person does show up and label it, there's still no "NAND gate" there... there's just a label in that person's mind, saying, "that thing there is a NAND gate".

Not understanding this basic concept (that reality does not contain any labels, and has no "is-ness") leads to all sorts of confusion.

(Sadly, this kind of confusion is also the natural human state.)