There is do "default" to fall back to when you "distrust your brain". Any act of "distrust" must be accompanied with a specific suggestion for improvement, which, where available, should surely be taken.

Look up "Neurath's Boat" (sometimes "Neurath's Ship").

We are like sailors who on the open sea must reconstruct their ship but are never able to start afresh from the bottom. Where a beam is taken away a new one must at once be put there, and for this the rest of the ship is used as support. In this way, by using the old beams and driftwood the ship can be shaped entirely anew, but only by gradual reconstruction.

The set of all possible inputs is larger and much more diverse, than the set of all human brains.

Do you mean set of all possible sequences of inputs? As one sample of input (dominated by visual perception ~10^8 cone and rod cells * ~10 bit per cell =10^9 bits) is much less diverse than brain that contains ~10^14 synapses.

The wast majority of inputs will be processed the same way, by the most brains.

If you talk about the sequence of all inputs from birth to current moment, including genetic information, then yes, the sequence uniquely defines brain structure and output (and the sequence is partially dependent on previous outputs of brain). But this means that brain participates in its own development, and you can't say that inputs is all we need, as those inputs depend on brain's reactions (brain in vat is not counter example).

If you talk about some recent part of input sequence, than I can't see a basis for your assertion. If we have input space of N elements, output space of M elements, where N>>M, and M brains with different mappings from input to output, then counter example is that i-th brain always outputs i-th output.

Here is relevant article "Thou art physics" with relevant links.

Sorry for divergence from main topic, but I find it inappropriate and dangerous when brain is seen not as a "substrate" of conscious agent, but as a toy of laws of physics/circumstances. Especially because latter looks like rational point of view.