Memories can be collapsed under percepts.
In answer to your broader question - yup: you've hit upon epistemic nihilism, and there is no real way around it. Reason is Dead, and we have killed it. Despair.
...Or, just shrug and decide that you are probably right but you can't prove it. There's plenty of academic philosophy addressing this (See: Problem of Criterion) and Lesswrong covers it fairly extensively as well.
http://lesswrong.com/lw/t9/no_license_to_be_human/ and related posts.
http://lesswrong.com/lw/iza/no_universally_compelling_arguments_in_math_or/
Rather than going on a reading binge I recommend to just continue mulling it over until it clicks into place, because, similar to the whole "dissolve free will" thing, it feels clear in hindsight yet it is not easy to explain or understand explanations others provide.
I'll give it a shot anyway: The essential point is that ultimately you are a brain and you gonna do things the way your brain is designed to do them. Assuming you've satisfactorily resolved the whole moral nihilism thing (even though there is no divine justification for morality, we can still talk about what is moral and what isn't because morality is inside us), resolving epistemic nihilism follows an analogous chain of thought: There is not and cannot be any justification for human methods of inference[morality], but it still is our method of inference[morality] and we're gonna use it regardless.
However, our concepts of truth and goodness allow us to pose the questions:standard responsesIsm persuaded of it, but is it really true? I approve of it, but is it really good?
The simple version of internalising truth and goodness by rubber stamping prevailing attitudes is not satisfactory; the complex version......is complex.
Standard methods of inferring knowledge about the world are based off premises that I don’t know the justifications for. Any justification (or a link to an article or book with one) for why these premises are true or should be assumed to be true would be appreciated.
Here are the premises:
“One has knowledge of one’s own percepts.” Percepts are often given epistemic privileges, meaning that they need no justification to be known, but I see no justification for giving them epistemic privileges. It seems like the dark side of epistemology to me.
“One’s reasoning is trustworthy.” If one’s reasoning is untrustworthy, then one’s evaluation of the trustworthiness of one’s reasoning can’t be trusted, so I don’t see how one could determine if one’s reasoning is correct. Why should one even consider one’s reasoning is correct to begin with? It seems like privileging the hypothesis, as there are many different ways one’s mind could work, and presumably only a very small proportion of possible minds would be remotely valid reasoners.
“One’s memories are true.” Though one’s memories of how the world works gives a consistent explanation of why one is perceiving one’s current percepts, a perhaps simpler explanation is that the percepts one are currently experiencing are the only percepts one has ever experienced, and one’s memories are false. This hypothesis is still simple, as one only needs to have a very small number of memories, as one can only think of a small number of memories at any one time, and the memory of having other memories could be false as well.