My other thought is to also get a linear algebra book that covers infinite dimensional vectors.

This is useful for, say, the hydrogen atom or the simple harmonic oscillator, but you can learn a lot just from the spin 1/2 quantum mechanics, which is quite finite-dimensional. It is sufficient for all of quantum information, EPR, Bell inequalities, etc. If you are interested in "quantum epistemology", Scott Aaronson's Quantum Computing since Democritus is an excellent read and would not overtax your math skills.