The practical point is that, if not all knowledge reduces to mathematical patterns of physical objects (the sort of thing that we can organize and learn from textbooks), then the actual project of reductionists becomes futile at a really early stage- we'd have to give up on fully understanding even a worm brain, since we could never have the knowledge of its worm-qualia.

I want to respond to your claim more thoroughly, but my response essentially consists of the second and third posts here. If you want to pick up this conversation on those threads, I'm all for it.

Also, welcome to Less Wrong!