This is a good idea, and solves the similar problem with finitely many prisoners getting at most one guess incorrect. But...

Let the size of the finite subset go to infinity.

I don't see how this immediately gives a solution; any finite set of hats has either an even or an odd number of red hats, but an infinite set of hats may have an infinite number of red hats and an infinite number of black hats, and infinities are neither odd nor even.