So our inner states are the objects, consciousness is the modality, and then who's doing the perceiving?

A physical substrate, of course. Note that I am not using the word "brain": we don't know anything about what the substrate looks like from a materialistic point of view, except that it is somehow strongly related to the brain, and the fine-grained states and processes of the physical substrate should comprehensively explain our inner phenomenology.[1]

[1] This is a surprisingly strong condition. Fourier analysis on a waveform cannot comprehensively account for our auditory perceptions, because it doesn't account for psychoacoustics. Spectrum analysis on a source of visible light cannot explain the perception of color, since e.g. people have different cone photoreceptors, etc.