Huh, didn't think about it before, but now it seems obvious. I've been explaining to people that only single idealized photons are massless, and any system containing more than one photon (except maybe as a coherent state) has rest mass, but never thought to apply it to light in a medium with n>1.

Its not because the index is > 1. Interestingly, we do say photon velocity = c/n so presumably in some real sense a photon in an index n>1 medium must have a finite rest mass, but that is not what I figured out with waveguides.

With waveguides, there is a cutoff frequency, and the group-velocity of radiation in the waveguide gets lower as frequency drops until at the cutoff frequency the group-velocity is zero (and the phase velocity is infinite.) So we have a photon at cutoff frequency with zero group velocity but still finite energy hfc (planck's constant h times the cutoff frequency fc). so its restmass expressed in units of energy is hfc. Impressively, as you "accelerate" the photon, and calculate its total energy, you do find that it is hfc * ( 1 + (v_g^2)/2 + higher order terms ) which is identical for regular particles with rest mass in the theory of relativity.

In principle, a low enough loss waveguide held vertically in earth's magnetic field would have hfc stationary photons from its top end accelerated by gravity come out the bottom end with finite group velocity and therefore higher frequency f > fc (becasue hf is the energy of any photon, so if its energy increased from falling through a gravity potential, its frequency must have increased too). This is a wierd example of the well-known general relativistic blue-shift from light falling down a gravitational well.

Its been years since I thought about this stuff.