Incidentally, we can prove to some extent that different people do perceive colours differently. If you get a lamp producing a single-wavelength red light, another lamp producing a single-wavelength blue light, and a third lamp producing a single-wavelength violet light, then you can point the red and blue lamps at the same piece of white paper, and adjust their brightnesses until the combination looks just like the pure purple light. But then there will be people who disagree with you! They'll think that you need more blue, or more red!

EDIT: The technical details above are wrong, it's not possible to mix two pure wavelengths to match the colour of another pure wavelength. However there are multi-wavelength mixtures that look the same to one person but not to another.