Your understanding of the word 'colour' does not match what properties of the world your brain is trying to identify and categorize when it interprets 'colour'. The interesting constant property of objects in the world that makes 'colour' useful to your visual system for purposes of object identification and categorization is really the surface properties that interact with incident lighting. Your brain attempts to ignore effects due to lighting variation and assign a 'colour' label to objects that is more or less an invariant property of the surface under a variety of different lighting conditions. This is in general not a solvable problem since the same incident photons can be produced by a number of different lighting and material combinations. Optical illusions like this merely reveal the heuristics your visual system uses to identify the relevant constant aspects of the scene and ignore the irrelevant lighting variation. They generally work quite well.