OK. Then your point is that people believe in physical reality, that exists independently of them, only because of intuition - the way their minds are shaped. This is correct as a description of why people in fact believe in it.

The rejection of physical realism is solipsism. It is not a fruitful position, however, in the sense that people who say they don't believe in physical reality still act as if though they believe in it. They don't get to ignore pain, or retreat into an imaginary world inside their heads. I believe this is known as the "I refute it thus!" kicking-a-stone argument.

My argument against moral realism does not work against physical realism. My argument is basically "show me the evidence", and physical anti-realism rejects the very concept of evidence. Physical realism is a requirement for my argument and for every other argument about the physical world, too.

Regarding the more general point that we only believe in physical realism because of intuitions, and we have similar intuitions for moral realism. Once we understand why a certain intuition exists, evolutionarily speaking, that accounts for the entirety of the evidence given by the intuition.

For instance we have a strong intuition that physics is Aristotelian in nature, and not relativistic or quantum. We understand why: because it is a good model of the physical world we deal with at our scale; relativistic and quantum phenomena do not happen much at our scale, so evolution didn't build us to intuit them.

Similarly, we have moral intuitions, which both say things about morals and also say that morals are objective. From an evolutionary perspective, we understand why humans who believed their morals to be objective tended to win out over those who publicly proclaimed they were subjective and malleable. And that's a complete explanation of that intuition; it doesn't provide evidence that morals are really objective.