My understanding is that many kind of pesticide and herbicide were proven to have adverse health effects, like cancer. But not all of them.

In the lack of information of which specific pesticide/herbicide/... was used on a given piece of "conventional food", and in-depth knowledge of the toxicity of each of them, I consider the "chose organic when possible" to be a reasonable safety measure. Some of the "conventional food" will not be any worse than the organic, some will be worse, and since I can't tell them apart, I consider "non-organic" to be weak evidence towards "slightly toxic".

The other point is, regardless of the effect of the food itself, some of the "conventional farming" does damages to the ecosystem - there is evidence it does pollute underground water, kill bees, ... Not all of it, but once again, I lack the ability to tell apart a "low-impact" conventional product from a "high impact" one, so I tend to chose organic also because of the effects not to me when I eat it, but to the everyone when it's produced.

Both of them are relatively weak reasons, but buying organic when possible only has a weak cost (it's not that much more expensive, and I can afford it).