It is plausible that an element of affirmative action could have crept into the awarding of the Fields Medal. It is not unreasonable to suspect that it has. Any number of biases might creep in to the awarding of a prize, however major it is. For example, it could well be that a disproportionate number of Norwegians or Swedes have won the Nobel relative to their accomplishments, because of location.

Sure. In the case of the Nobel prizes this claim has been made before. In particular, the claim is frequently made that the Nobel Prize in literature has favored northern Europeans and has had serious political overtones. There's a strong argument that the committee has generally been unwilling to award the prize to people with extreme right-wing politics while being fine with rewarding them to those on the extreme left. Moreover, you have cases like Eyvind Johnson who got the prize despite being on the committee itself and being not well known outside Sweden. (I'm not sure if any of his major works had even been translated into English or French when he got the prize.) And every few years there's a minor row when someone on the lit committee decides to bash US literature in general, connecting it to broad criticism of the US and its culture (see for example this).

There's also no question that politics has played heavy roles in the awarding of the Peace Prize.

And in the sciences there has been serious allegations of sexism in the awarding of the prizes. The best source for this as far as I'm aware is "The Madame Curie Complex" by Julie Des Jardins (unfortunately it isn't terribly well-written, at times exaggerates accomplishments of some individuals, sees patterns where they may not exist, and suffers from other problems.)

But, saying "it isn't unreasonable to suspect X" is different from asserting X without any evidence.