Slight correction. It's not more wealthy people, it's people willing to pay more, although they tend to be more wealthy as well.

Fair trade can also cause problems for the farmers because of price discrimination. If the manufacturer wants to price-discriminate, they only want to produce a limited amount of fair trade product. So the farmers produce fair trade raw material, but they can't sell all of it as fair trade, because the manufacturers only certify a limited quantity of it. The farmers who paid the increased cost of making fair trade material, but could not sell it as fair trade, lose out.