According to the World Health Organization:

GM foods currently available on the international market have passed risk assessments and are not likely to present risks for human health. In addition, no effects on human health have been shown as a result of the consumption of such foods by the general population in the countries where they have been approved. Continuous use of risk assessments based on the Codex principles and, where appropriate, including post market monitoring, should form the basis for evaluating the safety of GM foods.

The reasoning most laypersons employ when thinking about GMOs is totally broken. For example, people worry about whether GMOs are less nutritious or unhealthy in some other sense. But there is no reason to think GMOs as a category would be skewed in any one direction, since any given plant might have different modified genes. It needs to be looked at in a case by case basis. Furthermore, GMOs aren't being developed in some Wild West free market. They are among the most carefully regulated substances you can ingest.

ETA: As for the concern about self-replication, that shouldn't be too serious of a problem. Domesticated organisms are optimized for thriving under human care, and pretty much every domesticated plant and animal does poorly in the wild.