There are several reasons. For one thing, the FDA has never declared that genetically modified food is entirely safe for human consumption. There have never been long-term scientific studies on the effects of eating GM food. With a huge percentage of the food on grocery store shelves being genetically modified - low estimates place it at around 60-70% of processed food alone - that's cause for concern.
There have been cases where cross-contamination of GM foods with non-GM foods have caused severe allergic reactions.
I never would have thought of this, but there are also religious implications. There are some religions that don't eat pork, for example. But if pig genes are introduced into, say, tomatoes, someone could be eating something from a pig without even knowing it.
GM technology is not cheap, and thus is not affordable for farmers in poor countries. Experts say that the increase of GM foods could lead developing countries to depend even more on industrial countries. This is NOT what developing countries need in order to get on their feet.
GM seeds are genetically uniform. This reduces genetic diversity and makes whole crops far more susceptible to pests and disease - which, in turn, increases the need for pesticide use.
There are certainly benefits to GM food. But I think it should be labeled as such, because as the consumer, we have a right to know and make that decision for ourselves.