Some people told me I shouldn't be vegetarian because I kill plants.

Unless they were fruitarians, I'd be quite surprised if they took that argument seriously rather than using it as an attempt to make you look inconsistent.