We don't say that one is prime.

One definition of a prime, of course, is "a number whose only factors are itself and 1, except for 1 itself". Another, however, is "a number with exactly two factors", which is probably the simpler than "a number whose only factors are itself and 1". And if 1 were prime, it would be a highly exceptional one, in that there would be many places to say "all prime numbers except 1".

e^x is often said to be the only function that is its own derivative, as if the zero function somehow didn't count.

The only functions defined over all real numbers that are their own derivatives are those of the form k*e^x for some real number k. These include not only e^x but 2e^x and 0e^x.