We don't say that one is prime.

One definition of a prime, of course, is "a number whose only factors are itself and 1, except for 1 itself". Another, however, is "a number with exactly two factors", which is probably the simpler than "a number whose only factors are itself and 1". And if 1 were prime, it would be a highly exceptional one, in that there would be many places to say "all prime numbers except 1".

e^x is often said to be the only function that is its own derivative, as if the zero function somehow didn't count.

The only functions defined over all real numbers that are their own derivatives are those of the form k*e^x for some real number k. These include not only e^x but 2e^x and 0e^x.

ke^x is its own derivative for any k, including 0. It's a lot more convenient for 1 not to be prime. But 0! = 1, for example.

Is there such a convention?

Yes -- at least in the sense that I have found familiarity with (and sympathy toward) this practice to be an effective shibboleth for distinguishing the mathematically sophisticated.

(It's kind of like how it's a warning sign when someone doesn't think the word "dictionary" should be in the dictionary.)

One is not prime. The zero function is a trivial function; it actually doesn't count (for reasons that are technical).