Syllable-initially they're pretty obviously consonants (yam vs am). There are also lots of languages that have phonological rules that involve replacing semi-vowels with other consonants or vice versa, which is a pretty strong argument for them being part of the class of consonants in those languages. For the other stuff, what polymathwannabe said. This stuff is well-studied in linguistics and particularly in phonetics.
How do you tell if it's a consonant or just part of a diphthong? For example, is the y in may a consonant? If so, how about the y sound in mate?
Also, by that reasoning, even the r in beard, which has another vowel in it, would have to be a vowel since it's immediately followed by a consonant.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one.
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.