Linguists tend to be a bit more specific than me. There may be a slight difference between /i/ and /j/, but they're really close. It doesn't seem to be enough to justify one being a vowel and the other being a consonant.
I tried listening to the recordings of /i/ vs /j/ on Wikipedia. /i/ just had /i/, but the recording for /j/ is /ja/, so it's hard to concentrate on the /j/. It sure sounds a lot like /ia/. Similarly, /w/ had /wa/, which sounds a heck of a lot like /ua/.
I feel like /y/ just means that you start out transitioning from /i/ to another vowel. You tend to emphasize the following vowel more. But since you could be transitioning to any vowel, it doesn't make sense that /y/ represents the transition itself. The only constant is it starts out as /i/.
A particularly interesting case is /jiː/ (Old English pronoun that is now spelled "ye"). It's clearly not just /i/, and /ii/ would sound identical. But it does seem to be somewhat of a palindrome. The /i/ at the end is extended longer, but the sounds are the same forwards and backwards. There's a slight change in the sound or emphasis between them, so it might be /ieiː/ or something where it moves to a subtly different vowel and back.
I am not interested in being an introductory phonology/phonetics textbook, but if you want to know why linguists think that semivowels should be considered a separate category to vowels, there is plenty of writing out there on the subject.I'm bowing out from further participation.
If it's worth saying, but not worth its own post (even in Discussion), then it goes here.
Notes for future OT posters:
1. Please add the 'open_thread' tag.
2. Check if there is an active Open Thread before posting a new one.
3. Open Threads should be posted in Discussion, and not Main.
4. Open Threads should start on Monday, and end on Sunday.