My impression - and admittedly this is just an impression - is that there are few enough x-risk workers that someone doesn't have to be especially good at doing x-risk work to be able to do more good on the margin as an x-risk worker compared to doing other things. This seems to be especially true for people who are good at motivating themselves (so that they don't need a lot of managerial support) and who are willing to do things that aren't particularly glamorous (but it's probably true even when neither of those is the case, so don't use that as an excuse).

"I might not like it" sounds like a fully general argument to me, and there are cheap tests you can do on the other issues. I suggest sending a resume to SIAI or someplace similar; if they think you wouldn't be useful there, you're at least no worse off than you are now.