I believe this is likely a smaller model rather than a bigger model so I wouldn't take this as evidence that gains from scaling have plateaued.
Developing skills related to AI puts you in a better position to make AI go well. At least for me, this outweighs the other concerns that you've mentioned.
Note: This doesn't mean that you should take a job that advances fundamental AI capabilities. This would probably be net-negative as things are already moving far too fast for society to adapt. But it sounds like you're more considering jobs related to AI applications, so I'd say to go for it.
You mention that society may do too little of the safer types of RL. Can you clarify what you mean by this?
This fails to account for one very important psychological fact: the population of startup founders who get a company off the ground is very heavily biased toward people who strongly believe in their ability to succeed. So it'll take quite a while for "it'll be hard to make money" to flow through and slow down training. And, in the mean time, it'll be acceleratory from pushing companies to stay ahead.
I've heard people suggest that they have arguments related to RL being particularly dangerous, although I have to admit that I'm struggling to find these arguments at the moment. I don't know, perhaps that helps clarify why I've framed the question the way that I've framed it?
I think it's still valid to ask in the abstract whether RL is a particularly dangerous approach to training an AI system.
Oh, this is a fascinating perspective.
So most uses of RL already just use a small-bit of RL.
So if the goal was "only use a little bit of RL", that's already happening.
Hmm... I still wonder if using even less RL would be safer still.
Under the current version of the interactive model, its median prediction is just two decades earlier than that from Cotra’s forecast
Just?
The biggest danger with AIs slightly smarter than the average human is that they will be weaponised, so they'd only safe in a very narrow sense.
I should also note, that if we built an AI that was slightly smarter than the average human all-round, it'd be genius level or at least exceptional in several narrow capabilities, so it'll be a lot less safe than you might think.