This might depend in part on what kind of odds you give places like Meta, OpenAI, and whether you think that they're fundamentally doomed because of the structure of the problem.
If you think that OpenAI is absolutely doomed at how they proceed -- marginal improvements make no difference -- adding conscientious people makes no difference, so time of conscientious people is better spent anywhere else.
If you think that OpenAI stands a good chance of figuring things out -- marginal improvements could make a big difference -- then the kind of reasoning you point to is more persuasive.
I think Soares thinks we're in the first world, so his prescriptions makes sense for that. (Probably also for deontological considerations, but again, these also look different in different worlds.)
1
1Thoughtfulness, pro-sociality, and conscientiousness have no bearing on people's ability to produce aligned AI.
They do have an effect on people's willingness to not build AI in the first place, but the purpose of working at Meta, OpenAI, and Google is to produce AI. No one who is thoughtful, pro-social, and conscientious is going to decide not to produce AI while working at those companies, while still having a job.
Hence, the effect of discouraging those sorts of people from working at those companies has no net increase in Pdoom.
If you want to avoid building unaligned AI, you should avoid building AI.
1
1
Partially in reply to "Request: stop advancing AI capabilities".
Consider this argument:
Let's grant that these points as true. X field is indeed bad or dangerous, people who are some mix of pro-social and thoughtful and conscientious will recognize that, and they are exhorted not to join that field.
What happens next, causally?
(1) If the field has a limited supply of people with relevant background to make contributions in the field, the result will be to slow progress in the field.
(2) If the field is below a "critical mass" of contributors to it, it might fail to make progress entirely.
(3) However, if the field does already have a critical mass of people making contributions, the end result will be adverse selection against pro-sociality, thoughtfulness, and conscientiousness in the field.
I have a profound respect, at a distance, for MIRI and Soares. I think we actually agree about the dangers, and I agree that interpretability research (for instance) is a very important under-resourced problem that would benefit from contributors.
But I also come to one different conclusion — the more thoughtful, pro-social, and conscientious people working at Meta, OpenAI, Google, etc... the more likely things go well. I think there's a critical mass already in place for continued growth of machine learning capabilities on the path to AGI, and having people who care and are thoughtful in the room participating in their development is a good thing and not a bad thing. I would respectfully advocate the opposite position of Soares, and encourage thoughtful people to consider joining the strongest teams in the field and being a voice of reason in the room while shaping development.