Probably no-one. And today no-one really holds nations with nuclear capabilities accountable. How are physicists any worse?

Consider that physicists have fewer potential reasons to use nuclear weapons than do nations, or to go to "war" with one another. If they attacked one another, or third parties, they would not be able to protect themselves from (non-nuclear) retaliation as rulers of nations do, because they wouldn't have large territories and conventional armies. If they won the war, they wouldn't get any economical benefits; their only possible reason for using the nuclear weapons would be to kill some people. I consider that individual people, particularly those who are well-off and have a lot of money to solve problems with, are much less likely to want to kill others than are nations.