One of the entrances to Theaterberg is closed, but you can take the other one or just sneak your way past the barricades.

I found the first part of the post a little bit cryptic (no pun intended). Since the second part is not just aimed at an audience with technical security knowledge, maybe at least including Shannon's maxim and a link to outfo-hazard (ok, that one is not technical) would help. Though after googling it, I still don't understand the part about Shannon's maxim, e.g.,

Generating a password with a high amount of "entropy" is just a way of ensuring that password crackers are very unlikely to break them without violating Shannon's maxim.

If Shannon's maxim is: "The enemy knows the system", violating means that they do not know it? How does not knowing the system help crackers crack high-entropy passwords? Or is "violating Shannon's maxim" to say that the attacker knows the secrete key? In that case, wouldn't it be better to say "violating Kerckhoff's principle"? (I prefer Kerckhoff's principle anyway, Shannon's maxim seems IMO just a more cryptic (sorry again) restatement of it.)

Otherwise, nice post.

Considering that the default alternative would be no alignment research, I would say, yes, it is a net positive. But I also agree that alignment research can be dual use, which would be your second and third point. I don't think the first one is a big problem, since comparatively few AI researchers seems to care about the safety of AGI to start with. Even if you believe that some approaches to alignment will not help and can only provide a safe sense of certainty, pursuing them grows the field and can IMO only help attract more attention from the larger ML community. What do you imagine a solution to the last point to look like? Doesn't preventing malicious actors from seeking power mean solving morality or establishing some kind of utopia or so? Without having looked into it, I am pessimistic that we can find a way to utopia through political science.

Thank you so much for your thoughtful comment! Specific pieces of advice like these are exactly what I am looking for. One question: what is the rationale behind the brightly colored items?

Agreeing with your post, I think it might be important to offer the people you want to reach out to a specific alternative what they should work on instead (because otherwise we are basically just telling them to quit their job, which nobody likes to hear). One such alternative would be AI alignment, but maybe that is not optimal for impatient people. I assume that researchers at OpenAI and DeepMind are in it because of the possibilities of advanced AI and that most of them are rather impatient to see them realized. Do you think it would be a good idea to advocate that those to don't want to work on alignment work on shallow AI instead?

I am also thinking of this blog post, arguing that "It’s Our Moral Obligation to Make Data More Accessible" because there is a lot proprietary data out there, which only one company/institution has access to and that stifles innovation (and it's possible to do so while respecting privacy). This also means that there is potentially a lot of data no (or few) shallow, safe ML algorithms have been tried on and that we might be able to get a substantial fraction of the benefits of AGI by just doing more with that data.
There are of course downsides to this. Making data more accessible increases the number of applications of AI and could thus lead to increased funding for AGI development. 

EDIT: Just realized that this is basically the same as number 4 in The case for Doing Something Else:

Establish international cooperation toward Comprehensive AI Services, i.e., build many narrow AI systems instead of something general

About the "senior doctors increase death rates" part, the imho most plausible explanation the authors of the study give is the following:

Second, declines in intensity of care during meetings—driven either by changes in physician composition and practice styles, reluctance to perform interventions in patients whose primary cardiologist is unavailable, or reluctance of cardiologists to intervene in high-risk patients without adequate back-up—may produce mortality reductions [...]

Which suggests that we should develop new criteria for when to operate or that patients should ask their doctor about downsides of an intervention. When framed as a question of the seniority of the surgeons, the solution that came to my mind was to seek treatment from younger doctors (or at the extreme, to fire all older doctors), so something completely different and possibly much less effective.

Concerning this passage about doctors:

You used that word “doctor” and my translator spit out a long sequence of words for Examiner plus Diagnostician plus Treatment Planner plus Surgeon plus Outcome Evaluator plus Student Trainer plus Business Manager. Maybe it’s stuck and spitting out the names of all the professions associated with medicine.

Does anybody know whether this idea is discussed anywhere else? Not being a medical expert myself I am sceptical that this idea is applicable to the full extent that seems to be implied here. It would only work if good feedback loops are established between the professions, so that information can backpropagate when the surgeon discovers that the wrong treatment was chosen and that was due to a wrong diagnosis which was made because the examiner overlooked some symptom. This point made by Petja Ylitalo above seems to be related:

To diagnose well you need to know which things are relevant for treatment, which means learning most of same knowledge as the treater would have. Information transfer between humans is time-taking and lossy, so the Treater would most likely have much less knowledge to base his treatment on than the Diagnoser, leading to increased risks of wrong treatment (for a practical example if the initial diagnosis was wrong, original diagnoser would be in a much better position to notice this when getting more information during treatment than a Treater who did not see the original data).

I do not know what exactly an Outcome Evaluator's job would be (in most cases patients should be able to evaluate the outcome themselves just fine), but I imagine somebody collecting data on remissions or, depending on the severity of the illness, survival rates and making this information public combined with the most relevant patient characteristics. I think that would be a great way to help doctors improve their skills and to improve patients' decision making, so I agree that makes sense.

There already is some specialization in medicine. Doctors are not supposed to be experts on everything and radiologists do diagnoses, but (mostly) no treatments. General practitioners often do preliminary diagnoses and send people off to specialists who then do their own diagnosis and treat the disease. In many cases this is all that is needed. The last time I went to see my general practitioner it was because I had clear symptoms of borreliosis. The doctor looked at me, said "Well that looks like borreliosis.", did a blood test that confirmed this suspicion and prescribed what I understood to be the standard antibiotic against borreliosis. After a few weeks he did a second blood test to see whether the bacteria had gone (they had). I don't see a need for specialized examiners, diagnosticians, treatment planners or outcome evaluators here, nor for the general practitioners' profession in general, nor in the emergency room. Furthermore, in these cases splitting up seems unfeasible because the services of general practitioners need to be widely available and cannot fully be carried out online and in the ER time is of the essence, which prohibits the slow communication entailed by this model, although the idea probably makes sense for chronic diseases and cancer. I think orthopaedists often recommend and then perform surgery, which seems clearly bad. It needs to be discussed to what extent encouraging patients to seek second opinions would already help alleviate the problem without reorganizing the whole medical sector.

I think training on the job in general is very useful (I wish I would have got trained on the job instead of studying at uni) and in all fields I would like to see more practitioners becoming teachers. That does not mean that teaching and performing are perfectly correlated and we cannot have some differentiation between teachers and practitioners. Just like not all drivers are driving teachers, but all driving teachers are drivers, all doctor trainers should be or have been practising doctors. I assume/hope that surgical residents perform their first operations under the supervision of somebody who not only knows how to perform the procedure and can advise them, but also can intervene themselves if need be, i.e. under the supervision of a surgeon and not just some sort of surgeon trainer, who has never held a scalpel in his life. No doctor should be forced to train residents, but isn't that the case anyway?