Great post, Daniel! 

I would expect that a misaligned ASI of the first kind would seek to keep knowledge of its capabilities to a minimum while it accumulates power. If nothing else, because by definition it prevents the detection and mitigation of its misalignment. Therefore for the same reasons this post advocates for openness past a certain stage of development, the unaligned ASI of the first kind would move towards a concentration and curtailing of knowledge (I.e. it would not be the kind of AI that stops the finding and fixing of its misalignment if it allowed 10x-1000x more human brain power investigating itself).

One way to increase the likelihood of keeping itself hidden is by influencing the people that already possess knowledge of its capabilities to act toward that outcome. So even if the few original decision makers with knowledge and power are predisposed to eventual openness/benevolence, the ASI could (rather easily, I imagine) tempt them away from said policy. Moreover, it could help them mitigate, reneg on,  neutralize or ignore any precommitments or promises previously made in favour of openness. 

I really enjoyed this post, Richard! The object level message is inspiring, but I also found myself happily surprised at the synthesis of three books which, on their surface, are about quite different subjects! Just throwing my +1 here as someone interested in reading more posts like this. 

 

I definitely agree with this last point! I've been on the providing end of similar situations with people in cybersecurity education of all sorts of different technical backgrounds. I've noticed that both the tester and the "testee" (so to speak) tend to have a better and safer experience when the cards are compassionately laid out on the table at the end. It's even better when the tester is able to genuinely express gratitude toward the testee for having taught them something new, even unintentionally. 

Hey Ben, thanks for this! I especially appreciate the section on Slack norms; I've found that teams have a tendency to default to DMs without explicit, mindful management of channels, channel membership and expectation-setting. 

One dynamic I've noticed: when team members get really into a conversation and it becomes a "centithread", it often benefits from outsider-nudging to remind folks that they'd be better off jumping in a call, and then summarizing the discussion. 

Awesome, I'll be checking this out for sure. I recently began studying computer security; do you have any more recommendations?

I appreciate the initiative to send meta-sources rather than single pieces.

Added to my reading list, thanks!

Thanks Gunnar. Luke may not have linked his thread, because I did so in the OP.

Thanks, Luke. I'll be checking your physics recommendations out soon.

If you are often travelling over bridge by car, having a car-knife could be handy in case you go over. The device generally comes equipped with a seat belt cutter, pressurized hammer, and flashlight.