Sorry for the slow reply.
I'm flattered that my thoughts as someone who has no computer science degree and just a couple years of professional programming experience are considered valuable. So here's more info-dumping (to be taken with a grain of salt, like the previous info dump, because I don't know what I don't know):
My comment on different sorts of programming, and programming cultures, and how tolerant they are of human error. Quick overview of widely used bug reduction techs (code review and type systems should have been included).
Ben Kuhn says that writing machine learning code is unusually unforgiving, which accords well with my view that data science programming is unusually unforgiving (although the reasons aren’t completely the same).
Improving the way I managed my working memory seemed important to the way I reduced bugs in my code. I think by default things fall out of your working memory without you noticing, but if you allocate some of your working memory to watching your working memory, you can prevent this and solve problems in a slower but less error-prone way. The subjective sense was something like having a "meditative bulldozer" thinking style where I was absolutely certain of what I had done with each subtask before going on to the next. It's almost exactly equivalent to doing a complicated sequence of algebraic operations correctly on the first try. It seems slower at first, but it's generally faster in the long run, because fixing errors after the fact is quite slow. This sort of perfectionistic attention to detail was actually counterproductive for activities I worked on after quitting my job, like reading marketing books, and I worked to train it out. The feeling was one of switching in to a higher mental gear: I could no longer climb steep hills, but I was much faster on level ground. (For reference: my code wasn't completely bug-free by any means, but I did have a reputation on the ops team of deploying unusually reliable data science code, I once joked to my boss on our company bug day that the system I maintained was bug-free and I didn't have anything to do, which caused her to chuckle in assent, and my superiors were sad to see me leave the company.)
For our company hack day, I wrote a Sublime Text extension that would observe the files opened and searches performed by the user in order to generate a tree diagram attempting to map the user’s thought process. This seemed helpful for expanding my working memory with regard to a particular set of coding tasks (making a scattered set of coherent modifications in a large, thorny codebase).
Another thing that seemed useful was noticing how unexpected many of the bugs I put in to production were and trying to think about how I might have anticipated the bug in advance. I noticed that the most respected & senior programmers at the company I worked all had a level of paranoia that seemed excessive to me as a newbie, but I gradually learned to appreciate. (I worked at a company where we maintained a web app that was deployed every day, and we had a minimal QA team and no staging of releases, so us engineers had lots of opportunities to screw up.) Over time, I developed a sort of “peripheral vision” or lateral thinking capability that let me forsee some of these unexpected bugs the way the more senior engineers did.
Being stressed out seemed to impede my performance in writing bug-free code significantly (esp. the “peripheral vision” aspect, subjectively). The ideal seems to be a relaxed deep flow state. One of my coworkers was getting chewed out because a system he wrote continued causing nasty consumer-facing bugs through several rewrites. I was tasked with adding a feature to this system and determined that a substantial rewrite would be necessary to accommodate the feature. I was pleased with myself when my rewrite was bug-free practically on the first try, but I couldn’t help but think I had an unfair advantage because I wasn’t dealing with the stress of getting chewed out. I’m in favor of blameless post-mortems (larger antipattern: System 1 view of bugs as aversive stimuli to be avoided rather than valued stimuli to be approached; procrastinating on difficult tasks in favor of easy ones can be a really expensive and error-prone way to program. You want to use the context that’s already loaded in your head efficiently and solve central uncertainties before peripheral ones.)
(Of course I’m conveniently leaving out lots of embarrassing failures… for example, I once broke almost all of the clickable buttons on our website for several hours before someone noticed. I don’t think I ever heard of anyone else screwing up this bad. In my post-mortem I determined that my feeling that I had already done more than could reasonably expected on this particular task caused me to stop working when I ran out of steam instead of putting in the necessary effort to make sure my solution was totally correct. It took me a while to get good at writing low-bug code and arguably I’m still pretty bad at it.)
Do you have any thoughts on the best routes to push on this?
Hm, I guess one idea might be to try to obtain the ear of leading thinkers in the programming world--Joel Spolsky and Jeff Atwood types--to blog more about the virtues of bug prevention and approaches to it. My impression is that leading bloggers have more influence on software development culture than top CTOs or professors. But it wouldn't necessarily need to be leading thinkers spearheading this; anyone can submit a blog post to Hacker News. I think what you'd want to do is compile a big list of devastating bugs and then sell a story that as software eats the world more and more, we need to get better at making it reliable.
These are quick notes on an idea for an indirect strategy to increase the likelihood of society acquiring robustly safe and beneficial AI.
Motivation:
Most challenges we can approach with trial-and-error, so many of our habits and social structures are set up to encourage this. There are some challenges where we may not get this opportunity, and it could be very helpful to know what methods help you to tackle a complex challenge that you need to get right first time.
Giving an artificial intelligence good values may be a particularly important challenge, and one where we need to be correct first time. (Distinct from creating systems that act intelligently at all, which can be done by trial and error.)
Building stronger societal knowledge about how to approach such problems may make us more robustly prepared for such challenges. Having more programmers in the AI field familiar with the techniques is likely to be particularly important.
Idea: Develop methods for training people to write code without bugs.
Trying to teach the skill of getting things right first time.
Writing or editing code that has to be bug-free without any testing is a fairly easy challenge to set up, and has several of the right kind of properties. There are some parallels between value specification and programming.
Set-up puts people in scenarios where they only get one chance -- no opportunity to test part/all of the code, just analyse closely before submitting.
Interested in personal habits as well as social norms or procedures that help this.
Daniel Dewey points to standards for code on the space shuttle as a good example of getting high reliability code edits.
How to implement:
Ideal: Offer this training to staff at software companies, for profit.
Although it’s teaching a skill under artificial hardship, it seems plausible that it could teach enough good habits and lines of thinking to noticeably increase productivity, so people would be willing to pay for this.
Because such training could create social value in the short run, this might give a good opportunity to launch as a business that is simultaneously doing valuable direct work.
Similarly, there might be a market for a consultancy that helped organisations to get general tasks right the first time, if we knew how to teach that skill.
More funding-intensive, less labour intensive: run competitions with cash prizes
Try to establish it as something like a competitive sport for teams.
Outsource the work of determining good methods to the contestants.
This is all quite preliminary and I’d love to get more thoughts on it. I offer up this idea because I think it would be valuable but not my comparative advantage. If anyone is interested in a project in this direction, I’m very happy to talk about it.