How did safety engineering get invented for different disciplines, and how do their invention relate to engineering and theory?

Inspired by davidad's tweets: 1, 2, 3

It seems commonsense that a deeper (theoretical) understanding helps both engineering as well as safety engineering. Which one do you think does theory help more? And which development helped grow theory research more?

My intuition is that:

1. First we started building something by trial-and-error, empirical results.
2. We formulated some safety best practices. But they are all heuristics from the trial-and-error.
3. Then we started gaining theoretical understanding of what we are doing.
4. Only then do we become able to advance "safety engineering".
5. At the same time, we also get much better at building that thing - much better at engineering.

How well does this mesh with real-life? In the bridges' case, safety engineering was invented separately, well after we understood how to build bridges - and well after we built a lot of bridges. The pioneers in safety engineering oft have formal math background. This seems to match the intuition above.

That said -

• We did build a lot of bridges, and a lot of them failed, before safety engineering came about. And how much did theories for safety engineering help with bridge capability?
• Did the field advance by novel theory works? Or was it more about the application of existing theories?
• Related to that question is: did safety engineering require an entirely different set of theories that have little to do with bridge capability? (This seems obviously true to me: for example, environmental wear-and-tear and the process of metal rusting does not affect capability, but we need to understand them for safety.)

is this part of the reason so many AI researchers think it's cool and enlightened to not believe in highly general architectures

I do hear No Free Lunch theorem get thrown around when an architecture fails to solve some problem which its inductive bias doesn't fit. But I think it's just thrown around as a vibe.

Love the post! http://pragmaticaisafety.com/ is down for me right now though. Do the authors still endorse this sequence?

Just spent one year in academia; my experience trying to talk to researchers about AGI match what Dan wrote about.

(ramblingly) Does the No Free Lunch Theorem imply that there's no one single technique that would always work for AGI alignment? Initial thought is probably not, because the theorem states that the performance of all optimization algorithms are identical across all possible problems. However,  AGI alignment is a subset of these problems.

GPT-4 can do math because it has learned particular patterns associated with tokens, including heuristics for certain digits, without fully learning the abstract generalized pattern.

This finding seems consistent with some literatures, such as this where they found that if the multiplication task has an unseen computational graph, then performance deteriorates rapidly. Perhaps check out the keyword "shortcut learning" too.

Game Design

The videos under this category fits better the label "game development" instead. Game Design is more focused on designing rules, mechanics, sometimes narratives, instead of programming.

Is the event happening on June 11th or July 9th?

I think there should be more effort into researching the limits of controllability for self-improving machines. That aspect of rapid self improvement seems pretty important to me since it's there regardless of which architecture we use to get to the singularity. If the singularity is dangerous no matter how we get there, or how aligned our first try is, then, [clears throat and raises sign] don't build AGI?

I bought the device and watched Interstellar on top of Mt. Hood with the stars as the background. It was a phenomenal experience. That said, having to bear the weight of the device for 2.5 hours, and other limits such as FOV & lens glare makes me hesitant to say movie's the one killer app right now. I don't think there is a killer app yet - Apple wants us to come in for that.