John Carpenter's The Thing (1982) is a masterclass in practical rationality, a cornerstone of effective decision-making under uncertainty—a concept deeply valued by the LessWrong community. The film’s narrative hinges on a group of Antarctic researchers encountering a shape-shifting alien capable of perfectly imitating its hosts, forcing them to confront dire stakes with limited information. Their survival depends on their ability to reason under pressure, assess probabilities, and mitigate catastrophic risks, making the movie a compelling example of applied rationality.
Key Lessons in Practical Rationality:
One needs only to read 4 or so papers on category theory applied to AI to understand the problem. None of them share a common foundation on what type of constructions to use or formalize in category theory. The core issue is that category theory is a general language for all of mathematics, and as commonly used just exponentially increase the search space for useful mathematical ideas.
I want to be wrong about this, but I have yet to find category theory uniquely useful outside of some subdomains of pure math.
"That is, where are the Matt Levines of, say, chemistry or drug development1,"
You are looking for Derek Lowe's "In the pipeline." It appears on hacker news occasionally.
I am looking for results showing that various approaches to provable safety are impossible or that such proofs are of a particular complexity class. I have Yampolskiy's paper "Impossibility Results in AI: A Survey," but I am looking for more targeted results that would help guide research into provable safety. Many of the results seem to be from the Computability theory and are so general that they are not that useful.
A theorem stating that one cannot formally verify all possible mathematical proofs does little to say about which constrained system can be verified.
I would also be interested in impossibility results in non-trivial toy models of alignment problems (RL environments) that are not simply the corollary of the much more general theorems.
Lastly, given everything written above, I would also like any other reference/information that a person may reasonably expect me to find interesting and generally related.
The crux of these types of arguments seems to be conflating the provable safety of an agent in a system with the expectation of absolute safety. In my experience, this is the norm, not the exception, and needs to be explicitly addressed.
In agreement with what you posted above, I think it is formally trivial to construct a scenario in which a pedestrian jumps in front of a car, making it provably impossible for a vehicle to stop in time to avoid a collision using high school physics.
Likewise, I have the intuition that AI safety, in general, should have various "no-go theorems" about unprovability outside a reasonable problem scope or that finding such proofs would be np-hard or worse. If you know of any specific results( outside of general computability theory) , could you please share them? It would be nice if the community could avoid falling into the trap of trying to prove too much.
(Sorry if this isn't the correct location for this post.)
A general meetup without a single topic. This meeting will also focus on sorting out the logistics of future events in Houston.
I think it is very unclear that we want fewer 'maladaptive' people in the world in the sense that we can measure with personality traits such as the big five.
Would reducing the number of outliers in neuroticism also reduce the number of people emotionally invested in X-risk? The downstream results of such a modification do not seem to be clear.
It seems like producing a more homogeneous personality distribution would also reduce the robustness of society.
The core weirdness of this post to me is that the first conditioning on LLM/AI does all the IQ tasks, and humans are not involved in auditing that system in a case where high IQ is important. Personally, I feel like assuming that AI does all the IQ tasks is a moot case. We are pets or dead in that case.