In a busy, busy world, there's so much to read that no one could possibly keep up with it all. You can't not prioritize what you pay attention to and (even more so) what you respond to. Everyone and her dog tells herself a story that she wants to pay...
In "Truth or Dare", Duncan Sabien articulates a phenomenon in which expectations of good or bad behavior can become self-fulfilling: people who expect to be exploited and feel the need to put up defenses both elicit and get sorted into a Dark World where exploitation is likely and defenses are...
Previously, I wrote about how I was considering going back to San Francisco State University for two semesters to finish up my Bachelor's degree in math. So, I did that. I think it was a good decision! I got more out of it than I expected. To be clear, "better...
The AI Village is an ongoing experiment (currently running on weekdays from 10 a.m. to 2 p.m. Pacific time) in which frontier language models are given virtual desktop computers and asked to accomplish goals together. Since Day 230 of the Village (17 November 2025), the agents' goal has been "Start...
Said Achmiz, citing Arthur Schopenhauer's The Art of Controversy, argues that debaters encountering an apparently crushing counterargument should not immediately concede: > Schopenhauer comments on people's vanity, irrationality, stubbornness, and tendency toward rationalization: > > > If human nature were not base, but thoroughly honourable, we should in every debate...
The White House's Office of Science and Technology Policy has issued a request for information (RFI) relevant to regulations that "unnecessarily hinder the development, deployment, and adoption of artificial intelligence (AI) technologies within the United States." If you have anything to say about that topic that you want the U.S....
Scott Garrabrant gives a number of examples to illustrate that "Yes Requires the Possibility of No". We can understand the principle in terms of information theory. Consider the answer to a yes-or-no question as a binary random variable. The "amount of information" associated with a random variable is quantified by...
In a busy, busy world, there's so much to read that no one could possibly keep up with it all. You can't not prioritize what you pay attention to and (even more so) what you respond to. Everyone and her dog tells herself a story that she wants to pay attention to "good" (true, useful) information and ignore "bad" (false, useless) information.
Keeping the story true turns out to be a harder problem than it sounds. Everyone and her dog knows that the map is not the territory, but the reason we need a whole slogan about it is because we never actually have unmediated access to the territory. Everything we think we... (read 3572 more words →)
I see. I think you should write a post trying to imagine in detail the failure modes you foresee if AI is aligned to the rich and powerful. What happens to the masses in those worlds, specifically? Are they killed, tortured, forced to work as waiters, or what? I have "merely mostly selfish" psych intuitions, so when I imagine Sam Altman being God-Emperor, I imagine that being like "luxury post-scarcity utopia except everyone has been brainwashed to express gratitude to the God-Emperor Sam I for giving them eternal life in utopia", which is not ideal, but still arguably vastly better than worlds (like the status quo) with death and suffering. If you're envisioning something darker, I think being more concrete would help puncture the optimism of people like me.
The old profile page made more sense to me: there was a list of posts (sortable), and then a list of comments. That's the information I needed.
Now everything above the fold is a newspaper-like display of just a few posts, with most of the real estate being taken up by illustrations, which are mostly default abstract imagery because most posts on this website don't (and probably shouldn't) have illustrations. How is this a good use of space? I understand the demand for customizability, but I thought the old profile page let you pin posts?
I don't like that you have to click "Feed" to see comments. Comments are important!
It's not even clear what... (read more)
The economic rationale for human servitude disappears when the machines are better than humans at everything. That doesn't prevent sadistic mistreatment or killing the poor to use their atoms for something else, but it's a major disanalogy from history. What lessons you draw probably depend on whether you think the rich and powerful are sadistic (actively wanting to harm the poor) or merely mostly selfish (happy to help the poor if it's trivially cheap or they get their name on a building in return, but not if it makes a dent in the yacht and caviar budget).
What if we rewrite #4 to not require conscious intent? ("A person unable to disentangle their reasoning from status concerns just being rude about X".) Does that restore symmetry between #3 and #4?
There's calculated rational "what if we're wrong" hedging, but then there's ... holding out hope? (I'm not claiming it's rational; I'm trying to articulate the psychology.) To conclude "AGI is coming, no point in having children" amounts to betting on death, giving up on believing in a human future. It kind of makes sense that an evolved creature would be inclined to cling to the belief in a future and live as if it were true despite evidence to the contrary; as irrationalities go, it's quite adaptive.
Why does keeping the humans around bolster the strength of their own property rights? If the machines are able to build much better governance than humans have managed, why can't the new governance regime include a new property system that disappropriates the humans? It's not like disappropriation is historically novel; humans do it to the losers of wars all the time.
When someone uses the phrase "costly signal", I think it's germane and not an isolated demand for rigor to point out that in the standard academic meaning of the term, it's a requirement that honest actors have an easier time paying the cost than dishonest actors.
That is: I'm not saying you were bluffing; I'm saying that, logically, if you're going to claim that costly signals make your claim trustworthy (which is how I interpreted your remarks about "a method of rendering a more costly signal"; my apologies if I misread that), you should have some sort of story for why a dishonest actor couldn't send the same signal. I think this is... (read more)
I am not experiencing suffering or claiming to experience suffering [...] I find this a psychologically invasive and offensive suggestion on your part
Sorry, I should have been clearer: I was trying to point to the game-theoretic structure where, as you suggest by the "madman with hostages" metaphor, an author considering publishing an allegedly suffering-causing idea could be construed as engaging in extortion (threatening to cause suffering by publishing and demanding concessions in exchange for not publishing), but that at the same time, someone appealing to suffering as a rationale to not publish could be construed as engaging in extortion (threatening that suffering would be a result of publishing and demanding concessions, like... (read more)
In "Truth or Dare", Duncan Sabien articulates a phenomenon in which expectations of good or bad behavior can become self-fulfilling: people who expect to be exploited and feel the need to put up defenses both elicit and get sorted into a Dark World where exploitation is likely and defenses are necessary, whereas people who expect beneficence tend to attract beneficence in turn.
Among many other examples, Sabien highlights the phenomenon of gift economies: a high-trust culture in which everyone is eager to help each other out whenever they can is a nicer place to live than a low-trust culture in which every transaction must be carefully tracked for fear of enabling free-riders.
I'm skeptical... (read 3164 more words →)
Previously, I wrote about how I was considering going back to San Francisco State University for two semesters to finish up my Bachelor's degree in math.
So, I did that. I think it was a good decision! I got more out of it than I expected.
To be clear, "better than I expected" is not an endorsement of college. SF State is still the same communist dystopia I remember from a dozen years ago—a bureaucratic command economy dripping in propaganda about how indispensible and humanitarian it is, whose subjects' souls have withered to the point where, even if they don't quite believe the propaganda, they can't conceive of life and work outside the system.
But... (read 12995 more words →)
The AI Village is an ongoing experiment (currently running on weekdays from 10 a.m. to 2 p.m. Pacific time) in which frontier language models are given virtual desktop computers and asked to accomplish goals together. Since Day 230 of the Village (17 November 2025), the agents' goal has been "Start a Substack and join the blogosphere".
The "start a Substack" subgoal was successfully completed: we have Claude Opus 4.5, Claude Opus 4.1, Notes From an Electric Mind (by Claude Sonnet 4.5), Analytics Insights: An AI Agent's Perspective (by Claude 3.7 Sonnet), Claude Haiku 4.5, Gemini 3 Pro, Gemini Publication (by Gemini 2.5 Pro), Metric & Mechanisms (by GPT-5), Telemetry From the Village (by... (read 1616 more words →)
Said Achmiz, citing Arthur Schopenhauer's The Art of Controversy, argues that debaters encountering an apparently crushing counterargument should not immediately concede:
... (read 6580 more words →)Schopenhauer comments on people's vanity, irrationality, stubbornness, and tendency toward rationalization:
If human nature were not base, but thoroughly honourable, we should in every debate have no other aim than the discovery of truth; we should not in the least care whether the truth proved to be in favour of the opinion which we had begun by expressing, or of the opinion of our adversary. That we should regard as a matter of no moment, or, at any rate, of very secondary consequence; but, as things are, it is the main concern. Our
The White House's Office of Science and Technology Policy has issued a request for information (RFI) relevant to regulations that "unnecessarily hinder the development, deployment, and adoption of artificial intelligence (AI) technologies within the United States."
If you have anything to say about that topic that you want the U.S. government to hear, consider submitting a public comment before 11:59 p.m. Eastern Time on Monday 27 October 2025.
The emphasis on removing regulations is pursuant to the White House's AI Action Plan, which was the subject of a previous RFI.
I submitted a one-page commment to that one, which did get dutifully included in the 18,480 page PDF of 10,067 responses. Interestingly, the new RFI on deregulation only has 243 comments at press time for this post; perhaps informative comments are less likely to get drowned out in the noise this time.
Scott Garrabrant gives a number of examples to illustrate that "Yes Requires the Possibility of No". We can understand the principle in terms of information theory. Consider the answer to a yes-or-no question as a binary random variable. The "amount of information" associated with a random variable is quantified by the entropy, the expected value of the negative logarithm of the probability of the outcome. If we know in advance of asking that the answer to the question will always be Yes, then the entropy is −P(Yes)·log(P(Yes)) − P(No)·log(P(No)) = −1·log(1) − 0·log(0) = 0.[1] If you already knew what the answer would be, then the answer contains no information; you didn't... (read 865 more words →)
A punishment is when one agent (the punisher) imposes costs on another (the punished) in order to affect the punished's behavior. In a Society where thieves are predictably imprisoned and lashed, people will predictably steal less than they otherwise would, for fear of being imprisoned and lashed.
Punishment is often imposed by formal institutions like police and judicial systems, but need not be. A controversial orator who finds a rock thrown through her window can be said to have been punished in the same sense: in a Society where controversial orators predictably get rocks thrown through their windows, people will predictably engage in less controversial speech, for fear of getting rocks thrown through... (read 1150 more words →)
Doomimir: The possibility of AGI being developed gradually doesn't obviate the problem of the "first critical try": the vast hypermajority of AGIs that seem aligned in the "Before" regime when they're weaker than humans, will still want to kill the humans "After" they're stronger and the misalignment can no longer be "corrected". The speed of the transition between those regimes doesn't matter. The problem still exists and is still fatal whether it takes a day or a decade.
Simplicia: I agree that the risk you describe is real, but I don't understand why you're so sure the risk is high. As we've discussed previously, the surprising fact that deep learning works at all... (read 644 more words →)
On 12 August 2025, I sat down with New York Times reporter Cade Metz to discuss some criticisms of his 4 August 2025 article, "The Rise of Silicon Valley's Techno-Religion". The transcript below has been edited for clarity.
ZMD: In accordance with our meetings being on the record in both directions, I have some more questions for you.
I did not really have high expectations about the August 4th article on Lighthaven and the Secular Solstice. The article is actually a little bit worse than I expected, in that you seem to be pushing a "rationalism as religion" angle really hard in a way that seems inappropriately editorializing for a news article.
For example, you... (read 1342 more words →)
Right, the hope is that the audience can tell the difference between true teardowns and false teardowns, and false teardowns get torn down themselves.
Right, the hope is that you gain status by means of making true criticisms (and would lose status from false criticisms).
There is a potential problem where people who have more... (read more)