Quick Takes

I wish the median AI safety researcher were much more ambitious with the problems they choose to tackle. Unfortunately, job and funding incentives are biased against research ambition. I’d thus like to celebrate the people who have taken risks to pursue ambitious AI safety research directions that have not panned out (or have yet to pan out!). These people will not have received riches and accolades from the field, or even their close peers. For much of the work that falls in this category, it might be obvious to many others at the time that it isn’t going to lead anywhere. Unless their efforts were likely to be actively harmful, I still want to celebrate these people for their courage, and for pushing against the incentive gradient. I was planning on making a long list of work that I thought fell into this category, but quickly felt uncomfortable including and excluding people’s work without more thorough analysis that I didn’t think was worth it. Maybe some day I will. If you think that you’ve done work that falls into this category, thank you for your +EV. Humanity is grateful for your efforts.

In my recent analysis of AI usage, I’ve read (and typed) “genuinely” so many times it stopped being a real word to me. There’s a metaphor in there, somewhere.

i feel like the fundamental mistake the project of rationality made was that "cognitive biases" is not in practice the right way to think about the way humans are irrational if your goal is to be very instrumentally rational. one hypothesis is the correct frame is to first deeply understand how the emotional system works, and then to think about ways to master that system to achieve rationality. (yes, i know that buried somewhere in the sequences it says something like "humans aren't ideal intelligences with cognitive biases bolted on. we are the cognitive biases, they are just trying to approximate rationality".)

I strongly suspect significant fractions of Magnifica Humanitas, the Pope's new encyclical on AI, is AI-generated or AI-assisted. Evidence includes: * Pangram (Some examples include paragraphs 7-8. To a lesser degree, paragraphs 120-128) * * I think you notice it too if you read it compared to other paragraphs * Also on a statistical level, the em-dash ("—") was used 127 times in the recent encyclical. * * Checking the last five encyclicals, it varies from 0 times to 23 times previously. * "genuinely" was used 9 times and "genuine" overall 22 times in today's encyclical. * * compared to 0 and 5 times, respectively, in the previous encyclical from 2024 (which appears to be of similar length). In total, maybe 10-15% of the final draft was written by AI. Because the level of AI-writing varies significantly from section to section (from ~0% to ~100%), and because the styles are quite different, I suspect some cardinals who ghostwrote/contributed to it use AI-assistance much more than other people.

"—and forgive us our sims, as we forgive those who sim against us—"

When evolutionary pressure is too high, you may get a population that is perfectly optimised for its current environment. Because of goodhart’s law, this means that the population is very vulnerable to a change in environment, such as a new virus, which may spread through the population and wipe it all out. Therefore a certain amount of slack/diversity within the population is adaptive in the face of Knightian uncertainty about future events.

Reading a bit about "natural language autoencoder" technique. It seems they are treating the activations as the distribution to be encoded into a latent space, and using natural language as the latent space. Given my experience with autoencoders, the latent space is kinda arbitrary, determined by the relationship of locations in the distribution to one another, and the initialization. So it seems clear to me that the encoder would learn to output text that the decoder can use to reconstruct the activation, it just isn't clear to me why that text should have any meaning to humans reading it that necessarily relates to what the activation means. Does anyone know more about the technique who could answer this question, or verify my intuition that this technique doesn't guarantee an accurate description of latent semantics?