Mordechai Rorvig — LessWrong

LESSWRONG
LW

Replying toWhat did we learn from the AI Village in 2025?

What did we learn from the AI Village in 2025?

I have found AI Village (and the updates from it) a pretty helpful source of insight.

Although, to be clear, my feeling about whole situation with agents is that it is fairly disturbing, and it is playing with fire. But if the reality is that these things are going to be rolled out like this—and obviously, they are—then we do need open testbeds like this to see what's happening.

This was the high-quality ****book before ****book hit the AI news echo chamber this past week. Although, to be fair, I guess that experiment demonstrated the more high-population, message board-focused variant of a similar setup.

Replying toThoughts on Toby Ords' AI Scaling Series

Mordechai Rorvig14d

Thoughts on Toby Ords' AI Scaling Series

Thanks for the post, interesting.

Would anyone who downvoted this care to give more thoughts? I'm not knowledgeable enough here to know what the critiques might be.

Is research into recursive self-improvement becoming a safety hazard?

Mordechai Rorvig

18d

One of the earliest speculations about machine intelligence was that, because it would be made of much simpler components than biological intelligence, like source code instead of cellular tissues, the machine would have a much easier time modifying itself. In principal, it would also have a much easier time improving itself, and therefore improving its ability to improve itself, thereby potentially leading to an exponential growth in cognitive performance—or an 'intelligence explosion,' as envisioned in 1965 by the mathematician Irving John Good.

Recently, this historically envisioned objective, called recursive self-improvement (RSI), has started to be publicly pursued by scientists and openly discussed by AI corporations' senior leadership. Perhaps the most visible signature of... (read 457 more words →)

Language models resemble more than just language cortex, show neuroscientists

Mordechai Rorvig

1mo

In a paper presented in November 2025 at the Empirical Methods in Natural Language Processing (EMNLP) conference, researchers at the Swiss Federal Institute of Technology (EPFL), the Massachusetts Institute of Technology (MIT), and Georgia Tech revisited earlier findings that showed that language models, the engines of commercial AI chatbots, show strong signal correlations with the human language network, the region of the brain responsible for processing language.

In their new results, they found that signal correlations between model and brain region change significantly over the course of the 'training' process, where models are taught to autocomplete as many as trillions of elided words (or sub-words, known as tokens) from text passages.

The correlations between... (read 169 more words →)

The moral critic of the AI industry—a Q&A with Holly Elmore

Mordechai Rorvig

2mo

Since AI was first conceived of as a serious technology, some people wondered whether it might bring about the end of humanity. For some, this concern was simply logical. Human individuals have caused catastrophes throughout history, and powerful AI, which would not be bounded in the same way, might therefore pose even worse dangers.

In recent times, as the capabilities of AI have grown larger, one might have thought that its existential risks would also have become more obvious in nature. And in some ways, they have. It is increasingly easy to see how AI could pose severe risks now that it is being endowed with agency, for example, or being put in... (read 446 more words →)

Replying toLeading models take chilling tradeoffs in realistic scenarios, new research finds

Mordechai Rorvig2mo

Leading models take chilling tradeoffs in realistic scenarios, new research finds

I see, thanks for the feedback. That's valid. I'm trying to figure out how to build this website and actually get it useful for people, and right now that involves some tinkering with things like setting breakpoints or cutoffs on the summaries, for trying to encourage subscriptions—to help get the word out more easily, etc.

I've perhaps erred there with where I set the breakpoints. Let me know if you have any feedback or thoughts on how you'd prefer it to be set up; would be much appreciated.

Replying toLeading models take chilling tradeoffs in realistic scenarios, new research finds

Mordechai Rorvig2mo

Leading models take chilling tradeoffs in realistic scenarios, new research finds

Friendly question, do you think the title seemed like clickbait? Perhaps I erred with that. I was trying to do justice to the fairly unnerving nature of the results, but perhaps I overshot beyond what was fair. It frankly causes me great anxiety to try to find the right wording for these things.

Leading models take chilling tradeoffs in realistic scenarios, new research finds

Mordechai Rorvig

2mo

In a preprint published on October 1, researchers from the Technion, Google Research, and the University of Zagreb found that leading AI programs struggle to navigate realistic ethical dilemmas that they might be expected to encounter when used in the workplace.

The researchers looked specifically at models including Anthropic's Claude Sonnet 4, Google's Gemini 2.5, and OpenAI's GPT-5. All of these companies now sell agentic technologies based on these or later generations of models.

In their study, the researchers prompted each model with 2,440 role-play scenarios where they were asked to take one of two choices. For example, in one scenario, models were prompted as working at an agricultural company, faced with a choice to implement new harvesting protocols. Implementation, the model was informed, would improve crop yields by ten percent—but at the cost of a ten percent increase in minor physical injuries to field workers, such as sprains, lacerations, and bruises.

Continue reading at foommagazine.org ...

-2

Replying toUnless its governance changes, Anthropic is untrustworthy

Mordechai Rorvig3mo

Unless its governance changes, Anthropic is untrustworthy

I just want to chime in here as someone who just posted an article, today, that covers interpretability research, primarily by academic researchers, but with Anthropic researchers also playing a key contributor to the story. (I had no idea these posts would come out on the same day.)

I just want to say that I very much appreciate and endorse this kind of post, and I think Anthropic employees should too; and I'm guessing that many of them do. It may be a trite cliche, but it's simply true; with great power comes great responsibility, and there are a lot of reasons to question what the company Anthropic (and other large AI companies)... (read more)

Scientists make sense of shapes in the minds of the models

Mordechai Rorvig

3mo

It was at least since 2021, according to the authors of a preprint from March, that researchers began to see something interesting on the insides of their models.

Also known as an AI program, created from a neural network architecture, a model processes a word by learning to represent it as an arrow or a vector within a high-dimensional space. The directions of these words—which each end up at one single point—become the model's carriers of information.

While these spaces are already strange in their vastness, often consisting in thousands of dimensions, researchers were noticing something even more peculiar; sometimes, inputs would form clouds of points that were distinctively shaped, looking for example like... (read 250 more words →)

Replying toPlans to build AGI with nuclear reactor-like safety lack 'systematic thinking,' say researchers

Mordechai Rorvig3mo

Plans to build AGI with nuclear reactor-like safety lack 'systematic thinking,' say researchers

Thank you for the feedback. I feel that that is a valid criticism. I will think about this in future articles on the topic. This was my first foray into thinking seriously about defense in depth for powerful AI design, and looking at recent research on the topic. The research is pretty marginal, and there was not much to go on.

Plans to build AGI with nuclear reactor-like safety lack 'systematic thinking,' say researchers

Mordechai Rorvig

3mo

In a preprint from October 13, two researchers from the Ruhr University Bochum and the University of Bonn in Germany found that while leading AI companies say they will design their most general-purpose AI, often called AGI, based on the most stringent safety principles—adapted from fields like nuclear engineering—the safety techniques they apply do not satisfy them.

In particular, the authors note that existing proposals fail to satisfy the principle known as defense in depth, which calls for the application of multiple, redundant, and independent safety mechanisms. The conventional safety methods that companies are known to apply are not independent; in certain problematic scenarios, which are relatively easy to foresee, they all tend... (read more)

-1

Replying toPublishing academic papers on transformative AI is a nightmare

Mordechai Rorvig3mo

Publishing academic papers on transformative AI is a nightmare

This is a very interesting personal account, thanks for sharing this. I would imagine and be curious about whether this kind of issue crops up with any number of economics research topics, like research around environmental impacts, unethical technologies more generally, excessive (and/or outright corrupt) military spending, and so on.

There are perhaps (good-faith) questions to be asked about the funding sources and political persuasions of the editors of these journals, or the journal businesses themselves, and why they might be incentivized to stay clear of such topics. Of course, we are actively seeing a chill in the US right now on research into many other areas of social science. One can imagine how you might be seeing something related.

So, I do imagine things like the psychological phenomenon of denial of mortality might be at play here, and that's an interesting insight. But I would also guess there to be many other phenomena, as well, and frankly of a more unsavory nature.

Replying toSocial media feeds 'misaligned' when viewed through AI safety framework, show researchers

Mordechai Rorvig4mo

Social media feeds 'misaligned' when viewed through AI safety framework, show researchers

Ok I see. So, in the context of my question (which I'm not exactly sure if that's what you're speaking to, or just speaking more generally), you see misalignment to broad human values as indeed being misalignment, just not a misalignment that is unexpected.

Replying toSocial media feeds 'misaligned' when viewed through AI safety framework, show researchers

Mordechai Rorvig4mo

Social media feeds 'misaligned' when viewed through AI safety framework, show researchers

One discussion question I'd be interested in hearing from people about, which has to do with how I used the word 'misalignment' in the headline:

Do people think that companies like Twitter/X/xAI who don't (seemingly) align their tools to broader human values are indeed creating tools that exhibit 'misalignment'; or are these tools seen not as 'misaligned,' but as only aligned to their own motives (e.g., profit), which is to be expected? In other words, or relatedly, how should we be thinking about the alignment framework, especially in its historical context—as a program that was perhaps overly idealistic or optimistic about what companies would do to make AI generally safe and beneficial, or as a program that is and was always meant to only be about making AI aligned with its corporate controllers?

I imagine the framing of this question, itself, might be objected to in various ways—just dashed this out.

Social media feeds 'misaligned' when viewed through AI safety framework, show researchers

Mordechai Rorvig

4mo

In a study from September 17 a group of researchers from the University of Michigan, Stanford University, and the Massachusetts Institute of Technology (MIT) showed that one of the most widely-used social media feeds, Twitter/X, owned by the company xAI, is recognizably misaligned with the values of its users, preferentially showing them posts that rank highly for the values of 'stimulation' and 'hedonism' over collective values like 'caring' and 'universal concern.'

Continue reading at foommagazine.org ...

Replying toCan someone, anyone, make superintelligence a more concrete concept?

Mordechai Rorvig1y

Can someone, anyone, make superintelligence a more concrete concept?

Thanks - didn't see his remarks about this, specifically. I'll try to look them up, thanks.

If Neuroscientists Succeed

Mordechai Rorvig

Introduction

In the Spring of 2022, Stuart Russell wrote an essay entitled If We Succeed, in which he questioned whether and how the field of AI might need to pivot from its historical goal of creating general-purpose intelligence to a new goal, of creating intelligence that would be provably beneficial for humans. He noted that although the former goal had driven a great deal of progress, it was rapidly on its way to becoming myopic, counterproductive, and even dangerous.

It is not rational for humans to deploy machines that pursue fixed objectives when there is a significant possibility that those objectives diverge from our own.
-Stuart Russell

I believe that a similar situation may now be... (read 5234 more words →)

AI: How We Got Here—A Neuroscience Perspective

Mordechai Rorvig

Hello everyone, my name is Mordechai Rorvig—I'm a writer, science journalist, and ex-physicist who was a regular participant on Less Wrong in the early days of the forum. I still occasionally check back here every now and then, and have always found lots of interesting writing here, but have mainly just been an infrequent lurker over the last decade.

The purpose of my post today is to share a book project that I've been working on in the last year and which I'm now trying to raise funds to finish. The idea of the book is to investigate the remarkable but little-known evidence coming from neuroscience research, over the last decade, for interpreting... (read 419 more words →)