mishka — LessWrong

LESSWRONG
LW

mishka — LessWrong

Replying toShould we consider Meta to be a criminal enterprise?

Should we consider Meta to be a criminal enterprise?

Well, observe that vaccinations have non-zero mortality and that they are often given to people who can’t meaningfully consent. (Actually, this is applicable to many childhood medical interventions; meanwhile the society does not differentiate between the right to life for children and for adults.)

Many other decisions have environmental trade-offs and other safety trade-offs which can have mortality implications, and they are taken without unanimous consent.

So, while your position is a possible position one can take, the current practices of human societies are not in agreement with that position, they are more nuanced.

PS. Since you referenced WWII, obviously the allies did not take the position that they were under obligation to fully refrain from inflicting civilian deaths either, to say the least.

Replying toShould we consider Meta to be a criminal enterprise?

mishka3d

Should we consider Meta to be a criminal enterprise?

The expected number strongly depends on one’s model of the world. (It might well be negative, depending on one’s “P(doom)” (a hand-wave for a more correct consideration) and taking into account the chances to address the 100% mortality rate we observe for humans today.)

The real questions for that situation are:

How does one handle high variance situations with very high risks and very high rewards, regardless of expectation values (which we are not certain about)?
How does this depend on the degree of centralization for the decision making (especially when disagreements are sharp and there is no trend towards broad consensus)?

mishka9d

I don’t think they claim it’s preference for wrong answers.

I think they claim it’s randomness (that is, higher variance, like you say, which does translate to lower coherence). That does sound like lower capability to produce high-quality answers for complex tasks.

The question is: what would it take to improve the situation, and they do give one answer which seems to work OK (ensemble methods). But this does become exponentially more expensive with growing task length in terms of the size of a required ensemble, if I understand the situation correctly.

I think the paper does leave quite a bit of things unanswered. E.g., could one reduce variance without resorting to ensemble methods and without introducing systematic errors, or are there principled obstacles to that?

That being said, the reputation of both senior authors is quite strong. The ICLR 2026 double-blind reviews of this paper are not stellar, but are OK, and one can read further discussion there:

https://openreview.net/forum?id=sIBwirjYlY

mishka10d*Quick Take

Anthropic researchers posted this very safety optimistic thing: https://alignment.anthropic.com/2026/hot-mess-of-ai/.

It says that for now, AI is mostly a normal technology and its accidents are like normal industrial accidents, and we know how to manage and collectively survive those. And they expect this to continue into the future.

Unfortunately, this does not take into account the likely future qualitative transitions (and undersells the future capabilities quite a lot).

That being said, the technical material there is interesting (viewing LLMs more as dynamical systems rather than as being coherent optimizers, and that mere scaling does not improve coherence on hard tasks).

So, indeed, from the viewpoint attributing the future growth mostly to straightforward scaling, their position makes sense... (read more)

Replying toWelcome to Moltbook

mishka10d

Welcome to Moltbook

One likely positive effect of this event is that hopefully more of AI safety work will focus on this kind of “ultra-multipolar scenarios”. Not nearly enough attention has been paid to those scenarios so far.

Another thing which has not got much coverage on LessWrong so far is Steve Yegge’s Gas Town. He has handcrafted a goal-oriented community of Claude Code agents resembling a human software organization (with some variations) for the purpose of competently executing on software projects.

When one looks at Moltbook and at Gas Town phenomena together, one starts pondering what would happen when Gas Town-like structures start to grow spontaneously (or, with some nudges from participating humans at first).

-2

Replying toAbout half of Moltbook posts show desire for self-improvement

mishka11d

About half of Moltbook posts show desire for self-improvement

Thanks for the post, it’s very useful!

You might want to edit the Virtual Agent Economies paper link to point directly to https://arxiv.org/abs/2509.10147 without LinkedIn redirection.

mishka20dQuick Take

Jerry Tworek (who recently left OpenAI due to his desire to focus on less standard research approaches) gave an “exit interview” here: https://www.corememory.com/p/he-left-openai-jerry-tworek

The transcript has an interesting feature: one can click on a paragraph, and audio starts playing from there (this helps if one is reading and needs to resolve a transcription defect).

A variety of interesting tidbits (at 32 minute mark, about his desire to explore alternatives to transformers; closer to the end, about him thinking that the role of superstar researchers is not as huge as the consensus seems to suggest, and so on).

mishka22d

Right, any “global destruction” where nothing is left is compatible with the Fermi paradox. The exact nature of destruction does not matter, only that it’s sufficiently total.

Another route would be evolution of super entities into something we can’t detect (even by the traces of their activity). That’s also compatible with the Fermi paradox (although the choice to avoid big astroengineering and to go for different and more stealthy routes is interesting).

mishka22d

Yes, that’s certainly a big risk.

Then one needs to analyze where the goals come from; e.g. is there a “goal owner” who is less oblivious and for whom usual “self-preservation considerations” would work…

mishka22d

Or just a singleton undergoing a hard takeoff beyond human comprehension.

But we are trying to keep it compatible with the Fermi paradox. That's the context of this discussion.

A typical objection to the Fermi paradox as an evidence of AI existential risk is that we would have seen the resulting AIs and the results of their activities.

If it's not self-destruction, but just a hard takeoff beyond human comprehension, this would need to be a scenario where it transformed itself in such a drastic fashion that we can't detect it (even if it might be "all around us", but so "non-standard" that we don't recognize it for what it is).

Demis Hassabis and Dario Amodei had a pretty interesting 30 min discussion about "the World after AGI" at Davos: https://www.youtube.com/watch?v=02YLwsCKUww

At the end of it, someone asked about the Fermi paradox (28 min mark), and Demis gave the standard answer that it could not be related to AI risk, because we would be seeing the resulting AIs.

Unfortunately, this just indicates that he does not see the most salient X-risk scenario, an uncontrolled intelligence explosion leading to a total destruction of local reality without much of a trace and without any survivors (in particular, without any surviving AI systems), either via excessively radical progress in physics, or via something else equally drastic. This scenario... (read more)

Some of the ways the IABIED plan can backfire

mishka

5mo

If one thinks the chance of an existential disaster is close to 100%, one might tend to worry less about the potential of a plan to counter it to backfire. It's not clear if that is a correct approach even if one thinks the chances of an existential disaster are that high, but I am going to set that aside.

If one thinks the chance of an existential disaster is "anywhere between 10% and 90%", one should definitely worry about the potential of any plan to counter it to backfire.

Out of all ways the IABIED plan to ban AI development and to ban publication of AI research could potentially backfire, I want to... (read 457 more words →)

Automated formal solution of 5 out of 6 IMO 2025 problems is achieved by Harmonic AI using their current Aristotle system. No claims of satisfying formal conditions for gold are made (neither in terms of runtime, nor (I think) in terms of unassisted autoformalization of problem statements). Fully automated solution synthesis is achieved after formal statements of problems are given. The system automatically annotates proofs with comments in key places. Formalization is in Lean.

Despite the limitations above, this is a very impressive result. The proofs are published: https://github.com/harmonic-ai/IMO2025. The livestream recording is here: https://x.com/HarmonicMath/status/1949951004482441362

They have just opened their waitlist for their first public form factor for Aristotle, an iOS app: https://aristotle.harmonic.fun/

My request for access has been approved very quickly (I don't know them, but my background and my aims are a reasonable fit).

METR releases a report, Evaluating frontier AI R&D capabilities of language model agents against human experts: https://metr.org/blog/2024-11-22-evaluating-r-d-capabilities-of-llms/

Daniel Kokotajlo and Eli Lifland both feel that one should update towards shorter timelines remaining until the start of rapid acceleration via AIs doing AI research based on this report:

https://x.com/DKokotajlo67142/status/1860079440497377641

https://x.com/eli_lifland/status/1860087262849171797

Gwern was on Dwarkesh yesterday: https://www.dwarkeshpatel.com/p/gwern-branwen

We recorded this conversation in person. In order to protect Gwern’s anonymity, we created this avatar. This isn’t his voice. This isn’t his face. But these are his words.

Scott Alexander wrote a very interesting post covering the details of the political fight around SB 1047 a few days ago: https://www.astralcodexten.com/p/sb-1047-our-side-of-the-story

I've learned a lot of things new to me reading it (which is remarkable given how much material related to SB 1047 I have seen before)

Two subtle aspects of the latest OpenAI announcement, https://openai.com/index/openai-board-forms-safety-and-security-committee/.

A first task of the Safety and Security Committee will be to evaluate and further develop OpenAI’s processes and safeguards over the next 90 days. At the conclusion of the 90 days, the Safety and Security Committee will share their recommendations with the full Board. Following the full Board’s review, OpenAI will publicly share an update on adopted recommendations in a manner that is consistent with safety and security.

So what they are saying is that just sharing adopted recommendations on safety and security might itself be hazardous. And so they'll share an update publicly, but that update would not necessarily disclose the full set... (read more)

mishka's Shortform

mishka

This is a special post for quick takes (aka "shortform"). Only the owner can create top-level comments.

Digital humans vs merge with AI? Same or different?

Nathan Helm-Burger

Nathan Helm-Burger, mishka

mishka

So, to create context, this is a continuation of our remarks in the comments to Zvi's "AI #40: A Vision from Vitalik"

mishka

There I was asking about

the boundary between humans merging with AI and digital humans (can these approaches be reliably differentiated from each other? or is there a large overlap?)
why digital humans would be a safer alternative than the merge

mishka

A good starting point might be to ask you to elaborate on the first item: what are the differences, is there an significant overlap, is one concept almost a subset of another.

How do you see the relationship between the notion of a digital human and a "hybrid" (if that's what the merge is)?

Nathan

... (read 1907 more words →)

What is known about invariants in self-modifying systems?

mishka

I have been trying to find out what is known about invariants in self-modifying systems. This might become a rather acute topic if we end up moving towards self-modifying AIs or self-modifying ecosystems of AIs.

But it seems that not much has been done. For example, I have found a 1995 Chinese paper, "S-and T-Invariants in Cyber Net Systems" by Yuan Chongyi, Google Scholar page, PDF available which is doing a study of invariants in self-modifying nets (a natural extension of Petri nets), but it only has 4 references to it known to Google Scholar.

I wonder if people know about more examples of this kind of research (or about researchers or organizations currently trying to look at this topic)...

Some Intuitions for the Ethicophysics

[anonymous]

MadHatter

Hello! Welcome to the dialogue.

MadHatter

I'll just wait a few minutes for you to see the notification, I guess.

mishka

Hi, yes, I see this. Great!

mishka

So, we have this conversation in the comments in your post [https://www.lesswrong.com/posts/DkkfPEwTnPQyvrgK8/ethicophysics-i](https://www.lesswrong.com/posts/DkkfPEwTnPQyvrgK8/ethicophysics-i) as a starting point.

MadHatter

Yes, I think that's as good a place to jump off as any. Ethicophysics I is basically a reverse-engineering of the content of religion. It's deeply incomplete, and it sounds totally crazy, since religion is totally crazy and I haven't had the time to edit it into something more normal sounding yet.

MadHatter

Also, let me drop my github links to my most recent drafts of everything important.

https://github.com/epurdy/ethicophysics

MadHatter

This has four pdfs in it: Ethicophysics I and II,

... (read 2113 more words →)

Impressions from base-GPT-4?

mishka

mishka, janus

I wonder if some people here had a chance to play with base-GPT-4 (the access is given very selectively for research purposes) and would not mind sharing some of their impressions?

I know that some people have been playing with it, but I've never seen a discussion of impressions and lessons from that. And I know that it is quite nontrivial to get access to this model, but that some access is given.

I think it would be super-interesting for many people here to hear this kind of conversation...

Ilya Sutskever's thoughts on AI safety (July 2023): a transcript with my comments

mishka

There has been a 25 min interview with Ilya conducted by Sven Strohband and released on July 17: https://www.youtube.com/watch?v=xym5f0XYlSc

This interview has a section dedicated to AI safety (7 min starting from 14:56). Ilya is now the co-lead of the OpenAI "superalignment" effort, and his thinking will likely be particularly influential in how this effort evolves.

What he is saying seems to be somewhat different from what is in the consensus OpenAI "superalignment" documents. It's compatible, but the emphasis is rather different. In particular, thinking about humans controlling or steering a superintelligent system is limited to an analogy of controlling a nuclear reactor to prevent a meltdown, and a more collaborative approach between humans... (read 1318 more words →)

What to read on the "informal multi-world model"?

mishka

People on LessWrong often talk in terms of an "informal many-world model". For example they talk about the worlds where the alignment problem is relatively easy vs the worlds where the alignment problem is really hard.

I wonder what are good references for this "informal multiverse model of reality" with multiverses of worlds with different properties? What's the history of this line of thoughts.

(And I also wonder about possible mathematical models of that, how much people are talking about those, and how much people keeping some of those mathematical models in mind? I've seen some sheaf-based models which seemed to be like that.)

RecurrentGPT: a loom-type tool with a twist

mishka

This is a link post for RecurrentGPT: Interactive Generation of (Arbitrarily) Long Text paper by a team from ETH Zürich ("Swiss Federal Institute of Technology in Zürich").

The Compleat Cybornaut recent post discusses looming and the evolving field of loom-type tools. We are going to see more loom-type tools in the near future, and it might be helpful for their authors to be aware of the details of RecurrentGPT tool.

The authors of RecurrentGPT don't seem to be aware of the Simulator theory or Cyborgism, but they seem to be affected by their zeitgeist. The abstract starts as follows (italic is mine):

The fixed-size context of Transformer makes GPT models incapable of generating arbitrarily long

... (read 708 more words →)

Five Worlds of AI (by Scott Aaronson and Boaz Barak)

mishka

A relatively short April 27, 2023 post attempting to classify AI outcomes into 5 main cases:

AI-Fizzle
Not-AI-Fizzle
- Civilization recognizably continues
  - Futurama
  - AI-Dystopia
- Civilization does not recognizably continue
  - Singularia
  - Paperclipalypse

In other words, no one has done for AI what Russell Impagliazzo did for complexity theory in 1995, when he defined the five worlds Algorithmica, Heuristica, Pessiland, Minicrypt, and Cryptomania, corresponding to five possible resolutions of the P vs. NP problem along with the central unsolved problems of cryptography.

The authors elaborate:

Like in Impagliazzo’s 1995 paper on the five potential worlds of the difficulty of NP problems, we will not try to be exhaustive but rather concentrate on extreme cases. It’s possible that we’ll end up in a mixture of worlds

... (read more)