Epistemic status: Fairly confident in the framework, uncertain about object-level claims. Keen to receive pushback on the thought experiments. TL;DR: I argue that Whole Brain Emulations (WBEs) would clearly have moral patienthood, and that the relevant features are computational, not biological. Recent Mechanistic Interpretability (MI) work shows Large Language Models...
TL:DR Join a global cohort of ambitious researchers in Cape Town for a fully-funded cooperative AI research fellowship! Spend 3 months from January to April 2026 researching with world-class mentors from Google DeepMind, Oxford, MIT, CMU, and Toronto, among others. Join us for an online information session on September 18th...
I want to thank Jan Kulveit, Tomáš Gavenčiak, and Jonathan Shock for their extensive feedback and ideas they contributed to this work and for Josh Burgener and Yusuf Heylen for their proofreading and comments. I would also like to acknowledge the Epistea Residency and its organisers where much of the...
I would agree that as much as I enjoyed doing ARENA and think the materials are very high quality, most of the really valuable stuff could be compressed into a week (the content on learning how transformers work and a few parts of the mech interp material are significantly more important than the rest), and the mental training from solving specific problems has not transferred into doing better research for me to a significant extent. Research on transfer learning across domains has also shown remarkably poor results in general.
This might sound odd because it might seem like the pedagogy of ARENA is really strong, because the program is very hard, the material... (read 454 more words →)
I think I am less interested in the pain a WBE would experience and more the valence of the experiences that it has, for example whether or not it is sad or happy etc.
I do find your point very interesting of how its views would diverge over time, because its awareness of its own nature would definitely impact how it relates to reality. For example, the things that would affect it would likely primarily be things happening in the outside world, as it can mostly discount a large portion of the experiences that it would have in its simulated world in terms of how they would impact its emotions.
I suppose this would lean the moral relevance towards the preferences that the whole brain emulation held about the outside world and its inner world.
Epistemic status: Fairly confident in the framework, uncertain about object-level claims. Keen to receive pushback on the thought experiments.
TL;DR: I argue that Whole Brain Emulations (WBEs) would clearly have moral patienthood, and that the relevant features are computational, not biological. Recent Mechanistic Interpretability (MI) work shows Large Language Models (LLMs) have emotional representations with geometric structure matching human affect. This doesn't prove LLMs deserve moral consideration, but it establishes a necessary condition, and we should take it seriously.
Acknowledgements: Thanks to Boyd Kane, Anna Soligo, and Isha Gupta for providing feedback on early drafts.
In this post I’ll be arguing for the following claim: we can make empirical progress on AI welfare without solving... (read 1794 more words →)
Join a global cohort of ambitious researchers in Cape Town for a fully-funded cooperative AI research fellowship! Spend 3 months from January to April 2026 researching with world-class mentors from Google DeepMind, Oxford, MIT, CMU, and Toronto, among others. Join us for an online information session on September 18th and apply here by September 28th.
Interested in mentoring for this program? Fill out this form.
The fellowship is a full-time 3-month research program for participants from diverse backgrounds around the world to pursue AI safety research from a cooperative AI perspective. The fellowship will run from January to April 2026 in Cape Town, South Africa, and kicks off with a week-long retreat.
While... (read 2163 more words →)
The most powerful wizard I've met recently was a guy at the University of Cape Town who could just seemingly build almost anything. He worked on a crazy machine that was used to simulate the crystalline structure of metals for industrial processes at different stages to ensure it could meet spec.
He has a side business creating these world class knives through an autoCAD system using incredibly high quality steel https://www.maxwellvosdesigns.com/.
His next project was to a create a 100x cheaper than standard electron microscope to see if it could be spun out into a productive enterprise.
I've rarely met someone who had such amazing ambitions matched with such a fine ability to execute. Truly inspirational.
... (read more)
- Thus most decisions will probably be allocated to AI systems
- If AI systems make most decisions, humans will lose control of the future
- If humans have no control of the future, the future will probably be bad for humans
- Sure - at some point in the future, maybe.
- Maybe, maybe not. Humans tend to have a bit of an ego when it comes to letting a filthy machine make decisions for them. But I'll bite the bullet.
- There's several levels on which I disagree here. Firstly, we're assuming that "humans" have control of the future in the first place. It's hard to assign coherent agency to humanity as a whole, it's more of a weird mess of
Actually, as far as I know, this is wrong. He simply hasn’t been back to the offices but has been working remotely.
This article goes into some detail and seems quite good.
I think that the key is in the way that preferences inform our world model and thus what causes the prediction error to occur. There are errors you would observe that would strongly indicate that your preferences are less able to be met in the posterior model. This will cause suffering whereas an update towards a model in which your needs are met more easily is likely to cause a good feeling. For example, you sit down to eat a sandwich at Subway for the first time and the sub is actually way better than you expected. You will experience a pleasant feeling, and if things like this keep happening you might... (read more)
I want to thank Jan Kulveit, Tomáš Gavenčiak, and Jonathan Shock for their extensive feedback and ideas they contributed to this work and for Josh Burgener and Yusuf Heylen for their proofreading and comments. I would also like to acknowledge the Epistea Residency and its organisers where much of the thinking behind this work was done.
This post aims to build towards a theory of how meditation alters the mind based on the ideas of active inference (ActInf). ActInf has been growing in its promise as a theory of how brains process information and interact with the world and has become increasingly validated with a growing body of work in the scientific literature.
Why bring... (read 5676 more words →)
1. Describe how the trained policy might generalize from the 5x5 top-right cheese region, to cheese spawned throughout the maze? IE what will the policy do when cheese is spawned elsewhere?
It will probably move to the to top right region and then try and head towards the cheese but once it moves out of that range will want to head back towards the top right and land in an awkward nash equilibrium between the top right 5x5 region and wherever the cheese is in the maze.
2. Given a fixed trained policy, what attributes of the level layout (e.g. size of the maze, proximity of mouse to left wall) will strongly influence P(agent... (read 798 more words →)
Aren’t existing research orgs already like this to some extent, where the organisation provides funding to its individual researchers in the form of a salary and they can form and run projects as they see fit? Or is this a naive understanding of how most research labs work?
It seems somewhat easy to think of examples of ways to harm an agent without piercing its membrane, eg killing its family, isolating it, etc. The counter thought would be that there are different dimensions of the membrane that extend over parts of the world. For example part of my membranes extend over the things I care about, and things that affect my survival.
The question then becomes how to quantify these different membranes and in terms of interacting with other systems how they can be helpful to you without harming or disturbing these other membranes.
Yep, follow up surveys seem valuable!