The words must have a correlation somewhere in my mind or I wouldn't have thought them in that order,

I have a strong suspicion that is not so - that the brain just chatters to itself, it's pareidolia operating on static hiss on the neurons.

"Refuse to adjust your utility function because you will no longer be you, unless the adjustment improves you in terms of your own values" seems to be an important general principle, and it should be enough for Gandhi to turn down the pill.

The general principle is: cached is fast, cache-populating is slow. This goes for mind and "body" both, because the body does as its told, but it needs telling in a lot of detail and the control signals need to be discovered. Most people, for both mind and body, learn enough control signals for day-to-day use, and stop.

I do somewhat wonder what it would be like to know the control signals for all my muscles, Bene Gesserit style.

Er, yes? Feelings are evolution's way of playing carrot-and-stick with the brain. You really do not want to have an AI that needs spanking, whether it's you or a emotion module that does it: it's apt to delete the spanker and award itself infinite cake.

This is mistaken because systems can and do assemble out of sufficiently similar people pursuing self interest in a way that ends up coordinated because their motivations are alike. Capitalism is the simplest and most obvious example of such a system, but I'd argue things like patriarchy and racism are similar.

It means you could, in theory, run an AI on them (slowly).

You lose whatever information is no longer in the atoms, which might be a lot because the skull is not designed to assist cooling, and the brain is a considerable thermal mass. It's going to cool slowly, be shredded to mush by crystal formation, and be warped and cracked by thermal stress, while undergoing runaway chemical reactions and cell death. Your "limit of perfect technology" is then faced with an awe inspiring task of running the reaction products backwards, modelling and reversing the thermal damage, un-killing the cells, and splicing the cracks, in 3D on tissue that does not come with alignment hints, and then inferring a mind. There's going to be some level of physically unavoidable data loss even in the perfect case, the data is entailed in thermal noise and random photons and the damage is no longer reversible without reversing the universe. Presumably the perfect technology will paper over these cracks by copying in mind structures from Mr Perfectly Average. But the end result would be that you're less you.

Inability to cope with technology maximizing societies is kind of a special case. It applies to basically ALL animals, birds, fish, plants, and even to other humans who decided on being expedient technologists. If you can't call the Parlevar successful ("Before British colonisation in 1803, there were an estimated 3,000–15,000 Parlevar" -- Wikipedia) then you can't call any of the species successful that we wiped out or massively reduced.

I don't follow, care to explain?

FWIW the expedient technology route is the one taken by all other species that have any technology at all. A chimp drops his ant poking stick when he's done poking the ants. It's clearly capable of being an evolutionary success.

Sorry for the slow reply.

Hmm. I may be a bit biased because I don't really have a high valuation on being alive as such (which is to say utility[X] is nearly the same as utility[X and Julian is alive] for me, all other things being equal - it's why I am not signed up for cryonics).

However I think that any utility calculus that negatively values the fun you're not going to have when inevitably dead is as silly as negatively valuing the fun you didn't get to have because said events preceded your birth, and you inevitably can't extend your life into the past. You get more chance to fulfil your values in the real world by making use of your 2 minutes than by anticipating values that are not going to happen. And I do very much place utility on my values being fulfilled in a real, rather than self deceptive way.