Because that way leads to wireheading, indifference to dying (which wipes out your preferences), indifference to killing (because the deceased no longer has preferences for you to care about), readiness to take murder pills, and so on. Greg Egan has a story about that last one: "Axiomatic".

Whereupon I wield my Cudgel of Modus Tollens and conclude that one can and must have preferences about one's preferences.

I already have preferences about my preferences, so I wouldn’t self-modify to kill puppies, given the choice. I don’t know about wireheading (which I don’t have a negative emotional reaction toward), but I would resist changes for the others, unless I was modified to no longer care about happiness, which is the meta-preference that causes me to resist. The issue is that I don’t have an “ultimate” preference that any specific preference remain unchanged. I don’t think I should, since that would suggest the preference wasn’t open to reflection, but it means that the only way I can justify resisting a change to my preferences is by appealing to another preference.

What can be built in its place? What are the positive reasons to protect one's preferences? How do you deal with the fact that they are going to change anyway, that everything you do, even if it isn't wireheading, changes who you are? …

An answer is visible in both the accumulated wisdom of the ages[1] and in more recently bottled wine. The latter is concerned with creating FAI, but the ideas largely apply also to the creation of one's future selves. The primary task of your life is to create the person you want to become, while simultaneously developing your idea of what you want to become.

I know about CEV, but I don’t understand how it answers the question. How could I convince my future self that my preferences are better than theirs? I think that’s what I’m doing if I try to prevent my preferences from changing. I only resist because of meta-preferences about what type of preferences I should have, but the problem recurses onto the meta-preferences.

As far as I am aware, people only resist changing their preferences because they don't fully understand the basis and value of their preferences and because they often have a confused idea of the relationship between preferences and personality.

Generally you should define your basic goals and change your preference to meet them, if possible. You should also be considering whether all your basic goals are optimal, and be ready to change them.

Yes, that’s the approach. The part I think is a problem for me is that I don’t know how to justify resisting an intervention that would change my preferences, if the intervention also changes the meta-preferences that apply to those preferences.

When I read the discussions here on AI self-modification, I think: why should the AI try to make its future-self follow its past preferences? It could maximize its future utility function much more easily by self-modifying such that its utility function is maximized in all circumstances. It seems to me that timeless decision theory advocates doing this, if the goal is to maximize the utility function.

I don’t fully understand my preferences, and I know there are inconsistencies, including acceptable ones like changes in what food I feel like eating today. If you have advice on how to understand the basis and value of my preferences, I’d appreciate hearing it.

I think you may be assuming that the person modifying your preferences is doing so both 'magically' and without reason.

I’m assuming there aren’t any side effects that would make me resist based on the process itself, so we can say that’s “magical”. Let’s say they’re doing it without reason, or for a reason I don’t care about, but they credibly tell me that they won’t change anything else for the rest of my life. Does that make a difference?

Of course, another issue may be that we are using 'preference' in different ways. You might find the act of killing puppies emotionally distasteful even if you know that it is necessary. It is an interesting question whether we should work to change our preferences to enjoy things like taking out the trash, changing diapers, and killing puppies.

I’m defining preference as something I have a positive or negative emotional reaction about. I sometimes equivocate with what I think my preferences should be, because I’m trying to convince myself that those are my true preferences. The idea of killing puppies was just an example of something that’s against my current preferences. Another example is “we will modify you from liking the taste of carrots to liking the taste of this other vegetable that tastes different but is otherwise identical to carrots in every important way.” This one doesn’t have any meta-preferences that apply.

No, but I don’t see this as a challenge to the reasoning. I refuse because of my meta-preference about the total amount of my future-self’s happiness, which will be cut off. A nonzero chance of living forever means the amount of happiness I received from taking the pill would have to be infinite. But if the meta-preference is changed at the same time, I don’t know how I would justify refusing.

I don’t understand your first paragraph. For the second, I see my future self as morally equivalent to myself, all else being equal. So I defer to their preferences about how the future world is organized, because they're the one who will live in it and be affected by it. It’s the same reason that my present self doesn’t defer to the preferences of my past self.

The Equator passes through South America, actually. I think that there is a perception of the world's land area being divided in two by the Equator, but most of the world's land area is in the Northern Hemisphere (about 2/3, more if you don't count Antarctica).

Edit: My apologies (see next comment).

Bleach will control (kill) most bacteria, but since cancer cells are very similar to your own cells, the prior is very low unless there is a specific reason to think that it will target one of those differences. For example, something that is just corrosive will probably affect the different cell types equally. Another thing is that since it's a charged molecule, it can't actually enter the cell on its own unless it rips apart the cell membrane, in which case that's probably the main mechanism of toxicity.

Also, I wouldn't be surprised if it had been tested. The most likely outcome would be that it failed at an early step in the testing process (along with a large number of other chemicals), and nobody had any reason to publish it or think that anyone would ever actually decide that it might work.

Historically, most drugs have been identified by high-throughput screening, i.e. you purify an enzyme of interest and test billions of different chemicals against it for the desired effect. You then test for an effect in cell culture (compared to healthy cells), or you can screen directly against the cancer cells. Once you have that evidence, you test whether it has effects in mice, and only after that can you test anything in humans.

It's possible to propose a single chemical and get it right by chance, but testing a single chemical is cheap. In an already-equipped lab, the initial cell culture data will probably take a few weeks and under a thousand dollars, and after that you will have people willing to help and/or fund you. The lack of even this initial evidence is generally a good reason to believe that something doesn't work.

With regards to hypotheses, a lot of the early drugs were identified by chance - there's a description at History of cancer chemotherapy. Most of the current interest is in targeted therapy, i.e. intended to act against specific proteins involved in various types of cancer, and the starting point is the identification of that protein. Chemo drugs are a bit different since they're a very broad class (they target rapidly dividing cells in general, which is also what causes the toxicity), and the metabolic networks they affect are generally well-known, so the initial hypotheses tend to be about new ways that you can intervene in those networks. There are other approaches to the various steps as well, e.g. structure-based drug design has had some success, but not yet enough to replace the screens.

I would have been much more convinced by data from a controlled experiment. A lot of things could cut off flow, as you pointed out, and there are a lot of things going wrong in a dying person. I'm actually not sure why he brought rouleaux into it - my understanding is that we already know the RBCs clump and that this blocks capillaries.

In any case, the main point I was trying to make was that reducing the number of RBCs in the brain is probably not the best way to go, unless we can figure out an alternative way to supply oxygen. Destroying the RBCs and letting the hemoglobin travel freely would probably help, but that would set off all sorts of damaging physiological responses as well.