The linked paper explicitly assumes that

The evolution operator T is invertible.

But if you use QM in the conventional way, then this assumption doesn't hold. Suppose you have a state X1 which can evolve into either X2 or X3 with equal probability. You would say that state X1 evolves into the weighted set [1/2 X2 + 1/2 X3]. Shalizi proves that this set has no more entropy than X1 did.

But we, as observers or as part of that system, only get to look at one of the branches, either X2 or X3. Picking which of those two branches we get to look at adds one bit of new entropy, and this selection is not invertible. This is where the increase in entropy with time comes from. What Shalizi has done, is to use math in which all entropy originates in quantum branching, then forget that quantum branching happens.

I don't work for SI and this is not an SI-authorized response, unless SI endorses it later. This comment is based on my own understanding based on conversations with and publications of SI members and general world model, and does not necessarily reflect the views or activities of SI.

The first thing I notice is that your interpretation of SI's goals with respect to AGI are narrower than the impression I had gotten, based on conversations with SI members. In particular, I don't think SI's research is limited to trying to make AGI friendliness provable, but on a variety of different safety strategies, and on the relative win-rates of different technological paths, eg brain uploading vs. de-novo AI, classes of utility functions and their relative risks, and so on. There is also a distinction between "FAI theory" and "AGI theory" that you aren't making; the idea, as I see it, is that to the extent to which these are separable, "FAI theory" covers research into safety mechanisms which reduce the probability of disaster if any AGI is created, while "AGI theory" covers research that brings the creation of any AGI closer. Your first objection - that a maximizing FAI would be very dangerous - seems to be based on a belief, first, that SI is researching a narrower class of safety mechanisms than it really is, and second, that SI researches AGI theory, which I believe it explicitly does not.

You seem a bit sore that SI hasn't talked about your notion of Tool-AI, but I'm a bit confused by this, since it's the first time I've heard that term used, and your link is to an email thread which, unless I'm missing something, was not disseminated publicly or through SI in general. A conversation about tool-based AI is well worth having; my current perspective is that it looks like it interacts with the inevitability argument and the overall AI power curve in such a way that it's still very dangerous, and that it amounts to a slightly different spin on Oracle AI, but this would be a complicated discussion. But bringing it up effectively for the first time, in the middle of a multi-pronged attack on SI's credibility, seems really unfair. While there may have been a significant communications failure in there, a cursory reading suggests to me that your question never made it to the right person.

The claim that SI will perform better if they don't get funding seems very strange. My model is that it would force their current employees to leave and spend their time on unrelated paid work instead, which doesn't seem like an improvement. I get the impression that your views of SI's achievements may be getting measured against a metric of achievements-per-organization, rather than achievements-per-dollar; in absolute budget terms, SI is tiny. But they've still had a huge memetic influence, difficult as that is to measure.

All that said, I applaud your decision to post your objections and read the responses. This sort of dialogue is a good way to reach true beliefs, and I look forward to reading more of it from all sides.

There is a world out there. Bettering it is important, but the worn roads lead to nowhere. Most people follow them, and never consider doing anything else. If you ask them when they chose to do nothing, and they answer truthfully, most will say they never chose - they merely did what was expected of them.

You, however, have open eyes. You believe that studying and teaching math will make you happy, but studying other things will, with some small probability, position you to aid the world. All the philosophy of utility functions can do is wrap this choice up in more boilerplate; it can't help you make it. What can help is information about how happy you'll be, given each choice, and about how much you can actually help the world, given each choice.

I suspect that whether you study and teach math will affect your happiness much less than you think. It's also nearly certain that there are other good options you haven't considered. It depends a lot on details - what other aptitudes you have, and whether the school you'll be going to will help you study things other than math, in particular. And you haven't really explained what not-studying-math would mean; it would be easier to compare if that were replaced with something more concrete, like taking a gap year to study independently.

Does the utility function at the time of the choice have some sort of preferred status in the calculation

Yes, it does. Your present utility function may make reference to the utility functions of your future selves - eg, you want your future selves to be happy - but structurally speaking, present-day preferences about your future selves are the only way in which those other utility functions can bear on your decisions.

"Today, children," began the calm professional voice of the Transfiguration Professor, just as though nothing out of the ordinary had happened that week, "we shall learn how much effort it takes to sustain a Transfiguration, and why, at your age, you should not even try."

I'm not sure how my mind dug this up, but way back in Chapter 17, Harry visits Dumbledore's office and is overloaded with bizarreness: Dumbledore sets fire to a chicken, he gives him his father's rock, he gives him his mother's potions textbook which contains a terrible secret... but one of these things is not like the others. Dumbledore gave Harry his father's rock, with instructions that Harry satisfied by creating a magical ring and wearing it at all times.

Blur out all the hilarious details for a minute, and that scene is: Dumbledore made Harry create a magical ring and wear it at all times, and distracted him so well that he never thought about what the ring does. My hypothesis is that some aspect of magic is governed by an XP-like mechanic, and that sustained transfiguration (especially of large masses) is an unusually effective way of gaining magical power. Dumbledore wants Harry to exploit this, but he considers it a major secret, so he substituted a nonsensical explanation and prepared a collection of very flashy distractions to keep it from being questioned. He might've even left the real explanation in his pensieve, so that he wouldn't have to lie. Read in this light, the scene makes a whole lot more sense. It explains Harry's anomalous magical power. It explains Dumbledore's anomalous magical power.

It is also the only way Dumbledore could truly mark someone as an equal.

The correct response to that sort of stupidity is immediate tab-closing, unless it comes from an especially important person who can be swayed. Unfortunately, this still really gets me in person, where walking away is harder.

From my own experience as a programmer, I think this is idealized to the point of being false. Finding a few distantly-separated, interacting regions of code which don't respect a clean abstraction is pretty common, especially when debugging (in which case there is an abstraction but it doesn't work).

model the flow of instructions in your mind, how distant parts of the code interact together

Unless you're hacking you usually don't need to do this. You just need to understand what state the program is in before and after each operation. You never need to understand the whole thing at once, just understand one part at a time.

Er, what? You absolutely do need to model control flow, and how distant parts fit together. You should only think about state one operation at a time when you're confused, or suspicious of the code you're looking at, because step-by-step thinking is very slow and can't support most of the operations you'd want to do on a program.

I'm more interested in how to explain our state of the art understanding of giving a reductionist account of actual physics, which seems to terminate at questions about the ontological status of quantum amplitude.

The next step below quantum amplitude, if there is one, is Tegmarkian multiverses, which are a reduction fixpoint (they reduce to themselves). (There might be one intermediate in between - I have a strong suspicion that quantum amplitudes are a continuous approximation of something discrete). However, there is pretty good reason to believe that we cannot gather evidence about them, even in principle.