Scary enough for ya?

Sufficiently scary, yes.

That is equivalent to saying you can't understand how mathematics could be a construct; or how mathematical anti-realism could possibly be true.

I assign a respectable probability to anti-realism, and hold no disrespect for anyone who is an anti-realist, but I don't understand how anti-realism can be true. I've never heard a plausible model for why one thing should exist but not another. Tegmarkism sweeps away that problem, leaving the new problem of how to measure probability (why do we have the subjective experience of probability that we do when there are so many versions of myself?). I don't have a satisfactory answer for that question, but it feels like a real question, with meat to get at, whereas in an anti-realist universe the question of why some things exist and other don't seems completely hopeless.

I had a similar experience the first time I supplemented magnesium. Long lasting, non-jittery energy spike. I felt stronger (and empirically could in fact lift more weight), felt better, and was extremely happy. The effect decreased the next few times. After 4 doses (of 50% RDA, spread out over 2 weeks) I began to have adverse effects, including heart palpitation, weakness, and "sense of impending doom".

I wonder if there is a general physiological response to a sudden swing in electrolyte balance that causes the positive effect, rather than the removal of a deficiency.

If you wipe out the chemical gradient information then how do you know what sorts of ways that the dendrites should regrow in the weeks and months post-resuscitation?

If I wake up and I feel like myself on a second to second basis, I will not be upset if my path through mind space is drastically altered on a time scale of weeks and months, so long as it doesn't lead me to insanity. Hell, I hope I'll be able to drastically change my mind on that time scale anyway once I'm uploaded.

if a problem doesn't appear quickly, then it probably isn't that important...

I agree completely, especially about how close we probably are to a successful Biosphere, but just to throw out an example where this is wrong: vitamin B-12 deficiency usually takes a decade to demonstrate symptoms, and is fatal.

It is dangerous in the same way as bringing John Q. Snodgrass to trial for murder. We might overweight evidence in favor of the hypothesis.

Human intuition is a valuable heuristic. As a mathematician I constantly entertain hypotheses I don't believe to be true, for the simple reason that my intuition presented them to be considered. I don't believe I would be at all effective otherwise (although I did just now entertain the hypothesis, despite my lack of belief!)

firstly, a lot of aspects would not necessarily scale up to a smarter system, and it's sometimes hard to tell what generalizes and what doesn't.

I agree, but certainly trying to solve the problem without any hands on knowledge is more difficulty.

Secondly, it's very very hard to pinpoint the "intelligence" of a program without running it

I agree, there is a risk that the first AGI we build will be intelligent enough to skillfully manipulate us. I think the chances are quite small. I find it difficult to image skipping dog level intelligence and human level intelligence and jumping straight to superhuman intelligence, but it is certainly possible.

I agree with Allen and Wallach here. We don't know what an AGI is going to look like. Maybe the idea of a utility maximizer is unfeasible, and the AGIs we are capable of building end up operating in a fundamentally different way (more like a human brain, perhaps). Maybe morality compatible with our own desires can only exist in a fuzzy form at a very high level of abstraction, effectively precluding mathematically precise statements about its behavior (like in a human brain).

These possibilities don't seem trivial to me, and would undermine results from friendliness theory. Why not instead develop a sub-superintelligent AI first (perhaps an intelligence intentionally less than human), so that we can observe directly what the system looks like before we attempt to redesign it for greater safety.

Very interesting. It appears my own model of the brain included a false dichotomy.

If modules are not genetically hardwired, but rather develop as they adapt to specific stimuli, then we should expect infants to have more homogeneous brains. Is that the case?

When I read it I was imagining something tongue in cheeky like Pirates of Penzance. Dr. Seuss would have the advantage of great illustrations though.