I think it is going to pretty wonderful if this study is not reproducible, to serve as cautionary example.

Not only there has to be UFAI risk, the FAI development must reduce the risk, which to me looks like the most shaky of the propositions. A buggy FAI that doesn't break itself somehow is for certain unfriendly (e.g. it can want to euthanize you to end your suffering, or to cut apart your brain into 2 hemispheres to satisfy each hemisphere's different desires, or something much more bizarre), while some random AI out of AI design space may e.g. typically wirehead everything except curiosity, and then it'd just keep us in a sort of wildlife preserve.

Note: try to avoid just world fallacy. Just because you work harder to make friendlier AI doesn't necessarily make result friendlier. The universe doesn't grade for effort.

We humans do a lot of wireheaded stuff. Fiction, art, MSG in food, porn, cosmetic implants... we aren't doing it literally with a wire into the head, but we find creative ways to satisfy goals... maybe even the desire to build AI itself is a result of breaking the goal system. And the wireheadedness makes us tolerate other forms of life instead of setting on to exterminate everything as we already would have if we still pursued some reproductive goals.

Hell, we even wirehead our curiosity and quest for knowledge (see religions), entirely internally by breaking the goal system with self suggestion.

re: Benoit Mandelbrot and fractals, the fractals are way, way older. The actual study of them had to wait until computers.

ahh, by the way, the points i have the most confidence about: 4, 9 . It seems virtually certain for me that precautions will not be adequate. The situation is similar to getting a server of some kind unhackable on the first compile and run.

Same goes also for creation of friendly AI. The situation is worse than writing a first autopilot ever, and on the first run of that autopilot software, flying in it complete with automated takeoff and landing. The plane's just going to crash, period. We are this sloppy at software development and there is nothing we can do about it. The worst that can happen is the AI that is not friendly but does treat humans as special; it can euthanise humans even if we are otherwise useful for it, for example. A buggy friendly AI is probably the worst outcome. Seriously, the people who don't develop software got all sorts of entirely wrong intuitions with regards to ability to make something work right on the first try (even with automated theorem proving). Furthermore, a very careful try is a very slow one as well, and is unlikely to be the first. What I am hoping for is that the AIs will just quietly wirehead themselves.

Then there is "radiation hormesis", a hypothesis, that the radiation is good for you. Not "because of such and such specific response, radiation is good for you" - just a hypothesis that it is

Is this a fair description of the history and science behind hormesis?

I've certainly seen several politicians argue that "radiation is actually good for you," but I've yet to hear any actual radiation health physicists argue that point...

Well, the critique I have:

1: We don't know that AI can go FOOM. It may be just as hard to prevent self improving AI from wireheading (when it becomes super-intelligent) as it is to ensure friendliness. Note: perfect wireheading has infinite utility according to agent prone to wireheading; the length of wireheading experience in time (or it's volume in space) is then irrelevant. The whole premise of fear of UFAI is that intelligence (human intelligence) can have faulty self improvement; it's inconsistent to assume that about human intelligence but not about any AI.

2: We don't know that the AI would likely to be substantially unfriendly. Other humans, and especially groups of humans (corporations, governments) are non-you non-friendly-to-you intelligences too, with historical examples of extreme unfriendliness (i'm going to coin a law that the (un)friendly intelligence discussion is incomplete without mention of nazis), yet they can be friendly enough - permitting you to live normal life while paying taxes (but note the military draft, which happens when meta-organism is threatened). It is plausible enough that the AI would be friendly enough. Humans would be cheap to store.

3: We may get there by mind uploading, which seems to me like the safest option.

4: We don't actually know if FAI attempt is more, or less dangerous than messy AI like 'replicate function of cortical columns, simulate a lot of cortical columns'. FAI attempt could just as well be more dangerous. You get it wrong, AI euthanizes you with the best intentions. The extrapolated volition idea btw entirely ignores fact that you are a massively parallel system that can have different goals in different parts of itself (and the mankind too is that kind of system, albeit less well connected).

The argumentation everywhere has very low external probabilities (when I evaluate probabilities if I see conflicting arguments that are opposite and both look similarly plausible, I assume external probability of zero, even if its 1 argument vs 10; much more so for 10 arguments vs 10), and so acting upon those arguments has rather low utility values.

I think the assumption is that your decision theory is fixed, and the lesion has an influence on your utility function via how much you want to smoke (though in a noisy way, so you can't use it to conclude with certainty whether you have the lesion or not).

The decision theories need somewhat specific models of the world to operate correctly. In The Smoking Lesion, for example, the lesion has to somehow lead to you smoking. E.g. the lesion could make you follow CDT while absence of the lesion makes you follow EDT. It's definitely worse to have CDT if it comes at the expense of having the lesion.

The issue here is selection. If you find you opt to smoke, your prior for having lesion goes up, of course; and so you need to be more concerned about the cancer - if you can't check for the lesion you have to perhaps do chest x-rays more often, which cost money. So there's that negative consequence of deciding to smoke, except the decision theory you use needs not be concerned with this particular consequence when deciding to smoke, because the decision is itself a consequence of the lesion in the cases whereby the lesion is predictive of smoking, and only isn't a consequence of the lesion in the cases where lesion is not predictive.

Eliezer once pointed out that our intuitions on most formulations of the Prisoner's Dilemma are skewed by our notions of fairness, and a more outlandish example might serve better to illustrate how a genuine PD really feels.

It might also help to consider examples in which "cooperation" doesn't give warm fuzzy feelings and "defection" the opposite. Businessmen forming a cartel are also in a PD situation. Do we want businessmen to gang up against their customers?

This may be culturally specific, though. It's interesting that in the standard PD, we're supposed to be rooting for the prisoners to go free. Is that how it's viewed in other countries?

Is that actually uncommon?