the most important thing right now is that you have misunderstood my position on the question.

Thanks for the clarification!

But in my paper I was talking only about the apparent contradiction between (a) forcing people to do something as they screamed their protests, while claiming that those people had asked for this to be done, and (b) an assessment that this behavior was both intelligent and sane.

I'm glad to hear that we're focusing on this narrow issue, so let me try to present my thoughts on it more clearly. Unfortunately, this involves bringing up many individual examples of issues, none of which I particularly care about; I'm trying to point at the central issue that we may need to instruct an AI how to solve these sorts of problems in general, or we may run into issues where an AI extrapolates its models incorrectly.

When people talk about interpersonal ethics, they typically think in terms of relationships. Two people who meet in the street have certain rules for interaction, and teachers and students have other rules for interaction, and doctors and patients other rules, and so on. When considering superintelligences interacting with intelligences, the sort of rules we will need seem categorically different, and the closest analogs we have now are system designers and engineers interacting with users.

When we consider people interacting with people, we can rely on 'informed consent' as our gold standard, because it's flexible and prevents most bad things while allowing most good things. But consent has its limits; society extends children only limited powers of consent, reserving many (but not all) of them for their parents; some people are determined mentally incapable, and so on. We have complicated relationships where one person acts in trust for another person (I might be unable to understand a legal document, but still sign it on the advice of my lawyer, who presumably can understand that document, or be unable to understand the implications of undergoing a particular medical treatment, but still do it on the advice of my doctor), because the point of those relationships is that one person can trade their specialized knowledge to another person, but the second person is benefited by a guarantee the first person is actually acting in their interest.

We can imagine a doctor wireheading their patient when the patient did not in fact want to be wireheaded, for a wide variety of reasons. I'm neither a doctor nor a lawyer, so I can't tell you what sort of consent a doctor needs to inject someone with morphine--but it seems to me that sometimes things will be uncertain, and the doctor will drug someone when a doctor with perfect knowledge would not have, but we nevertheless endorse that decision as the best one the doctor could have made at the time.

But informed consent starts being less useful when we get to systems. Consider a system that takes biomedical data from every person in America seeking an organ transplant, and the biomedical data from donated organs as they become available, and matches organs to patients. Everyone involved likely consented to be involved (or, at least, didn't not consent if there's an opt-out organ donation system), but there are still huge ethical questions remaining to be solved. What tradeoffs are we willing to accept between maximizing QALYs and ensuring equity? How much biomedical data can we use to predict the QALYs from any particular transplant, and what constitutes unethical discrimination?

It seems unlikely to me that the main challenge of superintelligence is that a superintelligence will force or trick us into doing things that it is obvious that we don't want to do them. It seems likely to me that the main challenge is that it will set up systems, potentially with some form of mandatory participation, and we thus need to create a generalized system architect that can solve those ethical, moral, and engineering problems for us while designing arbitrary systems, without us having coded in the exact solutions.

Notice also that consent applies to individual rights, not community rights, but many people's happiness and livelihoods may rest on community rights. There are already debates over whether or not deafness should be cured: how sad to always be the youngest deaf person, or for deaf culture to disappear, but to avoid that harm, we need some people to be deaf instead of hearing, which is its own harm. Managing this in a way that truly maximizes human flourishing seems like it requires a long description.

Many human ethical problems are solved by rounding small numbers to zero, but superintelligences represent the ability to actually track those small numbers, which means entire legal categories that rest on a sharp divide between 0 and 1 could become smooth. For example, consider 'sexual harassment' defined as 'unwanted advances.' Should a SI censor any advances it thinks that the receiver will not want, or is that taking sovereignty from the receiver to determine whether or not they want any advance?

My understanding is, then, that this point I just made is now generally accepted.

Right, and I agree with it as well. I think the remaining useful insight of functional programming is that minimizing side effects increases code legibility, and if we want to be confident in the reasoning of an AI system (or an AI system wants to be able to confidently predict the impact of a proposed self-modification) we likely want the code to be partitioned as neatly as possible, so any downstream changes or upstream dependencies can be determined simply.

Neural nets, and related systems, do not have a preference for legibility built into the underlying structure, and so we may not want to use them or related systems for goal management code, or take especial care when connecting them to goal management code.

where the overall behavior cannot be hijacked by any one of the atoms in that ensemble.

Hmm. I'm going to have to think about this claim longer, but right now I disagree. I think the model of human reasoning that seems most natural to me is hierarchical control systems. When I think of "swarms," that implies to me some sort of homogeneity between the agents, such as might describe groups of humans but not necessarily individual humans. (If we consider humans as swarms of neurons, which is how I originally read your statement, then the 'swarm-like' properties map on the control loops of the hierarchical view.)

But it seems to me that if the atoms in the swarm have specialized roles (like neurons), then a small number of atoms behaving strangely can lead to the swarm behaving strangely. (This is easier to see in the controls case, but I think is also sensible in the swarm of neurons model.) I'm thinking of the various extreme cases of psychology as examples: stuff like destroying parts of cats' brains and seeing how they behave, or the various exotic psychological disorders whose causes can be localized in the brain, or so on. That a system is built out of many subcomponents, instead of being a single logical reasoner, does not seem to me to be a significant source of safety.

(Now, I do think that various 'moral congress' ideas might represent some sort of safety--if you need many value systems to all agree that something is a good idea, then it seems less likely that extreme alternatives will be chosen in exotic scenarios, and a single value system can be a composite of many simpler value systems. This is ideas like 'bagging' from machine learning applied to goals--but the gains seem minor to me.)