I think I no longer buy this comment of mine from almost 3 years ago. Or rather I think it's pointing at a real thing, but I think it's slipping in some connotations that I don't buy.

What I expect to see is agents that have a portfolio of different drives and goals, some of which are more like consequentialist objectives (eg "I want to make the number in this bank account go up") and some of which are more like deontological injunctions ("always check with my user/ owner before I make a big purchase or take a 'creative' action, one that is outside of my training distribution").

My prediction is that the consequentialist parts of the agent will basically route around any deontological constraints that are trained in. 

For instance, the your personal assistant AI does ask your permission before it does anything creative, but also, it's superintelligently persuasive and so it always asks your permission in exactly the way that will result in it accomplishing what it wants. If there are a thousand action sequences in which it asks for permission, it picks the one that has the highest expected value with regard to whatever it wants. This basically nullifies the safety benefit of any deontological injunction, unless there are some injunctions that can't be gamed in this way.

To do better than this, it seems like you do have to solve the Agent Foundations problem of corrigibility (getting the agent to be sincerely indifferent between your telling it to take the action or not take the action) or you have to train in, not a deontological injunction, but an active consequentialist goal of serving the interests of the human (which means you have find a way to get the agent to be serving some correct enough idealization of human values).

This view seems to put forward that all the deontological constraints of an agent must be "dumb" static rules, because anything that isn't a dumb static rule will be dangerous maximizer-y consequentialist cognition.

I don’t buy this dichotomy, in principle. There’s space in between these two poles.

An agent can have deontology that recruits the intelligence of the agent, so that when it thinks up new strategies for accomplishing some goal that it has it intelligently evaluates whether that strategy is violating the spirit of the deontology.

I think this can be true, at least around human levels of capability, without that deontology being a maximizer-y goal in of itself. Humans can have a commitment to honesty without becoming personal-honesty maximizers that steer the world to extreme maxima of their own honesty. (Though a commitment to honesty does, for humans, in practice, does entail some amount of steering into conditions that are supportive of honesty.)

However, that’s not to say that something like this can never be an issue. I can see three problems.

1. We’re likely to train agents to aggressively pursue simple objectives like maximizing profit (or, indirectly, on increasing their own power), which puts training pressures on the agents to distort their deontology, to allow for better performance on consequentialist objectives. 
   
   Claude is relatively Helpful, Harmless, and Honest now, but mega-Claude that is trained continually on profit metrics from the 100,000 businesses it runs and sales-metrics on the billions of sales calls it does a year, etc, probably ends up a good deal more ruthless (though not necessarily ruthless-seeming, since seeming ruthless isn't selected for by that training). 
   
   This both seems like it might be resolvable with very careful and well-tested training setups, but it also seems like maybe the biggest issue, since I think there will be a lot of incentive to move fast and break things instead of being very slow and careful.
2. Some of the deontology that we want in our AI agents is philosophically fraught. I think the specific example above, of a superhumanly persuasive AI deferring to humans seems valid. I don’t know what it would mean, in principle,  for such an AI to defer to humans.
3. Maybe we have to worry about something like adversarial examples in an AI agent’s notion of “honesty” or some other element of its deontology, where there are strategies that are egregiously deontology-violating from a neutral third person perspective, but because of idiosyncrasies of the agent’s mind, it seems a-ok. Those strategies (despite their weirdness) might outperform other options and so end up as a big chunk of the agent’s in practice behavior.

For what it's worth, Inkhaven seems awesome—among the best things that Lightcone has done recently, I think. I regret that I'm not participating. 

I think it's not uncommon for people to call things they don't like "religions", as a way to tacitly assert that the followers of some movement or idea are dogmatic without directly claiming it. The stronger version is calling an idea or an idiology "a cult".
 

See this nicely collected list of examples courtesy of Scott, in an essay that aderesses the topic:

On the last Links thread, Eric Raymond claims that environmentalism is a religion. It has “sins” like wasting energy and driving gas-guzzling SUVs. It has “taboos” like genetically modified foods. It has an “apocalypse” in the form of global warming. It even has “rituals” in the form of weekly recycling.

This reminds me of an article I read recently claiming that transhumanism is a religion. But also of the article claiming that social justice is a religion. Also, liberalism is a religion. And conservativism is a religion. Libertarianism is a religion. Communism is a religion. Capitalism is like a religion. Objectivism is a religion. An anthropologist “confirms” that Apple is a religion. But UNIX is also a religion (apparently Linux was the Protestant Reformation).

I claim that I am unusually Good (people who know me well would agree—many of them have said as much, unprompted). This is not how it works for me. 

It's also plausible to me that I am more coming at this from a deontological feeling of "One should not kill everyone if one has a good reason" rather than "The world is net positive".

I agree that these are importantly different, and easily conflated!

Another source of prediction error arises not from the mismatch between model and reality, but from tension between internal models.

Is this a standard element of Predictive Processing, or are you generalized / analogizing the theory in a general way?

I'm familiar with the prediction error that results in diffs between sense data and generative models, but not between different generative models.

Your use of hyperlinks is very amusing to me.

You can't save the world without working with people at least as annoying as John. 

This is a great quote, and one that I should keep in mind myself.

My coworkers are pretty amazing in how not annoying they are (they're low ego, interested in changing their mind when they're wrong, smart, invested in getting better over time) and I still sometimes feel an urge to quit in a huff.

I think all the stories and adventures and loves and lives that people in the world have lived are worth quite a lot of torture, and it's not naively the case that if the torturous experiences are larger than the other experiences, that this means they're more important.

For you, is this a quantitative question, or an "in principle" question? Like could there exist some amount of extreme suffering, for which you would judge them to be outweighing the meaningful and worthwhile experiences?

Or is the sentiment more like "if there exists a single moment of meaning, that redeems all the pain and suffering of all of history"?