I could be way off on this, but I cannot help but core here is less about complexity than it is about efficiency. The most efficient processes do all appear to be a bit simpler than they probably are. It's a bit like watching an every talented craftsman working and thinking "The looks easy." Then when you try you find out it was much more difficult and complicated than it appeared. The craftsman's efficiency in action (ability to handle/deal with the underlying complexity) masked the truth a bit.

I've had similar experiences where my intuition tells me to be cautious but I could not say why. When I've ignored those intuitions I've generally paid a price. So now I do give them consideration. 

In such situations it is probably good to take some time to sit back and try to identify some of the things that triggered the response. We are very good at pattern matching but also really good at filtering. Could be that intuitions like "getting bad vibes" is all about the interaction of the two.

But that is a pretty difficult task, we're asking our self to go back and review all the details we ignored and filtered. But I suspect it is a very good thing to try doing.

The desires-values schema reminds me of Hirshmann's The Passions and the Interests and the problem he (IIRC) sees Machiavelli dealing with. Machiavelli is advising kings who he sees as more driven by their passions but to be successful need to be driven more by their interests.

Anyone in the alignment space take a look at Machiavelli in the light of how to get some thing that is more powerful (king versus the advisor) better aligned? Probably need to stretch things a bit to see that type of alignment as being aligned with the general welfare of the kingdom but seems like he was dealing with a similar problem when looked at from certain angles.

I like the point about the need for some type of external competitive measure but as you say, they might not be a MMA gym where you need one.

Shifting the metaphor, I think your observation of the sucker punch fits well with the insight that for those with only a hammer, all problems look like nails. The gym would be someone with a screwdriver or riveter as well as the hammer. But even lacking the external check, we should always ask ourselves "Is this really a nail?" I might only have a hammer but if this isn't a nail while the results might be better than nothing (maybe?) I'm not likely to achieve the best that could be done some other way. And,of course, watching just what happens went the other hammer-only people solve the problem and compare results to those when we know the problem is a nail we might learn something from their mistakes.

I agree with the view that punishment is not really a great deterrent as many crimes are not committed from a calculated cost-benefit perspective. I do think we need to apply that type of thinking towards what we might do with that insight/fact of things.

On that point, would like to see more on your claim that we would get better bang for the buck as it were from more investment in preventing crimes. In this regard I'm thinking about the contrast between western legal views and places like China as well as the estimates on the marginal pecuniary costs of prevention to the marginal pecuniary savings from reduced punishment. Clearly two (among many) difference margins along which trade-offs will need to be made.

Another aspect that seems worth exploring (and I'm sure it has been but not sure where the literature stands on the question) is how, at least in my understanding of USA criminal law, victims of crimes are not often compensated (white collar, fraud, financial crimes are something of an exception) but the victims, as a part of society, are then paying costs to punish the criminal. Full prevention is not a reasonable assumption (not sure what level is a reasonable assumption) but we might find a better solution even at the current rate of prevention if more of those harmed by crimes were actually compensated for the harms rather than just imposing the punishment of the criminal actor. A primary reason for preventing the event of a crime is the prevention of the harm. But if the harm can be largely mitigated after the fact there is a degree of equivalent between it never having occurred and it's compensation (perhaps here we might think of law and punishment as a type of insurance).

I also think there is something to look at in terms of prevention of incidence of crimes due to incarceration -- a type of exile. There might be scope for approaches there that maintain that type of prevention for repeat offenders (those demonstrating a propensity for some bad behavior for whatever reason) that may be possible at a lower cost than prison incarceration. And what might the marginal gain in prevention be related to the cost of the solution. This is a somewhat different approach than the ex anti prevention approach applied to the general society but may be nearly as effective but substantially lower cost.

Agree. There is that old saying about even fools learning from their own mistakes but wise men learning from the mistakes of others. But if everyone is trying to hide their mistakes, that might limit how much learning the wise can do.

I had not really thought about this before, but after seeing your comment the question struck me if social/cultural norms about social status and "loosing face" don't impact scientific advancement.

Nice write up and putting some light on something I think I have intuitively been doing but not quite realizing it. Particularly the impact on growth of wealth. 

I was thinking that a big challenge for a lot of people is the estimated distribution - which is likely why so many non-technical rationales are given by many people. Trying to assess that is hard and requires a lot of information about a lot of things -- something the insurance companies can do (as suggested by another comment) but probably overwhelms most people who buy insurance.

With that thought, I was wondering if anyone has thought of shifting the equations a bit. Rather than working up some estimate of the probability space, why not put an equation together that you might be able to churn out some probability distributions given W, P, d_i and c_i. for the break-even case. I think most people would be able to digest that, event x_i has implied probability p_i, event x_j has implied probability p_j. Then the person can think if those probabilities actually make sense to them and their situation. 

Clearly, it could not be an exhaustive listing of events but I would think a table of three or four of the main events that carry the greatest losses would be a good starting point for most people.

What is the price of the past? Kind of leading question but I've found myself wondering at times about the old saying about those who don't know the past are doom to repeat it. 

It's not that I don't think there is a good point to that view. However, when I look at the world around me I often see something that is vastly different from that view. I've come to summarize that as those who cannot let go of the past will never escape it. The implication is that not only those "clingy" people but also those around them will continue living whatever past it is they are attached to. As this is generally bad events of the past that means we get stuck living with these bad outcomes in the world. 

So while keeping that knowledge of the past in mind may prevent repeating it, that might be accomplished by never actually having learned from or moved past the history. I'm unsure which is worse.

Yes, all those conjectures are possible as we don't yet know what the reality will be -- it is currently all conjecture.

The counter argument to yours I think is just what opportunities is the AI giving up to do whatever humans might be left to do? What is the marginal value of all the things this ASI might be able to be doing that we cannot yet even conceive of? 

I think the suggestion of a negative value is just out of scope here as it doesn't fit into theory of comparative advantage. That was kind of the point of the OP. It is fine to say comparative advantage will not apply but we lack any proof of that and have plenty of examples where it actually does hold even when there is a clear absolute advantage for one side. Trying to reject the proposition by assuming it away seems a weak argument.

I'm not sure that is the correct take in the context of Comparative Advantage.

It would not matter if the SI could produce more than humans in a direct comparison but what the opportunity cost for the SI might be. If the ASI is shifting efforts that would have produced more value to it than it gets from the $77 sunlight output AND that delta in value is greater than the lower productivity of the humans then the trade makes sense to the ASI.

Seems to me the questions here are about resource constraints and whether or not an ASI does or does not need to confront them in a meaningful way.