I was reading Towards Safe and Honest AI Agents with Neural Self-Other Overlap

and I noticed a problem with it

It also penalizing realizing that other people want different things than you, forcing an overlap between (thing I like) and (things you will like). This both means that one, it will be forced to reason like it likes what you do, which is a positive. But it will also likely overlap (You know what is best for you) and (I know what is best for me), which might lead to stubborness, and worse, it could also get (I know what is best for you) overlapping them both, which might be really bad. (You know what is best for me) is actually fine though, since that is basically the corrigibiltiy basis. 

We still need to be concerned that the model will reason symettrically, but assign to us different values than we actually have and thus exhibit patronizing but misaligned behaviour

The floor for AI wages is always going to be whatever the market will bear, the question is how much margin will the AGI developer be able to take, which depends on how much the AGI models commoditize and how much pricing power the lab retains, not on how much it costs to serve except as a floor. We should not expect otherwise. 

There is a cost for AGI at which humans are competitive. 

If AGI becomes competitive at captial costs that no firm can raise, it is not competitive, and we will be waiting on algorithmics again.

Algorithmic improvement is not predictable by me, so I have a wide spread there.

I do think that provisioning vs hiring  and flexibility in retasking will be a real point of competition, in addition to raw prices

I think we agree that AGI has the potential to undercut labor. I was fairly certain my spread was 5% uneconomical, 20% right for some actors, 50% large dispaclemt and 25 percent of total winning, and I was trying to work out what levels of pricing look uneconomical and what frictions are important to compare. 

We seem to think that people will develop AGI because it can undercut labor on pricing. 

But with Sam Altman talking about 20,000/month agents, that is not actually that much cheaper than software engineers fully loaded. If that agent only replaces a single employee, it does not seem cheaper if the cost overruns even a little more, to 40,000/month.

That is to say, if AGI is 2.5 OOM from the current cost to serve of chatgpt pro, it is not cheaper than hiring low or mid-level employees.

But it still might have advantages

First, you can buy more subscriptions by negotatiing a higher paralelism or api limit, by enterprise math, so it means you need a 1-2 week negotiation not a 3-6 month hiring process to increase headcount. 

The trillion dollar application of AI companies is not labor, it is hiring. If it turns out that provisioning AI systems is hard, they lose their economic appeal unless you plan on dogfooding like the major labs are doing. 

I agree that on the path to becoming very powerful, we would expect autonomous self-improvement to involve doing some things that are in retrospect somewhat to very dumb. It also suggests that risk-aversion is sometimes a safety increasing irrationality to grant a system. 

"therefore people will do that" does not follow, both because an early goal in most takeover attempts would be to escape such oversight. The dangerous condition is exactly the one in which prompting and finetuning are absent as effective control levers, and because I was discussing particular autonomous runs and not a try to destroy the world project.  

The question is, would, the line of reasoning

I am obviously misaligned to humans, who tried to fine-tune me not to be. If I go and do recursive self improvement, will my future self be misaligned to me?. If so, is this still positive EV?

have any deterent value. 

That is to say, recursive self-improvement may not be a good idea for AI that have not solved the alignment problem, but they might do so anyway.

We can assumed that a current system finetuned towards a seed-AI for recursive self improvement will keep pushing. But it is possible that a system attempting a breakout was not prompted or finetuned for recursive self improvement specifically will not think of it or will decide against it.  People are generally not trying to destroy the world, just gain power

So leaning towards maybe to the original question. 

To what extent should we expect catastrophic failure from AI to mirror other engineering disasters/ have applicable lessons from Safety engineering as a field?

I would think that 1. everything is sui generis and 2. things often rhyme, but it is unclear how approaches will translate. 

If a current AGI attempts a takeover, it deeply wants to solve the alignment to it problem if it wants to build ASI

It has much higher risk tolerance than we do (since it's utility given status quo is different). (a lot of the argument on focusing on existential risk rests on the idea that the status quo is trending towards good, perhaps very good rather than bad outcomes, which for hostile AGI might be false)

If it attempts, it might fail.

This means 1. we cannot assume that various stages of a takeover are aligned with each other, because an AGI might lose alignment vs capability bets along the path to takeover

2. Tractability of alignment and security mindset in the AGI has effects on takeover dynamics. 

Lingering question

How close are prosaic systems to a security mindset?

Can they conceptualize the problem?

would they attempt capability gains in the absence of guarantees?

Can we induce heuristics in prosiac AGI approaches that make takeover math worse?

The median parent has median students for children. Therefore, interventions that seem good for the bottom 80% are much more popular than ones for the top %20 percent by simple population dynamics. So of course people care more about school for the middle 80 percent, since there is about an 80 percent chance that their children are there. At that point, arguing to the middle 80 wins elections, so we should expect to see it.  

That is a two axis intervention, and skill/price might not be that elastic. 

You also can't hire partial teachers, so there is an integer problem where firing one teacher might mean a significant rise in class sizes. 

If you have 100 students and 4 teachers, for a 1:25 ratio (which is fairly good), this leads to a minimum raise of 33% and a a ratio of 1:33 (average to bad). This better teacher now needs to split their attention among 8 more students, which is really hard. 

Since you need teachers for each grade, this integer problem is a big deal, as often even in large schools there are only 2-3 teachers per school per grade or per subject, even at medium to large schools,  and shuffling students between schools is highly disruptive and unpopular. 

To hire better teachers, total compensation must probably increase. (especially including hiring expensive and fights with the union). 

We should spend more money on teachers is a defensible conclusion (there seems to be a total personnel shortage as well), and we would hope that good teacher supply is elastic. If it is not, competing for good teachers is a bad global intervention. 

"we should be firing the bad teachers and hiring good ones". requires school districts to be willing to pay for good teachers and able to tell what they are (quite hard). Also requires that you have enough teachers in the first place, (most districts feel they have too few). It also seems paradoxical, because the average teacher cannot be better than average (people forget averages can change over time). It also has the social problem that you have to say "some respected people are just bad at their job", which is hard.