Strong endorse; I was discussing this with Daniel, and my read of various materials is that many labs are still not taking this as seriously as they ought to - working on a post about this, likely up next week!

Very useful post! Thanks for writing it.

is robust to ontological updates

^ I think this might be helped by an example of the sort of ontological update you'd expect might be pretty challenging; I'm not sure that I have the same things in mind as you here

(I imagine one broad example is "What if AI discovers some new law of physics that we're unaware of", but it isn't super clear to me how that specifically collides with value-alignment-y things?)

I appreciate the question you’re asking, to be clear! I’m less familiar with Anthropic’s funding / Dario’s comments, but I don’t think the magnitudes of ask-vs-realizable-value are as far off for OpenAI as your comment suggests?

Eg, If you compare OpenAI’s reported raised at $157B most recently, vs. what its maximum profit-cap likely was in the old (still current afaik) structure.

The comparison gets a little confusing, because it’s been reported that this investment was contingent on for-profit conversion, which does away with the profit cap.

But I definitely don’t think OpenAI’s recent valuation and the prior profit-cap would be magnitudes apart.

(To be clear, I don’t know the specific cap value, but you can estimate it - for instance by analyzing MSFT’s initial funding amount, which is reported to have a 100x capped-profit return, and then adjust for what % of the company you think MSFT got.)

(This also makes sense to me for a company in a very competitive industry, with high regulatory risk, and where companies are reported to still be burning lots and lots of cash.)

If the companies need capital - and I believe that they do - what better option do they have?

I think you’re imagining cash-rich companies choosing to sell portions for dubious reasons, when they could just keep it all for themselves.

But in fact, the companies are burning cash, and to continue operating they need to raise at some valuation, or else not be able to afford the next big training run.

The valuations at which they are raising are, roughly, where supply and demand equilibriate for the amounts of cash that they need in order to continue operating. (Possibly they could raise at higher valuations from taking on less-scrupulous investors, but to date I believe some of the companies have tried to avoid this.)

Interesting material yeah - thanks for sharing! Having played a bunch of these, I think I’d extend this to “being correctly perceived is generally bad for you” - that is, it’s both bad to be a bad liar who’s known as bad, and bad to be good liar who’s known as good (compared to this not being known). For instance, even if you’re a bad liar, it’s useful to you if other players have uncertainty about whether you’re actually a good liar who’s double-bluffing.

I do think the difference between games and real-life may be less about one-time vs repeated interactions, and more about the ability to choose one’s collaborators in general? Vs teammates generally being assigned in the games.

One interesting experience I’ve had, which maybe validates this: I played a lot of One Night Ultimate Werewolf with a mixed-skill group. Compared to other games, ONUW has relatively more ability to choose teammates - because some roles (like doppelgänger or paranormal investigator, or sometimes witch) essentially can choose to join the team of another player.

Suppose Tom was the best player. Over time, more and more players in our group would choose actions that made them more likely to join Tom’s team, which was basically a virtuous cycle for Tom: in a given game, he was relatively more likely to have a larger number of teammates - and # teammates is a strong factor in likelihood of winning.

But, this dynamic would have applied equally in a one-time game I think, provided people knew this about Tom and still had a means of joining his team.

Possibly amusing anecdote: when I was maybe ~6, my dad went on a business trip and very kindly brought home the new Pokémon Silver for me. Only complication was, his trip had been to Japan, and the game was in Japanese (it wasn’t yet released in the US market), and somehow he hadn’t realized this.

I managed to play it reasonably well for a while based on my knowledge of other Pokémon games. But eventually I ran into a person blocking a bridge, who (I presumed) was saying something about what I needed to do before I could advance. But, I didn’t understand what they were saying because it was in Japanese.

I had planned to seek out someone who spoke Japanese, and ask their help translating for me, but unfortunately there was almost nobody in my town who did. And so instead I resolved to learn Japanese - and that’s the story of what led to me becoming fluent at a young age.

(Just kidding - after flailing around a bit with possibly bypasses, I gave up on playing the game until I got the US version.)

To me, this seems consistent with just maximizing shareholder value. … "being the good guys" lets you get the best people at significant discounts.

This is pretty different from my model of what happened with OpenAI or Anthropic - especially the latter, where the founding team left huge equity value on the table by departing (OpenAI’s equity had already appreciated something like 10x between the first MSFT funding round and EOY 2020, when they departed).

And even for Sam and OpenAI, this would seem like a kind of wild strategy for pursuing wealth for someone who already had the network and opportunities he had pre-OpenAI?

Just guessing, but maybe admitting the danger is strategically useful, because it may result in regulations that will hurt the potential competitors more. The regulations often impose fixed costs (such as paying a specialized team which produces paperwork on environmental impacts), which are okay when you are already making millions.

My sense of things is that OpenAI at least appears to be lobbying against regulation moreso than they are lobbying for it?

I don’t think you intended this implication, but I initially read “have been dominating” as negative-valenced!

Just want to say I’ve been really impressed and appreciative with the amount of public posts/discussion from those folks, and it’s encouraged me to do more of my own engagement because I’ve realized how helpful their comments/posts are to me (and so maybe mine likewise for some folks).

It’s interesting to me that the big AI CEOs have largely conceded that AGI/ASI could be extremely dangerous (but aren’t taking sufficient action given this view IMO), as opposed to them just denying that the risk is plausible. My intuition is that the latter is more strategic if they were just trying to have license to do what they want. (For instance, my impression is that energy companies delayed climate action pretty significantly by not yielding at first on whether climate change is even a real concern.)

I guess maybe the AI folks are walking a strategic middle ground? Where they concede there could be some huge risk, but then also sometimes say things like ‘risk assessment should be evidence-based,’ with the implication that current concerns aren’t rigorous. And maybe that’s more strategic than either world above?

But really it seems to me like they’re probably earnest in their views about the risks (or at least once were earnest). And so, political action that treats their concern as disingenuous is probably wrongheaded, as opposed to modeling them as ‘really concerned but facing very constrained useful actions’.