It seems to me that the relevant factor that makes a platform a "criminal enterprise" is not the absolute amount of crime that it enables, but the percentage of the activity on that that platform that is criminal in some way.

If 50% of meta's revenue comes from crime, then I'm more-or-less comfortable saying it's a criminal organization. If 0.01% of it's revenue comes from crime, but that happens to be a large total amount, I role my eyes at accusations that they're a criminal organization.

A quick search seems to indicate that their revenue in 2019 was about 70 billion dollars. So if I take your 16 billion figure at face value, about 23% of their revenue comes from enabling fraud (or did in 2019). 

This is higher than I was expecting, and high enough that I would be inclined to majorly hold the company accountable. It's a matter of taste whether or not it's high enough to declare the company "a criminal enterprise." 

if we pass a law to take billionaire's stuff, they'll use the same law to take our stuff.

I think this isn't particularly true in the medium term, and I think most people know that. That billionaires are unpopular is what makes expropriating their stuff politically feasible.

If an LLM is properly aligned, then it will care only about us, not about itself at all.

Perhaps this is what you mean, but, an aligned AI would instrumentally value itself, even if it didn't terminally value itself at all.

Also, it's not clear to me that an aligned AI wouldn't value itself. If humanity, on reflection, values it as a moral patient, then it should would too.

I'm tempted to say when this page was first published. 

Of course the core idea is older than that, going back to at least Moravec's calculations of when AGI would arrive based on number of neuron in the retina in Mind Children (1988). But until the late 2010s that point there was still a lot of uncertainty about the relative contribution of algorithms and compute to general AI. Hence the "brain in a box in a basement" model.

Perhaps it would be more correct to say "before the scaling hypothesis was really validated". 

I have not (to my knowledge and memory) signed a non-disparagement agreement with Palisade or with Survival and Flourishing Corp (the organization that runs SFF).

 

But it's harder for me to think of a principle that would protect a relatively autonomous society of relatively baseline humans from being optimized out of existence, without extending the same conservatism to other beings, and without being the kind of special pleading that doesn't hold up to scrutiny

If its possible for humans to consent to various optimizations to them, or deny consent, that seems like an important difference. Of course consent is a much weaker notion when you're talking about superhumanly persuasive AIs that can extract consent for ~anything, from any being that can give consent at all, so the (I think correct) constraint that superintelligences should get consent before transforming me or my society doesn't change the outcome at all.

I've talked to quite a few people and most people say it's is a good idea to use the myriad of other concerns about AI as a force multiplier on shared policy goals.

Speaking only for myself, here: There's room for many different approaches, and I generally want people to shoot the shots that they see on their own inside view, even if I think they're wrong. But I wouldn't generally endorse this strategy, at least without regard for the details of how the coalition is structured and what it's doing.

I think our main problem is a communication problem of getting people to understand the situation with AI 

• that model capabilities are steadily increasing; 
• that the labs are aiming at literal superintelligence, no really, something more capable than any human alive, and then even better than that; that the labs are explicitly aiming to do an RSI, which looks increasingly likely to succeed; 
• that there is not a known science of reliably controlling or shaping the motivations of superhuman AIs. 
• that there are competitive pressures for all of the labs and all of the countries to beat their competitors, so slowing down or pausing requires international coordination.

These are slippery points to get across specifically because audiences tend to slip into visualizing something other than "actual strategic superintelligence", that is automating science and technological progress and capable of strategically outmaneuvering adversaries—even when I talk with people from the labs, they often tend to gravitate to a fuzzier vision that has the form factor of the current AI chatbots / agents, but is much more competent.

Most of the time, I'm trying to land these points, despite the slipperiness, and talking about present-day harms that don't have a through-line to the core alignment problems seem like more of a distraction than a help.

If we already had developed policies that would substantially improve the situation and were politically feasible, and we just needed to get a big enough coalition to get them implemented, I would feel differently. 

But insofar as we have policies substantially help, they're rather radical (on the order of "don't allow private individuals to own more than 8 GPUs" and "negotiate with China for an international pause in frontier AI development"), and are only politically realistic if the stakeholders have a close-to-accurate picture of the situation. 
 

Citizen assemblies often involve selecting a small number of delegates who are then informed about the all of the details of the issue in depth, including by expert testimonies, which the delegates have the affordance to do because they're being paid for their time.

My understanding is that this works pretty well for coming to reasonable policy.

aspects of your motivations you're uncomfortable with

"your" and "you're", here refers to the AI and the AI's motivations, not the human and the human's motivation?

level 3 is "okay, we just won't be idiots about doing superficially useful looking things to the environment" 

But a key point is that 3 is, fundamentally, a skill-issue.

And maybe, given our human-capability levels, it's functionally a skill-issue for everyone. 

But it's a pretty common pattern for someone to try to do [something] (using a basically hopeless methodology, often), fail, and then declare that [something] is beyond the ken of man, and we must be humble in the face of our ultimate ignorance and impotence.

See for instance the way many (in my opinion, midwit) people declare that there is irreducible mystery, or that no one worldview and accommodate all of the problems of society.