current inference scaling methods tend to be tied to CoT and the like, which are quite transparent

Aschenbrenner in Situational Awareness predicts illegible chains of thought are going to prevail because they are more efficient. I know of one developer claiming to do this (https://platonicresearch.com/) but I guess there must be many.

Related, I have a vague understanding on how product safety certification works in EU, and there are multiple private companies doing the certification in every state.

Half-informed take on "the SNPs explain a small part of the genetic variance": maybe the regression methods are bad?

Not sure if I missed something because I read quickly, but: all these are purely correlational studies, without causal inference, right?

OpenAI is recklessly scaling AI. Besides accelerating "progress" toward mass extinction, it causes increasing harms. Many communities are now speaking up. In my circles only, I count seven new books critiquing AI corps. It’s what happens when you scrape everyone's personal data to train inscrutable models (computed by polluting data centers) used to cheaply automate out professionals and spread disinformation and deepfakes.

Could you justify that it causes increasing harms? My intuition is that OpenAI is currently net-positive without taking into account future risks. It's just an intuition, however, I have not spent time thinking about it and writing down numbers.

(I agree it's net-negative overall.)

Ok, that. China seems less interventionist, and to use more soft power. The US is more willing to go to war. But is that because the US is more powerful than China, or because Chinese culture is intrinsically more peaceful? If China made the killer robots first, would they say "MUA-HA-HA actually we always wanted to shoot people for no good reason like in yankee movies! Go and kill!"

Since politics is a default-no on lesswrong, I'll try to muddle the waters by making a distracting unserious figurative narration.

Americans maybe have more of a culture of "if I die in a shooting conflict, I die honorably, guns for everyone". Instead China is more about harmony&homogenity, "The CCP is proud to announce that in 2025 the Harmonious Agreement Quinquennal Plan in concluded successfully; all disagreements are no more, and everyone is officially friends". When the Chinese send Uighurs to the adult equivalent of school, Americans freak out: "What? Mandated school? Without the option of shooting back?"

My doubt is mostly contingent on not having first-hand experience of China, while I have of the US. I really don't trust narratives from outside. In particular I don't trust narratives from Americans right now! My own impression of the US changed substantially by going there in person, and I even am from an allied country with broad US cultural influence.

[Alert: political content]

About the US vs. China argument: have any proponent made a case that the Americans are the good guys here?

My vague perspective as someone not in China neither in the US, is that the US is overall more violent and reckless than China. My personal cultural preference is for US, but when I think about the future of humanity, I try to set aside what I like for myself.

So far the US is screaming "US or China!" while creating the problem in the first place all along. It could be true that if China developed AGI it would be worse, but that should be argued.

I bet there is some more serious non-selfish analysis of why China developing AGI is worse than US developing AGI, I just have never encountered it, would be glad if someone surfaced it to me.

I agree it's not a flaw in the grand scheme of things. It's a flaw for using it for consensus for reasoning.

I start with a very low prior of AGI doom (for the purpose of this discussion, assume I defer to consensus).

You link to a prediction market (Manifold's "Will AI wipe out humanity before the year 2100", curretly at 13%).

Problems I see with using it for this question, in random order:

1. It ends in 2100 so the incentive is effectively about what people will believe a few years from now, not about the question. It is a Keynesian beauty contest. (Better than nothing.)
2. Even with the stated question, you win only if it resolves NO, so it is strategically correct to bet NO.
3. It is dynamically inconsistent, if you think that humans have power over the outcome and that such markets influence what humans do about it. Illustrative story: "The market says P(doom)=1%, ok I can relax and not work on AI safety" => everyone says that => the market says P(doom)=99% because no AI safety work => "AAAAH SOMEONE DO SOMETHING" => marker P(doom)=1% => ...

This type of issue is a huge effective blocker for people with my level of skills. I find myself excited to write actual code that does the things, but the thought of having to set everything up to get to that point fills with dread – I just know that the AI is going to get something stupid wrong, and everything’s going to be screwed up, and it’s going to be hours trying to figure it out and so on, and maybe I’ll just work on something else. Sigh. At some point I need to power through.

Reminds me of this 2009 kalzumeus quote:

I want to quote a real customer of mine, who captures the B2C mindset about installing software very eloquently: “Before I download yet another program to my poor old computer, could you let me know if I can…”  Painful experience has taught this woman that downloading software to her computer is a risky activity.  Your website, in addition to making this process totally painless, needs to establish to her up-front the benefits of using your software and the safety of doing so.  (Communicating safety could be an entire article in itself.)