I'm quoting the one that was linked, AI Prospects: Understanding Options in a Hypercapable World.
In my opinion, an article like this is not worth the time to decipher. There are probably some good ideas here but they're buried under a low signal to noise ratio.
Intelligence is a resource, not an entity
This is like Ryle-inspired pseudo-philosophy, I don't understand what these terms mean and why I am being told not to confuse them. And it doesn't connect with his next claim that superintelligent AIs don't need to be agents when structured workflows can steer them into having capabilites. I wish he'd dwell on this point more, but he never brings it up again.
... (read more)The crucial question, then, is what we should do with AI, not what “it” will
A few weeks ago, researchers at Fudan University published a paper claiming frontier AI systems had surpassed the self-replicating red line. Notably, they were able to do this with Llama31-70B-Instruct and Alibaba’s Qwen25-72B-Instruct, two large language models relatively weaker than the most capable frontier models. They were able to self-replicate 50% and 90% of the time, respectively, by making running copies of themselves on a separate filesystem.
Should this be concerning? Yes and no.
The research had some statistical gaps, and the actual self-replicating process amounted to little more than an LLM agent operating a command line to copy and run files, check port numbers, write simple scripts and check for errors.
On the other... (read 1073 more words →)
I think we have to postulate that this component of the RL signal doesn't get to see the chain-of-thought
If this is true, o1 can produce reasoning that is unsound, invalid, vacuous, etc. and will still be rewarded by the RL framework as long as the conclusion is true. In classical logic you can even formulate arguments that are unsound, invalid and vacuous but still true if q is true, like p ^ ~q -> q.
o1 is planning to deceive because it has been rewarded for offering plausible answers, not correct answers
It is not necessary to presume deception because o1 does not need to produce sound reasoning in order to produce correct answers, let alone plausible ones.... (read more)
Yeah, I'm sure this is not a typical example of his writing style or exposition of the ideas he's advocated for over the bulk of his career.