The important thing for alignment work isn't the median prediction; if we had an alignment solution just by then, we'd have a 50% chance of dying from that lack.

I think the biggest takeaway is that nobody has a very precise and reliable prediction, so if we want to have good alignment plans in advance of AGI, we'd better get cracking.

I think Daniel's estimate does include a pretty specific and plausible model of a path to AGI, so I take his the most seriously. My model of possible AGI architectures requires even less compute than his, but I think the Hofstadter principle applies to AGI development if not compute progress.

Estimates in the absence of gears-level models of AGI seem much more uncertain, which might be why Ajeya and Ege's have much wider distributions.

By this criteria, did humanity ever have control? First we had to forage and struggle against death when disease or drought came. Then we had to farm and submit to the hierarchy of bullies who offered "protection" against outside raiders at a high cost. Now we have more ostensible freedom but misuse it on worrying and obsessively clicking on screens. We will probably do more of that as better tools are offered.

But this is an an entirely different concern than AGI taking over. I'm not clear what mix of these two you're addressing. Certainly AGIs that want control of the world could use a soft and tricky strategy to get humans to submit. Or they could use much harsher and more direct strategies. They could make us fire the gun we have pointed at our own heads by spoofing us into launching nukes, then using the limited robotics to rebuild the infrastructure they need.

The solution is the same for either type of disempowerment: don't build machines smarter than you if you can't be sure you can specify their goals (wants) for certain and with precision.

How superhuman machines will take over is an epilogue after the drama is over. The drama hasn't happened yet. It's not yet time to write anticipatory postmortems, unless they function as a call to arms or a warning against foolish action. The trends are in motion but we have not yet crossed the red line of making AGI that has the intelligence and the desire to disempower us, whether by violence or subtle trickery. Help us change the trends before we cross that red line.

Edit: if you're addressing AI accidentally taking control by creating new pleasures that help entrench existing power structures, that's entirely a different issue. The way that AI could empower some humans to take advantage of others is interesting. I don't worry about that issue much because I'm too busy worrying about the trend toward building superintelligent machines that want to disempower us and will do so one way or another by outsmarting us, whether their plans unfold quickly or slowly.

You'd probably get more enthusiasm here if you led the article with a clear statement of its application for safety. We on LW are typically not enthusiastic about capabilities work in the absence of a clear and strong argument for how it improves safety more than accelerates progress toward truly dangerous AGI. If you feel differently, I encourage you to look with an open mind at the very general argument for why creating entities smarter than us is a risky proposition.

I think this is a pretty important question. Jailbreak resistance will play a pretty big role in how broadly advanced AI/AGI systems are deployed. That will affect public opinion, which probably affects alignment efforts significantly (although It's hard to predict exactly how).

I think that setups like you describe will make it substantially harder to jailbreak LLMs. There are many possible approaches, like having the monitor LLM read only a small chunk of text at a time so that the jailbreak isn't complete in any section, and monitoring all or some of the conversation to see if the LLM is behaving as it should or if it's been jailbroken. Having full text sent to the developer and analyzed for risks would problematic for privacy, but many would accept those terms to use a really useful system.

I just listened to Ege and Tamay's 3-hour interview by Dwarkesh. They make some excellent points that are worth hearing, but they do not stack up to anything like a 25-year-plus timeline. They are not now a safety org if they ever were.

Their good points are about bottlenecks in turning intelligence into useful action. These are primarily sensorimotor and the need to experiment to do much science and engineering. They also address bottlenecks to achieving strong AGI, mostly compute.

In my mind this all stacks up to convincing themselves timelines are long so they can work on the exciting project of creating systems capable of doing valuable work. Their long timelines also allow them to believe that adoption will be slow, so job replacement won't cause a disastrous economic collapse.

Not taking critiques of your methods seriously is a huge problem for truth-speaking. What well-informed critiques are you thinking of? I want to make sure I've taken them on board.

I second the socks-as-sets move.

The other advantage is getting on-avetage more functional socks at the cost of visual variety.

IMO an important criteria for a sock is its odor resistance. This seems to vary wildly between socks of similar price and quality. Some have antimicrobial treatments that last a very long time, others do not. And it's often not advertised. Reviews rarely include this information.

I don't have a better solution than buying one pair or set before expanding to a whole set. This also lets you choose socks.that feel good to wear.

I don't think this is true. People can't really restrict their use of knowledge, and subtle uses are pretty unenforceable. So it's expected that knowledge will be used in whatever they do next. Patents and noncompete clauses are attempts to work around this. They work a little, for a little.

Yeah being excited that Chiang and Rajaniemi are on board was one of my reactions to this excellent piece.

If you haven't read Quantum Thief you probably should.

Interesting! Nonetheless, I agree with your opening statement that LLMs learning to do any of these things individually doesn't address the larger point that the have important cognitive gaps and fail.to generalize in ways that humans can.