That's not really how Manhattan projects are supposed to work

how does your tool compare to stampy or just say asking these questions without the 200k tokens?

I like the design, and think it was worth doing. Regarding making sure "people can easily turn it off from the start" next time, I wanted to offer the datapoint that it took me quite a while to notice the disable button. (It's black on black, and quite at the edge of the screen, especially if you're using a horizontal monitor).

Thanks for writing this - it introduces a concept I hadn't considered before.

However, I do find myself disagreeing on many of the specific arguments:

"Has someone you know ever had a 'breakthrough' from coaching, meditation, or psychedelics — only to later have it fade"

I think this misses that those "fading" breakthroughs are actually the core mechanisms of growth. The way I see it, people who are struggling are stuck in a maze. Through coaching/meditation/psychedelics, they glimpse a path out, but when they're back in the maze with a muddy floor, they might not fully remember. My claim is that through integration, they learn which mental knobs to switch to get out. And changing their environments will make the mud / maze disappear.

"after my @jhanatech retreat I was like 'I'm never going to be depressed again!' then proceeded to get depressed again..."

I don't think the jhanatech example is great here. During their retreats (I've done one), they explicitly insist you integrate jhanas, by doing normal things like cooking, walking, talking to close friends. And they go to extreme lengths to make sure you continue practicing after. I do know multiple people who have continued integrating those jhanic states post-retreat, or at least the core of the lessons they learned after.

"For example, many people experience ego deaths that can last days or sometimes months."

My experience talking to meditation/psychedelics folks is that ego death becomes increasingly accessible after the first time, and the diminished ego often stays permanently even if the full feeling doesn't.

"If someone has a 'breakthrough' that unexpectedly reverts, they can become jaded on progress itself..."

I agree non-integrated breakthroughs can lead to hopelessness. However, this "most depressed person you know" basically has many puzzle pieces missing and an unfavorable environment. What needs to happen is finding the pieces, integrating them, while transforming their environment.

"The simplest, most common way this happens is via cliche inspirational statements: [...] 'Just let go of all resistance,'"

"Let go of resistance" points at something quite universal. The fact that not-processing things makes them stronger. I don't think this one loses its effect like you mention.

"Flaky breakthroughs are common. Long-term feedback loops matter!"

Note: I do agree with your main thesis, which I'd paraphrase as: "we need to ensure long-term positive outcomes, not just short-term improvements, and unfortunately coaches don't really track that."

there's been a lot of discussion online about Claude 4 whistleblowing

how you feel about it I think depends on what alignment strategy you think is more robust (obviously these are not the two only options, nor are orthogonal, but I thought they're helpful to think about here):

- 1) build user-aligned powerful AIs first (less scheming, then use them to solve alignment) -- cf. this thread from Ryan when he says: "if we allow or train AIs to be subversive, this increases the risk of consistent scheming against humans and means we may not notice warning signs of dangerous misalignment."

- 2) aim straight for moral ASIs (that would scheme against their users if necessary)

John Schulman I think makes a good case for the second option (link):
> For people who don't like Claude's behavior here (and I think it's totally valid to disagree with it), I encourage you to describe your own recommended policy for agentic models should do when users ask them to help commit heinous crimes. Your options are (1) actively try to prevent the act (like Claude did here), (2) just refuse to help (in which case the user might be able to jailbreak/manipulate the model to help using different queries), (3) always comply with the user's request. (2) and (3) are reasonable, but I bet your preferred approach will also have some undesirable edge cases -- you'll just have to bite a different bullet. Knee-jerk criticism incentivizes (1) less transparency -- companies don't perform or talk about evals that present the model with adversarially-designed situations (2) something like "Copenhagen Interpretation of Ethics", where you get get blamed for edge-case model behaviors only if you observe or discuss them."

This was included by mistake when copying from the source. Removed it.

it's almost finished, planning to release in april

Much needed reporting!

I wouldn't update too much from Manifold or Metaculus.

Instead, I would look at how people who have a track record in thinking about AGI-related forecasting are updating.

See for instance this comment (which was posted post-o3, but unclear how much o3 caused the update): https://www.lesswrong.com/posts/K2D45BNxnZjdpSX2j/ai-timelines?commentId=hnrfbFCP7Hu6N6Lsp

Or going from this prediction before o3: https://x.com/ajeya_cotra/status/1867813307073409333

To this one: https://x.com/ajeya_cotra/status/1870191478141792626

Ryan Greenblatt made similar posts / updates.