A thing unclear from the interaction: it had seemed towards the end that "build a profile to figure out where the bottleneck is" was one of the steps towards figuring out the problem, and that the LLM was (or might have been) better at that part. And, maybe models couldn't solve you entire problem wholesale but there was still potential skills in identifying factorable pieces that were better fits for models.

So, I think I need to distinguish between "Feedbackloop-first Rationality" (which is a paradigm for inventing rationality training) and "Ray's particular flavor of metastrategy", which I used feedbackloop-first rationality to invent" (which, if I had to give a name, I'd call "Fractal Strategy"^[1], but that sounds sort of pretentious and normally I just call it "Metastrategy" even though it's too vague)

Feedbackloop-first Rationality is about the art of designing exercises, and thinking about what sort of exercises apply across domains, thinking about what feedback loops will turn to out to help longterm, and which feedbackloops will generalize, etc.

"Fractal Strategy" is the art of noticing what goal you're currently pursuing, whether you should switch goals, and what tactics are appropriate for your current goal, in a very fluid way (while making predictions about those strategy outcomes).

Feedbackloop-first-rationality isn't actually relevant to most people – it's really only relevant if you're a longterm rationality developer. Most people just want some tools that work for them, they aren't going to invest enough to be inventing their own tools. Almost all my workshops/sessions/exercises are better framed as Metastrategy. 

I recently tried to give an impromptu talk that focused on Feedbackloop-first rationality (forcing myself out of a comfort zone of talking about practical metastrategy), and it floundered and sputtered and I eventually pivoted to making it a demo of fractal strategy that worked much better. 

This is probably mostly because I just didn't prepare for the talk. But I think it's at least partly because I'd previously been conflating them, and also, that I don't really know that many people who feel like the target audience for feedbackloop-rationality itself.

1. ^{^}

   But it sort of defies having a name because it involves fluidly switching between so many modes and tactics, it's hard to pin down what the central underlying move is. The "fractal" part is just one 

Yeah this one has been pretty high on my list (or, a fairly similar cluster of ideas)

Nod, this feels a bit at the intersection of what I had in mind with "Cyborgism", and the "Schleppy work in narrow domains" section.

Some thoughts: for this sort of thing, there's a hypothesis ("making it easier to change representations will enable useful thinking in hmath", and a bunch of annoying implementation details you need to test the hypothesis (i.e. actually getting an LLM to do all that work reliably).

So my next question here is "can we test out a version of this sort of thing powered by some humans-in-a-trenchcoat", or otherwise somehow test the ultimate hypothesis without having to build the thing." I'm curious for your intuitions on that

I think there's a possibility for ui people to make progress on the reputation tracking problem by virtue of tight feedback loops relative to people thinking more abstractly about it.

Are there particular reputation-tracking-problems you're thinking of? (I'm sure there are some somewhere, but I'm looking to get more specific)

I'm working on a poweruser LLM interface but honestly it's not going to be that much better than Harpa AI or Sider.

Curated. This post's framing resonated a lot with my own framing. I think the questions of how to cultivate impact, agency and taste are some of the more important questions that LessWrong tackles.

Much of this post were phenomena I'd observed myself, but, a few particular framings stood out to me as helpful crystallizations:

The first was "Don’t rely too much on permission or encouragement." I think a few Lightcone employees have also been slowly learning something along these lines. Our CEO has a lot of taste and vision, but sometimes one of us comes up with an idea that doesn't immediately resonate with him, and it's not until we actually build some kind of prototype ourselves that other people start to believe in it.

Another was:

Unfortunately, I am here to tell you that, at least if you are similar to me, you will never feel smart, competent, or good at things; instead, you will just start feeling more and more like everyone else mysteriously sucks at them.

For this reason, the prompt I suggest here is: what does it seem like everyone else is mysteriously bad at? That’s probably a sign that you have good taste there.

I had heard this sort of idea before, but this was the first time I parsed it as a technique you could explore on purpose. (i.e. actively investigate what things people seem mysteriously bad at, and use that to guide where you can maybe trust your taste more).

Finally:

The first domain that I got some degree of taste in was software design, and I remember a pretty clear phase transition where I gained the ability to improve my designs by thinking harder about them. After that point, I spent a lot of time iterating on many different design improvements—most of which I never implemented because I couldn’t come up with something I was happy enough with, but a few of which turned into major wins.

Recently I've been exploring the "think real hard about things" paradigm. This paragraph helped flesh out for me that there are some prerequisites for "think real hard" to work. I think some of this are skills that work across domains (i.e. "notice when you can't actually explain something very clearly" -> "you're still confused, try to break the confusion down"). But, it makes sense that some of it is domain specific. 

There's an important question I previously would have framed: "If you're tackling an unfamiliar domain, the question is 'how much mileage can you get from general, cross-domain reasoning skills?". But, a different question is "what's the minimum amount of 'domain specific' skill you need in order for 'think about it in advance' to really help?".

I do agree it’s suggestive, I’d be interested to see practical -> different practical.

This sounds like it might be a Paper Trauma -ish thing, which might have a different specific mechanism. 

I'm not sure if this is fiction. I realize there's something nice about opening in media res but I think this could use better indication of who this is for or what it is.

Nod, but fwiw if you don’t have a cached answer, I am interested in you spending like 15 minutes thinking through whether there exist startup-centric approaches to helping with x-risk that are good.