Yeah this one has been pretty high on my list (or, a fairly similar cluster of ideas)

Nod, this feels a bit at the intersection of what I had in mind with "Cyborgism", and the "Schleppy work in narrow domains" section.

Some thoughts: for this sort of thing, there's a hypothesis ("making it easier to change representations will enable useful thinking in hmath", and a bunch of annoying implementation details you need to test the hypothesis (i.e. actually getting an LLM to do all that work reliably).

So my next question here is "can we test out a version of this sort of thing powered by some humans-in-a-trenchcoat", or otherwise somehow test the ultimate hypothesis without having to build the thing." I'm curious for your intuitions on that

I think there's a possibility for ui people to make progress on the reputation tracking problem by virtue of tight feedback loops relative to people thinking more abstractly about it.

Are there particular reputation-tracking-problems you're thinking of? (I'm sure there are some somewhere, but I'm looking to get more specific)

I'm working on a poweruser LLM interface but honestly it's not going to be that much better than Harpa AI or Sider.

Curated. This post's framing resonated a lot with my own framing. I think the questions of how to cultivate impact, agency and taste are some of the more important questions that LessWrong tackles.

Much of this post were phenomena I'd observed myself, but, a few particular framings stood out to me as helpful crystallizations:

The first was "Don’t rely too much on permission or encouragement." I think a few Lightcone employees have also been slowly learning something along these lines. Our CEO has a lot of taste and vision, but sometimes one of us comes up with an idea that doesn't immediately resonate with him, and it's not until we actually build some kind of prototype ourselves that other people start to believe in it.

Another was:

Unfortunately, I am here to tell you that, at least if you are similar to me, you will never feel smart, competent, or good at things; instead, you will just start feeling more and more like everyone else mysteriously sucks at them.

For this reason, the prompt I suggest here is: what does it seem like everyone else is mysteriously bad at? That’s probably a sign that you have good taste there.

I had heard this sort of idea before, but this was the first time I parsed it as a technique you could explore on purpose. (i.e. actively investigate what things people seem mysteriously bad at, and use that to guide where you can maybe trust your taste more).

Finally:

The first domain that I got some degree of taste in was software design, and I remember a pretty clear phase transition where I gained the ability to improve my designs by thinking harder about them. After that point, I spent a lot of time iterating on many different design improvements—most of which I never implemented because I couldn’t come up with something I was happy enough with, but a few of which turned into major wins.

Recently I've been exploring the "think real hard about things" paradigm. This paragraph helped flesh out for me that there are some prerequisites for "think real hard" to work. I think some of this are skills that work across domains (i.e. "notice when you can't actually explain something very clearly" -> "you're still confused, try to break the confusion down"). But, it makes sense that some of it is domain specific. 

There's an important question I previously would have framed: "If you're tackling an unfamiliar domain, the question is 'how much mileage can you get from general, cross-domain reasoning skills?". But, a different question is "what's the minimum amount of 'domain specific' skill you need in order for 'think about it in advance' to really help?".

I do agree it’s suggestive, I’d be interested to see practical -> different practical.

This sounds like it might be a Paper Trauma -ish thing, which might have a different specific mechanism. 

I'm not sure if this is fiction. I realize there's something nice about opening in media res but I think this could use better indication of who this is for or what it is.

Nod, but fwiw if you don’t have a cached answer, I am interested in you spending like 15 minutes thinking through whether there exist startup-centric approaches to helping with x-risk that are good. 

Well yeah, but the question here is "what should be community guidelines on specifically how to approach startups that are aimed at specifically helping with AI safety" (which may or may not include AI), not "what kinds of AI startups should people start, if any?"

I didn’t track this previously: how did they incentive themselves to reach the cap?