I'd be ~entirely comfortable with this given some constraints (e.g. a simple heuristic which flags the kind of suspicious behaviour for manual review, and wouldn't capture the vast majority of normal LW users). I'd be slightly but not strongly uncomfortable with the unconstrained version.

Glanced through the comments and saw surprisingly positive responses, but reluctant to wade into a book-length reading commitment based on that. Are the core of your ideas on alignment compressible to fulfil the compelling insight heuristic?

hmm, I both see the incentive issue and also that the current widespread downvote marginally mitigates this. Not sure if it helps a lot to have lurker downvotes, and expect there are notable costs. Do you think there is a karma bar below which the EV of a downvote from those users is negative? My guess is at least totally new users add painful noise in a net negative way often enough that their contribution to keeping things bad things low is not worthwhile, and pushes away some good contributors lowering average quality.

I suspect you might be underestimating the how much some users take a psychological hit if they put effort into something and get slapped down without comment, having those be somewhat reliably not misfiring seems important.

(this is probably fairly minor on your list of things, no worries if you disengage)

Sorry to hear about that experience. I think that "downvote" should be a power you unlock when you're well-established on LW (maybe at 1k karma or so) rather than being universally available. The sting of a downvote on something you think is important is easily 50x the reward of an upvote, and giving that power to people who have little context on the community seems very bad EV.

Especially with LW becoming more in the public eye, letting random internetgoers who register give any LWer negative feedback (which is often painful/discouraging) seems pretty likely to be detrimental. I'd be interested in takes from the LW team on this.

Edit: Man, I love the disagree vote separation. It's nice people being able to disagree with me without downvoting.

This seems very closely related to a classic sequences post: Disguised Queries

AFFINE (Agent Foundations FIeld NEtwork) was set up and applied for SFF funding on behalf of several ex-MIRI members, but only got relatively small amounts of funding. We're thinking about the best currently possible model, but it's still looking like individuals applying for funding separately. I would be keen for a more structured org to pop up and fill the place, or for someone to join AFFINE and figure out how to make it a better home for AF.

Good call! I have a use for this idea :)

Depends what they do with it. If they use it to do the natural and obvious capabilities research, like they currently are (mixed with a little hodge podge alignment to keep it roughly on track), I think we just basically for sure die. If they pivot hard to solving alignment in a very different paradigm and.. no, this hypothetical doesn't imply the AI can discover or switch to other paradigms.

I think doom is almost certain in this scenario.

[epistemic status: way too ill to be posting important things]

hi fellow people-who-i-think-have-much-of-the-plot

you two seem, from my perspective as having read a fair amount of content from both, to have a bunch of similar models and goals, but quite different strategies.

on top of both having a firm grip on the core x-risk arguments, you both call out similar dynamics in capabilities orgs capturing will to save the world and turning it into more capabilities progress^[1], you both take issue with somewhat different but i think related parts of openphil's grantmaking process, you both have high p(doom) and not very comfortable timelines, etc.

i suspect if connor explained why he was focusing on the things he is here, that would uncover the relevant difference. my current guess is connor is doing a kind of political alliancebuilding which is colliding with some of habryka's highly active integrity reflexes.

maybe this doesn't change much, these strategies do seem at least somewhat collision-y as implemented so far, but i hope our kind can get along.

1. ^{^}

   e.g. "Turning care into acceleration" from https://www.thecompendium.ai/the-ai-race#these-ideologies-shape-the-playing-field
   
   e.g. https://www.lesswrong.com/posts/h4wXMXneTPDEjJ7nv/a-rocket-interpretability-analogy?commentId=md7QvniMyx3vYqeyD and lots of calling out Anthropic

If it's easy for submitters to check a box which says "I asked them and they said full post imports are fine", maybe?

No strong takes on default, just obvious considerations you'll have thought of.