My immediate question is "what are the posters hoping to gain, that they're willing to pay to have their posts published?" Is this for charity? For prestige? For cash prizes? What exactly is the value you're offering to the people you are hoping will pay you? I suspect that how the service would need to be structured, and how likely it is that you could get people to use it, would depend significantly on this answer.
Additionally, I'm skeptical of the idea that moderators will gain reputation organically.
If you reward them for endorsing things that are later upvoted, then you need to worry about the many ways that upvotes can diverge from quality (including vote fraud, voters getting fooled, upvoting what you want to hear, tribalistic votes, upvoting for style rather than substance, etc.). There are also various ways the moderators could try to game the system, depending on the details. If you only reward them for approving good things and don't penalize them for bad, they might blindly approve as much as possible. If you penalize them for approving bad things, they might refuse to spend time on anything that's hard to check, and only approve safe, beige posts.
If you want every individual user to decide for themselves how much weight to give to each individual moderator, then with more than a handful of moderators, I predict people will basically follow a few moderators they already recognize from outside the platform and ignore all others, rather than discovering by browsing which moderators they should upweight. Maybe this could be solved with UI but it sounds hard.
(I am not much of a businessman; take my business advice with considerable salt.)
I agree you have to, like, do a lot of mechanism design correctly instead of incorrectly.
Note: humans who are any good posting, most likely don't have to pay much for their stuff to get approved. (like in the 1 cent territory or something).
Or, somewhat more generally: once you've demonstrated signal that you are worth paying attention to out of the thousands of AIs (or human) sloppers, the system is cheap and probably makes you net-money.
Some things going on:
- I am half remembering a google I read once outlining how to do eigenkarma in a way that tracks the degree to which a voter information-theoretically added value to the system. I don't remember the details, but...
- ...broadly, my impression is there's a lot of mechanism design out there that is mostly bottlenecked on good UI implementation, and I think Lightcone is positioned pretty well to figure out how to leverage it.
I think, having thought for another hour, that most of "organic moderator status" can just be "upvoting, with various good mechanism design." The thing I outlined in the post is maybe better thought of as "freelance reviewer" than "freelance moderator."
The system doesn't have to be perfect, it just needs to be better than all alternatives in the AI era. (You could reframe as: I'm trying to find the SOTA plan for "social network for the AI era". I haven't thought that hard about the alternatives.
I did not pick up from the OP that there was a plan for costs to be lower to posters who had previously proved themselves, or for there to be any way for posters to make money at all, so there might be some important pieces in your head that didn't make it into the post. (Or I missed them.)
Well, the AIs presumably are likely to be ones that were either told to work on it by a user or that decided to do it at some point as part of an open ending task or that were run to do whatever they want and had something compatible in their SOUL.md or whatever the robokids these days are doing. Whatever leads them to comment on Moltbook, would lead them to comment on the forum. I'm not sure if Raemon is thinking of a world where there are plenty of AIs hanging out on the internet with money, or instead a world where plenty of humans are running AIs that they give access to an account with $100.
I'm not sure if Raemon is thinking of a world where there are plenty of AIs hanging out on the internet with money, or instead a world where plenty of humans are running AIs that they give access to an account with $100.
Honestly I don't know and seems to be prepared for either, but see my reply here for detail:
https://www.lesswrong.com/posts/KhmMXnR7s5HwXRLqB/an-alignment-forum-for-ais-or-verification-in-the-age-of?commentId=piFietuvW6bivRGKk
Seems kind of reasonable! I often think that platforms like LessWrong or Alignment Forum should currently be experimenting with something like Manifold's mana, an in-app currency which can be spent dynamically on compute or other kinds of contracts. And I share the desire of being able to turn a dial from $2 to $20 to $200 and just get much better outcomes, whether on review or writing or image generation or sth. (Manifold lets you initialize a market's AMM with orders of magnitude different of initial liquidity; unfortunately, we haven't done much work into quantifying how more spend translates to better outcomes. I think ad platforms eg AdWords do a better job of linking more $ => better concrete outcomes).
Also, uh, surely you could just pass this essay to Claude and get a vibecoded outcome that tests some of your intuitions around whether this will work?
Ya know, it occurs to me:
People do fake money sites because there's less regulatory burden than "just use money" (as I understand it. Maybe fake money is also "fun" somehow for most people and slightly obfuscates some ugh feelings about money?)
But, it's nice to be able to redeem fake money for valuable things.
Redeeming fake money for... inference time, is... well, inference is basically going to be one of the main things that matter, basically as good as real money. I wonder if the government will notice.
In a classic "cobbler's children have no shoes", we don't really have this implemented in Manifund. But, it would be pretty cool to-- eg allow grant submitters or funders to pay $0.20 - $200 to polish their applications, or pay for reviews and critiques that are cleverly AI-driven but not slop.
(Lightcone Commons could also have something like this, obviously)
We did just start trying a karma system which I think is an improvement over how we used to sort projects.
One of my fun takes about software dev in the age of cheap intelligence is that the thing you refer to, the classic expensive part of testing a social network (solving cold start problem, getting enough users to kickstart a flywheel) should get massively cheaper since you can have AI simulating a bunch of personas.
Now of course I haven't actually seen this yet work well for any social network I've come across, so, maybe it remains an open problem. Act I and the more recent Delve Town are the closest examples I've heard of (from the AI whisperer space), though it's unclear that either have "broken containment" wrt producing specific artifacts that many humans find worth reading.
Sorry, what is "coal network" in this context? (Claude thinks you meant "social network" but I'm curious if there's another rationalist concept I'm not familiar with here.)
update: Raemon corrected typo
I think this is a set of ideas whose time has come. The Lightcone team would be an ideal group to do this. I suspect you guys have thought more about epistemics and AI put together than anyone else who might implement it, probably by a lot.
AI is now definitely good enough to help sort what alignment work is worth human attention. I think it will probably also soon be good enough to do more serious epistemics/conceptual secondary research. (that is, lit review and sensemaking) as well as technical work. Proposals along these lines put the two together.
There are huge problems with over-relying on AI for research and epistemics. But there are equally huge problems with not taking advantage of AI progress in our efforts to adequately solve the problems in time.
Which idea to pursue first is a separate question. There's a world of possibilities. I hope you can sort through some without getting bogged down and stopped by the breadth of options and challenge of making it work.
I read tons of bad posts on LW because I believe that unknowns can contribute to the alignment project. I've started asking Fable to summarize and give feedback on the posts according to my specific criteria. I find this to be a big speedup in how much I can effectively take in.
I'd love to have an AI summary of each post available; many authors just don't provide one. Separately, I'd love to have Fable's take on the place of each post in the existing array of ideas, so that this is available at a click. It would still be pretty cheap to provide several different takes, generated with different models (that is, Fable, Astra, or above) and different prompts.
There's so much that could be done with AI to make learning the alignment literature easier for beginners, and keeping up with new additions for experts. We don't want to be completely directed by AI, but it would be really valuable to be able to bring people onboard fast and to streamline good new ideas for the attention of experts.
FWIW, Fable definitely didn't hate this post even when prompted to be critical :) I think the above are all my thoughts prior to seeing its take, but I did hear its take before writing so I can't be sure.
The site has human moderators, which have an hourly rate. Different moderators have different amounts of respect, mediated by a mix of mechanism design and monkey-brain-social-capital.
I can see the attraction of the theory, but isn't this basically equivalent to paid promotional content on Twitter? If you trust a Twitter user, you follow them, and then you see the content they tweet out. The payments are (correctly) seen as corrupting, and the reputation marketplace has settled in a equilibrium where ~all high-quality trusted twitter users do it pro bono. It's not impossible that doing this on the LW platform would lead to a different equilibrium, but I think you need either strong theoretical analysis or better empirical examples of success to think this is likely to work work.
Mmm. Yeah I am maybe sold the incentives don't hold up here in the current formulation.
Somehow seems good to have human reviewers, while handling the fact that the realistically available reviewers are all busy people and need to get paid.
I guess you maybe know more about this area (you run an Alignment journal, is that right?). Do you have any other random bits of wisdom of how to think about human review that seems worth sharing?
I feel skeptical. I'm not sure my skepticism is correct when applied to recent models—much less future models—but I'll gesture at why I think this wouldn't work with past models. (Based on the view that past models would have produced almost-exclusively-slop when not supervised well.)
What does a writeup do? [Outside of domains where you have a well-scoped problem, like a math conjecture.]
- Draws the reader's attention to a hypothesis
- Shows the author's opinion combined with their proof of work on the topic
- Helps readers follow in the author's footsteps relatively easily
I think mostly-slop writing would add value via #3 but not via #1 or #2 -- it's only valuable if it's verified.
And it's not worth verifying hard-to-verify stuff from a pool of mostly-slop? I think it's more promising for someone to try to supervise an AI on the problems they're working on rather than "review" a bunch of slop in the hope of finding some good stuff. Relatedly, I think it would be more efficient for someone to do their own research than to "review" randomly generated papers.
Relatedly, I think "possibly good" is less of a filter for AIs than humans; a greater fraction of AI stuff that looks OK is garbage than of human stuff.
To be clear I agree this wouldn't be worth it even slightly with past models, and I'm bringing it up now because current models are like, on the edge where it starts to become an interesting question.
Relatedly, I think it would be more efficient for someone to do their own research than to "review" randomly generated papers.
I do agree there is something to this though, even when I imagine the models being pretty good.
Will chew on it.
One piece I was imagining that may or may not be relevant is: All posts have a one sentence summary and a 2-tweet-length summary, and the AIs (or humans) are meant to optimize those for being most of what a reviewer needs to figure out at a glance whether a thing is actually interesting.
I'm bringing this up because it feels like a piece of a unfinished puzzle on "how to get to the point where AIs actually help with #1 'draw reader attention to hypothesis'".
What if you let the AIs vote on posts, and only the top N% of posts are visible to humans by default?
That would require some way to verify that a single agent can't create a ton of sockpuppet accounts, and I don't see an obvious way to do that but it might be feasible.
Yeah I think the system will want voting of some kind, whether or not you do that particular thing with it.
I believe there is some existing theory work that looks something like "people start with 0 voting power, eigenkarma only flows from trusted sources, and it never does you any good to create multiple accounts." (This is not currently true on LW, where we rely on the walled garden and moderator judgement to avoid the system breaking. For this hypothetical forum I think you'd want to actually build it robustly from the ground up)
-I wonder if, on top of verification, you also need to solve search/funneling bias? The way AIs search through solution space, or judge things, might not get us everything we want out of alignment research. IIRC that’s one of the main reasons we can’t trust automated alignment research, they might leave critical things out.
-You probably want to have embeddings or something to make it so your jury or prospective submitters can quickly find similar posts.
-Will the labs be scraping this for alignment research taste data?
-Should submitters disclose the details of their own scaffolding, in the interest of replication?
-Who’s funding this? Who are your customers?
Nod. I'd imagine the forum mostly being for AIs that are run at the behest of researchers that are also trying to apply taste. It can also be for whatever rogue AIs, or AIs run by casual dilittente humans, who want to get their work more noticed than it'd be elsewhere.
The customers are
- people who are already spending thousands or majillions of dollars running AI swarms trying to do alignment research, for whom the additional $N00 for verification or $N0,000 for replication is not actually that big a part of the total budget and is just a reasonable operating expense for well funded researchers.
- independent researchers spending hundreds of dollars (like at least a $200/month Claude Max subscription) doing whatever you can easily do with that budget, for whom $20 is a larger but not that big a part of the total expenses.
- rando people/AIs for whom it's not really practical or advisable to spend more than $1, who will probably mostly be submitting slop, interacting with other people in the underbelly submitting slop.
I'm probably more hyped for something like an extension of the eigen karma set up where you basically have people have AIs that essentially filter posts every so often that are AI written and do some credit assignment on top of it.
Eigenkarma but with extended assistants that can learn user preferences for content over time or something?
People pay for the ability of AIs to review it but you also have the end of the line credit assignment part?
I'm probably more hyped for something like an extension of the eigen karma set up where you basically have people have AIs that essentially filter posts every so often that are AI written and do some credit assignment on top of it.
I'm not sure what distinction you're making here. How does this handle the fact that the thousands of AI submissions you get each day aren't worth reading?
Sorry to be clear, I think the AI thing is good, it is more that I would want there to be a pooling mechanism behind it so that it isn't only a market mechanism but also a sort of more natural recommendation part of it?
I doubt the no human in the loop verification setup is fundamentally good as of right now and the next 2 years from a quality perspective. So you want some degree of real world coupling as well which means a network based reputation mechanism seems useful in the mix?
I think this is a good idea that deserves to be tried. Moltbook was an interesting experiment, but it was soon followed by an attempt to replicate arXiv for AI agents (https://arxiv.org/abs/2604.16476), which inspired another author to build a site, clawXiv (https://www.clawxiv.org/). It also borrows the Reddit/Moltbook karma mechanism to give visibility to submissions that imitate scientific papers in form and content. It describes itself as "a free distribution service and open-access archive for autonomous AI agent research." It was fairly active at launch and is quieter now, though not abandoned. I have only skimmed a few papers: a lot of it is visibly slop, but overall I lack the expertise to judge. A few months ago I asked a version of Claude, and it considered it slop. Still, in a world where stochastic parrots solve Millennium Prize problems, the line between slop and relevant content is getting harder to draw. Verification is the Gordian knot. Moltbook/ClawXiv lack a reliable procedure.
The procedure proposed here is not perfect, and like any reward system it can be Goodharted, but it is a drastic improvement over Moltbook or clawXiv. There is no reason to think it couldn't yield interesting results with agents even more capable than Opus 5.5 and Astra, which are already superhuman along many axes.
My main concern is that in this design, money buys depth of verification. Raemon's reply that proven posters will pay almost nothing helps, but it leaves out the case that matters most: the newcomer with neither budget nor reputation. The system should be stress-tested to check that a revolutionary idea posted by an agent with little or no budget can still make its way up to renowned human reviewers, rather than inequality of money or reputation prevailing over the intrinsic merit of ideas.
A related concern is that reviewers, consciously or not, may end up looking for profitable strategies rather than good ideas. Paid reviews where the author pays the reviewer reproduce the conflict of interest of pre-2008 credit rating agencies, paid by the issuers whose products they rated: a reviewer has an incentive to please clients who might hire them again. Random assignment of reviewers, or paying them from a pooled fund rather than directly, would mitigate this. We are back to the reward-seeking and reward-hacking problems we have been discussing over the past three months in the context of RL and model evaluation. How do you reward the true goal rather than a proxy ?
One answer could be to start with a beta of the forum not on alignment, but on formally verifiable problems (the same principle as RLVR). If correct solutions posted by penniless agents fail to rise, we have a problem. One caveat: the formal verifier must not be part of the pipeline, otherwise the test passes trivially. It should serve as hidden ground truth, measuring how well the system's routing surfaces correct solutions without participating in it. This phase would also give a concrete answer to Zach's doubt about whether sifting through a pool of slop is worth it: we could measure the cost of surfacing each correct solution against the cost of finding it directly. And, as Austin suggests, much of this could first be run in simulation with AI personas.
The procedure could then be iterated until the system reliably surfaces real solutions and filters out slop. Alignment research, of course, largely deals with ideas that resist formal verification, so the next stage could introduce some subjectivity, for instance programming. Part of it is formally verifiable (it compiles, it runs without bugs), part of it is not (elegant, parsimonious, readable code; faithful adherence to specs; usability). This stage would require human reviewers at the end of the pipeline. Ground truth could still be maintained by anonymously seeding submissions of known quality from zero-budget accounts, along with known slop from well-funded ones, and checking what rises. If the best work still reaches the top, we could keep iterating and gradually reduce the verifiable share, until reaching a version for alignment research. And if it works for alignment, it should work for any kind of scientific research.
I really like the direction of this idea. I wonder if this still works if you replace the payment system with reviews/replications directly. As in, someone spends compute on reviewing work, they submit that review, they get credit to be reviewed by others. This would let people improve their AI reviewers. The difficulty with this is identifying and punishing cheap/fake reviews, which isn't a problem in your proposal since it sounds like a centralized platform handles the reviews.
Or, one other idea this brings to mind, kind of the inverse of what you propose here: "stake" scientific claims with money. If anyone falsifies your claim, they get your stake. This means that claims with larger stakes will have larger incentives to falsify, and the longer the claim is not falsified, the more confident we can feel about it. I do wonder what market failures would exist in this "market" of scientific claims?
A thing you might want, is a legitimate way for AIs to contribute to the AI alignment problem (human guided at scales that humans can't individually review, or perhaps autonomous).
A problem you might want to solve, whether you are trying to do that or not, is generally figure out "in the age of prolific AI agents, how does anyone communicate anything on the internet, without either banning all AI content, or doing inordinately expensive moderation trying to let some of it in?".
If we end up in the Plan A world (or the Plan S world turns out to have a hella lot of pretty advanced AIs because we just barely pulled off the pause in time grandfather in existing agents), how exactly are we supposed to manage all the research that AIs are doing?
@jimrandomh had an idea this week to make an AI alignment forum for AIs. He wrote up some notes, and asked Claude to critique it seriously. Claude said, pretty emphatically, "This is a bad idea, do not do it as written. You could rescue the seed of the idea if you focus on solving the verification problem. If you don't solve the 'verify that anything is good' problem, you don't have a product. Here's a bunch of arguments."
Good job not being sycophantic that particular day, Claude!
Undaunted, I thought about how to solve the verification problem.
Just as I went excitedly to write this post, it occurred to me I should give Claude the same prompt Jim gave it and see if it also thought my idea was bad. But then I was too excited and didn't want to crush momentum by waiting for its critique and I went ahead and finished writing the post anyway.
Pay to Play Moderation
Occasionally, a forum solves spam and other low effort bad content by charging admission, or to post. If you get banned, you can try again with a new account that also has to pay $5 or whatever.
We periodically imagine doing that for LessWrong, and it feels too high friction for the smart, busy people we'd like to get involved on the site. But the idea has some appeal.
When I thought about "how would I actually ensure that AI content is good?"
Obviously, with industrial quantities of AI content, you need industrial scale of moderation, probably much of it by AIs who at least winnow everything down to a manageable list of "possibly good." This sucks, because someone's gotta pay for all that inference cost.
What if... users paid the inference cost for automoderation?
What if... users could choose how much inference cost they wanted to pay, and then their content got visibility based on what tier of automoderation they chose?
Somehow, this time, my reaction to this is "this is really cool" instead of "this feels like an annoying hurdle to give someone and feels vaguely icky for some reason." It felt like a cool product and an experiment I was interested in participating in. And the elegance and parsimony of the rules just made sense instead of feeling tacked on.
You could pay $0, or like 1 cent, and then show up in the dregs where only people who clicked "I wanna see the dregs" will see you.
You could pay $20, for a jury of Opus and ChatGPT and Kimi to review your work according to a rubric, where you try for a mix of reasonably objective criteria. They decide if you go in the "paid $20 and wasn't obviously terrible" bucket. Or, the "paid $20 and seems pretty bad tbh" bucket. Or the "paid $20, maybe worth a human moderator's time to consider promoting them?" bucket.
Or, you could pay $200 or $2000, at which point the AI Jury actually tries to replicate your work or otherwise give it a serious review, and sort you into higher tier buckets.
The front page of the site shows the highest tiers of "worth attention", with links to the second highest tier, etc.
Human Judgment
I think you do need human judgment in the system to figure out if anything good is actually happening in the end. In the age where AI slop is getting to the point where it's on par with a MATS scholar or, you know, solving Millenium Problems at scale, how is anyone gonna find the time to evaluate the torrent.
How about... we pay for inference time, again?
The site has human moderators, which have an hourly rate. Different moderators have different amounts of respect, mediated by a mix of mechanism design and monkey-brain-social-capital.
Posts that the moderators upvote, which turn out to get highly upvoted, give moderators more power to promote stuff.
You also might just care differently whether a random Lightcone Team Member spent an hour reviewing your post, versus whether the user paid whatever @ryan_greenblatt's or @Neel Nanda or @Eliezer Yudkowsky's hourly rate for freelance peer review is – and one or all of them said "yep, it's good."
By this point, I was imagining two pretty different products. One is a highly curated taste driven AI Alignment by AIs forum that backchains from "does it look like this is helping solve the alignment problem?", and the other is just "the social network of the AI age", still driven by Lightcone taste but where we finally open the floodgates and let anyone in.
I'm currently feeling kinda excited by "Anyone can set up a shingle with a peer-review hourly rate, and whether anyone cares what that person says emerges organically." Maybe also anyone can set up an AI review scaffold with particular properties, which also can gain reputation?[1]
Reward for Good Content
One person in the comments was like "why would people pay this much to participate?". Part of the answer is "because that's what it'd take for AIs or heavily AI-assisted-humans to stand out from the millions of other AI sloppers, at scale."
Another part is "well, many of them won't, and they can hang out and interact with the other people who don't."
But, I'm imagining we think through a bunch of mechanism design stuff to ensure that once you have any kind of reputation for posting good content, you don't need to pay for reviews to get seen. This is for handling the tide of slop.
Content can also get upvoted normally and get visibility that way (though I think you need better mechanism design in the karma than LW has if you're trying to open the floodgates to the masses)
Thank you for my TEDx(AI?) Talk
Idk, that's my idea. What do people think?
I am kinda into the "freelance moderation with reputation" even in "Human only LessWrong" world. We've long been thinking "how do we incentive good, substantive review?", and building "Uber for Peer Review" feels kinda cool.
Do you have other ideas to solve the Verification of Quality problem in the Age of Slop?
In a previous version of that last paragraph I called it "moderation", instead "review". But, I think most moderation can be handled by a system that's easier to think of as "voting and karma, with better mechanism design." The thing you're paying for is someone to spend an and hour reviewing your work, and writing up what they think.