Appreciate you putting this info out there, some comments on various aspects of this:
Scott Alexander recently made a post about using AI to evaluate grants, and notes the relationship to AI forecasting. I think this post gives some insight into issues that may come up with applying AI in these types of domains. Reading this, one thing that occurred to me is that I could imagine asking an AI to predict how you would rank grants, and it giving recommendations that match these pretty well, and that goes doubly so now that you've made this post, where someone could point an AI to the post itself and advise using that info to inform the prediction.
The question that this raises for Scott Alexander's autograntor is, if we ask the autograntor to evaluate grants, how do we know whether or not it is implicitly predicting what the user or a human evaluator would rate the grant, vs giving its own independent assessment? I think a similar question applies to AI forecasting systems as well. These tasks are kind of like doing research about a particular topic, and my guess is that model developers have at least to a certain extent trained their models to try to be factual, site sources, and not over-claim on research tasks. Even if a model is capable of superhuman forecasting or grant evaluation, its possible that any particular system for using a model to do these things may fail to elicit those capabilities.
This makes it pretty challenging to evaluate such systems on tasks that aren't easily verifiable, in my view. A natural approach is to compare to expert humans (e.g. Scott comparing to human evaluators, comparing FutureSearch to human forecasters). This works well on systems that are far below human capabilities because there is a lot of low hanging fruit and improvements on the matches-humans metric likely reflects real improvement. It kind of works at around human level because at least matching human outputs is an existence proof that the AI systems can perform at human level. It seems to my though that this doesn't really work above human level because you can't say whether deviation from human ratings reflects degradation or improvement. Once you allow that having different output from a human might actually reflect superior performance, similarity to human output fails to be useful as a measure of how well the system is performing.
Additionally, even using similarity to human outputs as a gut-check like Scott seems to be doing raises the question, doesn't that select for less useful outputs for a similar reason to why funders may have been interested in using your ratings, which is that outputs/ratings that are highly correlated with existing sources of information are less useful because that information is already available? In my comment on Scott's post and elsewhere, I also argue that this can lead to larger problems if people "double-count" sources of correlated information.
I generally agree with and appreciate people involved in these types of things being considerate of privacy concerns and therefore support sometimes leaving out certain information for that reason.
With that in mind, I think a consideration to keep in mind when funding lobbying/politics type projects is that how a candidate/organization/movement get funding often becomes its own political sub-issue. A common talking point I hear is about general dissatisfaction with "money in politics" and I think there are many specific examples of people or causes facing criticism for who supports them with money. Political funding that seems valuable in the near term could potentially backfire in the slightly longer term if it is viewed by the public negatively. It would not surprise me if major pro-AI-lab donors like Greg Brockman actually net-hurt the causes they were donating to for this reason.
I think I have a dimmer view of liability as a policy tool compared to many people interested in AI safety. I think the ideal that people have in their heads when they think about liability is that companies respond to liability by taking actions that decrease the likelihood of accidents or other liability-related events occurring. I agree that this may happen to some extent but my view is that a much more common and larger effect of liability is that companies respond by trying to decrease the extent to which they produce discoverable documents that might make them look bad (for nearest unblocked strategy reasons). Thus, liability as a tool of policy has much more of an anti-transparency effect than it does a safety effect.
I am an evaluator for Lightcone Commons (LC). LC is a platform for applicants to submit projects, for evaluators to rank them and for funders to choose the funding orders of several evaluators. It is not a foundation, in that it doesn’t have money that it spends how it chooses. And I do not have a permanent position working for Lightcone Commons. I get paid 2% of any money that a funder chooses to direct via my funding order. This pays for my time and also for any research that I pay others to do.
That said, what were my general and specific funding preferences?
This is a version of what Lightcone published, it might be wrong, but theirs was definitely wrong (their headline numbers didn’t include funding that had been hidden)
Lightcone also has a beautiful page for looking through public information here.
General preferences:
A quick caveat. When I talk about “what I recommended”, I am going to talk about dollars that moved according to my recommendations. About $5m was funded according to my recommendations (for which I was paid $100k). However because other people also recommended those things (and because some things were vetoed by funders) the final $ that moved according to my preferences was $11.2m down my list. So when I say ‘my dollars’ I mean that the dollars of recommendation that are allocated via me, which might be much smaller than the total amount a given organisation received.
What I recommended in the part that was funded, by type1.
Some organisations (mainly relating to policy) wished to remain private, so I have removed them. Specific rationales for redacted entries:
• As above I generally think AI is very important and reducing risk without necessarily leading to an immediate pause is a high priority. That’s true of several of the private entries, that seemed common sense risk reduction and were trusted by politics people I trust. If you are a wealthy person who can give to Republicans I have more recommendations.
• Then there are some lobbying orgs that seemed very good value for money that lobbied on a range of topics, run by people I trust.
Otherwise this is my funding order:
Again, not all this is ‘my’ ‘money’ since often these were partially filled up by others who agreed with my recommendations.
Rationales:
I want to note briefly how frank I am being here. Please disagree with me and I hope in future rounds I will feel my rationales are more robust, but this was the work and I did it. I am not going to pretend I always had better reasons than I say here (though sometimes I did, and can’t say) and sometimes I will just have more to say. If you think something was ludicrous or reasoning bad, please say so.
I think that’s the meat of the discussion, but here are some other comments:
How I differ from the other evaluators
I was surprised that $5m was routed by me and have been told that it was probably because my list looked so different to others. I wasn’t sure how much money I was going to get from this, nor is this my full-time job. I think I would have spent more time on it if I had realised $5m was going to be allocated through me. I think had I done so perhaps I would have had even more go via me.
It is a bit strange to be someone who can so dramatically affect the lives of ones friends but also feels strongly motivated to ‘say’ true things. I expect this to only get stranger in future rounds. I enjoy people thinking I am worth arguing with and convincing, though it’s hard having to rate projects lowly that one knows.
Please apply!
If I can give one piece of advice, please apply if you have an effective project. I want good projects to recommend at the top of my order.
I intend to write another article about what a good project and application looks like, but in short, I am interested in projects that focus on:
And a good application looks like:
But I’ll write these more carefully in another blog. Apply!
https://www.lightconecommons.com/apply
Also I am leading an epistemics fund, sponsored by Ought. You can read more about that here and I’d appreciate retweets when I write about it tomorrow morning:
https://epistemic.fund/