I'm currently researching forecasting and epistemics as part of the Quantified Uncertainty Research Institute.
an estimated utility function is a practical abstraction that obscures the lower-level machinery/implementational details
I agree that this is what's happening. I probably have different intuitions regarding how big of a problem it is.
The main questions here might be something like:
My quick expected answers to this:
is that different "deliberation/idealization procedures" may produce very different results and never converge in the limit.
Agreed. This is a pretty large topic, I was trying to keep this essay limited. My main recommendation here was to highlight the importance of deliberation and potential deliberation levels, in part to better discuss issues like these.
Do you have a preferred distinction of value functions vs. utility functions that you like, and ideally can reference? I'm doing some investigation now, and it seems like the main difference is the context in which they are often used.
My impression is that Lesswrong typically uses the term "Utility function" to mean a more specific thing than what economists do. I.E. the examples of utility functions in economics textbooks. https://brilliant.org/wiki/utility-functions/ has examples.
They sometimes describe simple things like this simple relationship, as a "utility function".
I'm curious why this got the disagreement votes.
1. People don't think Holden doing that is significant prioritization?
2. There aren't several people at OP trying to broadly figure out what to do about AI?
3. There's some other strategy OP is following?
Also, I should have flagged that Holden is now the "Director of AI Strategy" there. This seems like a significant prioritization.
It seems like there are several people at OP trying to figure out what to broadly do about AI, but only one person (Ajeya) doing AIS grantmaking? I assume they've made some decision, like, "It's fairly obvious what organizations we should fund right now, our main question is figuring out the big picture."
Ajeya Cotra is currently the only evaluator for technical AIS grants.
This situation seems really bizarre to me. I know they have multiple researchers in-house investigating these issues, like Joseph Carlsmith. I'm really curious what's going on here.
I know they've previously had (what seemed to me) like talented people join and leave that team. The fact that it's so small now, given the complexity and importance of the topic, is something I have trouble grappling with.
My guess is that there are some key reasons for this that aren't obvious externally.
I'd assume that it's really important for this team to become really strong, but would obviously flag that when things are that strange, it's likely difficult to fix, unless you really understand why the situation is the way it is now. I'd also encourage people to try to help here, but I just want to flag that it might be more difficult than it might initially seem.
Thanks for clarifying! That really wasn't clear to me from the message alone.
> Though if you used Squiggle to perform an existential risk-reward analysis of whether to use Squiggle, who knows what would happen
Yep, that's in the works, especially if we can have basic relative value forecasts later on.
If you think that the net costs of using ML techniques when improving our rationalist/EA tools are not worth it, then there can be some sort of argument there.
Many Guesstimate models are now about making estimates about AI safety.
I'm really not a fan of the "Our community must not use ML capabilities in any form", not sure where others here might draw the line.
I assume that in situations like this, it could make sense for communities to have some devices for people to try out.
Given that some people didn't return theirs, I imagine potential purchasers could buy used ones.
Personally, I like the idea of renting one for 1-2 months, if that were an option. If there's a 5% chance it's really useful, renting it could be a good cost proposition. (I realize I could return it, but feel hesitant to buy one if I think there's a 95% chance I would return it.)
Happy to see experimentation here. Some quick thoughts:
Good point, fixing!