All of Vael Gates's Comments + Replies

FAQ

This is cool! Why haven't I heard of this?
Arkose has been in soft-launch for a while, and we've been focused on email outreach more than public comms. But we're increasingly public, and are in communication with other AI safety fieldbuilding organizations! 

How big is the team?

3 people: Zach Thomas and Audra Zook are doing an excellent job in operations, and I'm the founder.

How do you pronounce "Arkose"? Where did the name come from?

I think whatever pronunciation is fine, and it's the name of a rock. We have an SEO goal for arkose.org to surpass t... (read more)

Does Anyuan(安远) have a website? I haven't heard of them and am curious. (I've heard of Concordia Consulting https://concordia-consulting.com/ and Tianxia https://www.tian-xia.com/.)

Small update: Two authors gave me permission to publish their transcripts non-anonymously!

Interview with Michael L. Littman (https://docs.google.com/document/d/1GoSIdQjYh21J1lFAiSREBNpRZjhAR2j1oI3vuTzIgrI/edit?usp=sharing)

Interview with David Duvenaud (https://docs.google.com/document/d/1lulnRCwMBkwD9fUL_QgyHM4mzy0al33L2s7eq_dpEP8/edit?usp=sharing)

Two authors gave me permission to publish their transcripts non-anonymously! Thus:

• Interview with Michael L. Littman (https://docs.google.com/document/d/1GoSIdQjYh21J1lFAiSREBNpRZjhAR2j1oI3vuTzIgrI/edit?usp=sharing)

• Interview with David Duvenaud (https://docs.google.com/document/d/1lulnRCwMBkwD9fUL_QgyHM4mzy0al33L2s7eq_dpEP8/edit?usp=sharing)

Anonymous comment sent to me, with a request to be posted here:

"The main lede in this post is that pushing the materials that feel most natural for community members can be counterproductive, and that getting people on your side requires considering their goals and tastes. (This is not a community norm in rationalist-land, but the norm really doesn’t comport well elsewhere.)"

was this as helpful for you/others as expected?

I think these results, and the rest of the results from the larger survey that this content is a part of, have been interesting and useful to people, including Collin and I. I'm not sure what I expected beforehand in terms of helpfulness, especially since there's a question "helpful with respect to /what/", and I expect we may have different "what"s here.

are you planning related testing to do next?

Good chance of it! There's some question about funding, and what kind of new design would be worth funding,... (read more)

My guess is that people were aware (my name was all over the survey this was a part of, and people were emailing with me). I think it was also easily inferred that the writers of the survey (Collin and I) supported AI safety work far before the participants reached the part of the survey with my talk. My guess is that my having written this talk didn't change the results much, though I'm not sure which way you expect the confound to go? If we're worried about them being biased towards me because they didn't want to offend me (the person who had not yet pai... (read more)

Yeah, we were focusing on shorter essays for this pilot survey (and I think Richard's revised essay came out a little late in the development of this survey? Can't recall) but I'm especially interested in "The alignment problem from a deep learning perspective", since it was created for an ML audience.

Whoa, at least one of the respondents let me know that they'd chatted about it at NeurIPS -- did multiple people chat with you about it? (This pilot survey wasn't sent out to that many people, so curious how people were talking about it.)

Edited: talking via DM

Thanks! (credit also to Collin :))

Agreed that status / perceived in-field expertise seems pretty important here, especially as seen through the qualitative results (though the Gates talk did surprisingly well, given not an AI researcher, but the content reflects that). We probably won't have [energy / time / money] + [we have limited access to researchers] to test something like this, but I think we can hold "status is important" as something pretty true given these results, Hobbhann's (https://forum.effectivealtruism.org/posts/kFufCHAmu7cwigH4B/lessons-learned-from-talking-to-greater-than... (read more)

(Just a comment on some of the above, not all)

Agreed and thanks for pointing out here that each of these resources has different content, not just presentation, in addition to being aimed at different audiences. This seems important and not highlighted in the post.

We then get into what we want to do about that, where one of the major tricky things is the ongoing debate of "how much researchers need to be thinking in the frame of xrisk to make useful progress in alignment", which seems like a pretty important crux, and another is "what do ML researchers thi... (read more)

These results were actually embedded in a larger survey, and were grouped in sections, so I don't think it came off as particularly long within the survey. (I also assume most people watched the video at faster than 1x.) People also seemed to like this talk, so I'd guess that they watched it as or more thoroughly than they did everything else. We don't have analytics regretfully. (I also forgot to add that we told people to skip the Q&A, so we had them watch the first 48m.)

I told him I only wanted the bare-bones of interactions, and he's been much better to work with!

I see there's an associated talk now! https://www.youtube.com/watch?v=EIhE84kH2QI

Re: "Targeted Outreach to Experienced Researchers"

Please apply to work with the aforementioned AISFB Hub! I am actively trying to hire for people who I think would be good fits for this type of role, and offer mentorship / funding / access to and models of the space. Note that you'll need to have AI safety knowledge (for example, I want you to have read / have a plan for reading all of the main readings in the AGISF Technical Curriculum) and high generalist competence, as two of the most important qualifications.

I think most people will not be a good fit f... (read more)

Some Rudi communication style anecdotes:

• Rudi: "Aren't you a beautiful young woman!" almost immediately when we saw each other on video call for the first time (I identify as nonbinary) (<-- this anecdote is a from a few years ago and from memory, though, so he might have just said something quite similar)
• Rudi, in a Google Calendar invite note, as a closing: "Let's talk, dear Vael...as I recall, we liked each other a lot.:)"
  
  • Me in an email back:
    
    "(Ah, and just one other quick note: in the Google Calendar invite, you've included the line "Let's talk, dear V

Seems like it's great to do one-on-ones with people who could be interested and skilled from all sorts of fields, and top researchers in similar fields could be a good group to prioritize! Alas, I feel like the current bottleneck is people who are good fits to do these one-on-ones (I'm looking to hire people, but not currently doing them myself); there's many people I'd ideally want to reach. 

Thanks for doing that Kat!

Sure! This isn't novel content; the vast majority of it is drawn from existing lists, so it's not even particularly mine. I think just make sure the things within are referenced correctly, and you should be good to go!

With respect to the fact that I don't immediately point people at LessWrong or the Alignment Forum (I actually only very rarely include the "Rationalist" section in the email-- not unless I've decided to bring it up in person, and they've reacted positively), there's different philosophies on AI alignment field-building. One of the active disagreements right now is how much we want new people coming into AI alignment to be the type of person who enjoy LessWrong, or whether it's good to be targeting a broader audience. 

I'm personally currently of the o... (read more)

A great point, thanks! I've just edited the "There's also a growing community working on AI alignment" section to include MIRI, and also edited some of the academics' names and links.

I don't think it makes sense for me to list Eliezer's name in the part of that section where I'm listing names, since I'm only listing some subset of academics who (vaguely gesturing at a cluster) are sort of actively publishing in academia, mostly tenure track and actively recruiting students, and interested in academic field-building. I'm not currently listing names of resea... (read more)

I've been finding "A Bird's Eye View of the ML Field [Pragmatic AI Safety #2]" to have a lot of content that would likely be interesting to the audience reading these transcripts. For example, the incentives section rhymes with the type of things interviewees would sometimes say. I think the post generally captures and analyzes a lot of the flavor / contextualizes what it was like to talk to researchers.

It was formatted based on typical academic "I am conducting a survey on X, $Y for Z time", and notably didn't mention AI safety. The intro was basically this:

My name is Vael Gates, and I’m a postdoctoral fellow at Stanford studying how productive and active AI researchers (based on submissions to major conferences) perceive AI and the future of the field. For example:

- What do you think are the largest benefits and risks of AI?

- If you could change your colleagues’ perception of AI, what attitudes/beliefs would you want them to have?

My response rate w... (read more)

Thanks Alex :). Comment just on this section:
 

"The annoying thing here is that I believe the only difference between me and another task doer in this situation is that I have more accurate beliefs, or I have a higher belief threshold for making claims (or something similar, like that I only use statement for communicating beliefs and not for socially enforcing a commitment to myself)."

As someone who was in this situation with Alex recently (wanting a commitment from him, in order to make plans with other people that relied on this initial commitment),... (read more)

"Alpha Zero scales with more computing power, I think AlphaFold 2 scales with more computing power, Mu Zero scales with more computing power. Precisely because GPT-3 doesn't scale, I'd expect an AGI to look more like Mu Zero and particularly with respect to the fact that it has some way of scaling."

I thought GPT-3 was the canonical example of a model type that people are worried about will scale? (i.e. it's discussed in https://www.gwern.net/Scaling-hypothesis?)

Recently I was also trying to figure out what resources to send to an economist, and couldn't find a list that existed either! The list I came up with is subsumed by yours, except:
- Questions within Some AI Governance Research Ideas
- "Further Research" section within an OpenPhil 2021 report: https://www.openphilanthropy.org/could-advanced-ai-drive-explosive-economic-growth
- The AI Objectives Institute just launched, and they may have questions in the future