LESSWRONG
is fundraising!
LW
$

by Rohin Shah
AI Alignment Forum
1 min read
2

67

Ω 35

This is a linkpost for https://rohinshah.com/faq-career-advice-for-ai-alignment-researchers/

To quote Andrew Critch:

I get a lot of emails from folks with strong math backgrounds (mostly, PhD students in math at top schools) who are looking to transition to working on AI alignment / AI x-risk. There are now too many people “considering” transitioning into this field, and not enough people actually working in it, for me, or most of my colleagues at Stuart Russell’s Center for Human Compatible AI (CHAI), to offer personalized mentorship to everyone who contacts us with these qualifications.

From math grad school to AI alignment, Andrew Critch

I’m pretty sure he wrote that at least 4 years ago (2016 or earlier). The field has grown enormously since then, but so has the number of people considering it as a research area. So far, I’ve tried to give at least 10 minutes of my time to anyone who emails me with questions; that probably won’t be sustainable for much longer. So now I’m answering the questions I get most frequently.  I hope to keep this up to date, but no promises.

Usually, I write a blog post when I think I have something important and novel to say, that I am relatively confident in. That’s not the case for this post. This time, I’m taking all the questions that I frequently get and writing down what I’d say in response. Often, this is (a) not that different from what other people would say, and (b) not something I’m very confident in. Take this with more grains of salt than usual.

Thanks to Neel Nanda, Nandi Schoots, and others who wish to remain anonymous for contributing summaries of conversations.

See the linked post for the FAQ; which will hopefully be kept up to date over time.

CareersAI
Frontpage

67

Ω 35

FAQ: Advice for AI Alignment Researchers
New Comment
2 comments, sorted by
top scoring
Click to highlight new comments since:
[-]johnswentworthΩ7150

I still don’t know much about GANs, Gaussian processes, Bayesian neural nets, and convex optimization.

FWIW, I'd recommend investing the time to learn about convex optimization even if you don't think you need it yet. Unlike the other topics on this list, the ideas from convex optimization are relevant to a wide variety of things the name does not suggest. Some examples:

  • how to think about optimization in high dimensions (even for non-convex functions)
  • using constraints as a more-gearsy way of representing relevant information (as opposed to e.g. just wrapping that information into an objective function directly)
  • quantifying slackness/tautness (which is one of the main benefits of representing information as constraints)
  • useful intuitions for making numerical algorithms efficient, especially leveraging sparsity and recognizing failure modes of poor condition numbers

Personally, I consider convex optimization ideas fundamentally useful for thinking in a similar way to linear algebra or multivariate calculus.

(Note: I first learned the topic mainly from Boyd's lectures; he's unusually good at conveying many useful intuitions in those talks, so you might get less value from other lecturers or the canonical convex optimization book.)

Reply
[-]Rohin ShahΩ440

Yeah that's definitely the one on the list that I think would be most useful.

I may also be understating how much I know about it; I've picked up some over time, e.g. linear programming, minimax, some kinds of duality, mirror descent, Newton's method.

Reply
Moderation Log
2Comments