Mathematics is essentially the art/science of making some statements about some kinds of values, and proving those statements. You need some working definition of "value", "statement" and "proof".
Set theory is a possible approach to define values, because you can use sets to emulate things like numbers, graphs, functions, etc. Logic is about making statements. Gödel theorem is related to proofs. If you want to teach a computer to do math, you probably want to work with these.
How is math related to alignment? Well, most things about alignment we don't know how to solve yet -- like, how to actually extract the human values. But assuming we solve this problem one day, we will probably also want the AI to reason correctly. That involves some math, for example using the Bayesian theorem. Should the AI try to improve itself, it better be good at reasoning about code and algorithms, which involves more math. And it may need to be good at reasoning about math itself.
This is not a 100% proof that at some moment the set theory will be needed (perhaps the AI can avoid this part of math, or can rediscover it on its own using some other parts of math), but given that it plays an important role in math today, and that math in general will be needed, it seems like a good idea for the AI researcher to know it.
(This is just my guess, I am not involved with MIRI.)
It's just generally useful math background. Things like set theory, logic, category theory, etc. are the modern building blocks of mathematical modeling. I don't think there's anything specific about at theory and alignment that's important, only that you can't get very far in things directly relevant to alignment, like decision theory, without a good baseline of set theory knowledge.
Thank you, but I would say it is too general answer. For example, suppose your problem is to figure out planet motion. You need calculus, that's clear. So, according to this logic, you would first need to look at the building blocks. Introduce natural numbers using Peano axioms, then study their properties, then introduce rational, and only then construct real numbers. And this is fun, I really enjoyed it. But does it help to solve the initial problem? Not at all. You can just introduce real numbers immediately. Or, if you care only about solving mechanics... (read more)