User Comment Replies

Gauging Interest for a Learning-Theoretic Agenda Mentorship Programme

This is slightly off-topic, but you mentioned that you think that other research agendas could be fruitful. How would you rate singular learning theory (SLT) in this context? Do you see connections between SLT and LTA, for example if you try to generalize SLT to reinforcement learning? Are there dangerous topics to avoid?

Bandgaps, Brains, and Bioweapons: The limitations of computational science and what it means for AGI

Frank_R2y20

Arguments like yours are the reason why I do not think that Yudkowskys scenario is overwhelmingly likely ( P > 50%). However, this does not mean that existintial risk from AGI is low. Since smart people like Terence Tao exist, you cannot prove with complexity theory that no AGI with the intelligence of Terence Tao can be build. Imagine a world where everyone has one or several AI assistants whose capabilities are the same as the best human experts. If the AI assistants are deceptive and are able to coordinate, something like slow disempowerment of human... (read more)

The Learning-Theoretic Agenda: Status 2023

Frank_R2y10

You are right and now it is clear, why your original statement is correct, too. Let $U$ be an arbitrary computable utility function. As above, let $u = U (o_{0} a_{0} o_{1} a_{1} \dots)$ and $(u - ϵ, u + ϵ) \subseteq (a, b)$ with $ϵ > 0$ and $ϵ \in Q$ . Choose $P$ as in your definition of "computable". Since $P (s, ϵ)$ terminates, its output depends only on finitely many $o_{i_{1}}, \dots, o_{i_{k}}, a_{j_{1}}, \dots, a_{j_{l}}$ . Now

{s^{'} = o_{0}^{'} a_{0}^{'} o_{1}^{'} a_{1}^{'} \dots | o_{i_{1}}^{'} = o_{i_{1}}, \dots, a_{j_{l}}^{'} = a_{j_{l}}}

is open and a subset of $U^{- 1} (u - ϵ, u + ϵ)$ , since $| P (s^{'}, ϵ) - u | < ϵ$ .

The Learning-Theoretic Agenda: Status 2023

Frank_R2y20

I have discovered another minor point. You have written at the beginning of Direction 17 that any computable utility function $U : (O \times A)^{ω} \to [0, 1]$ is automatically continuous. This seems to be not always true.

I fix some definitions to make sure that we talk about the same stuff. For reasons of simplicity, I assume that $O$ and $A$ are finite. Let $(O \times A)^{ω}$ be the space of all infinite sequences with values in $O \times A$ . The $i$ -th projection $p_{i} : (O \times A)^{ω} \to O \times A$ is given by

p_{i} (o_{1} a_{1} o_{2} a_{2} \dots) = o_{i} a_{i}

The product topology is defined a... (read more)

2Vanessa Kosoy2y

Your alleged counterexample is wrong, because the U you constructed is not computable. First, "computable" means that there is a program P which receives s and ϵ∈Q as inputs s.t. for all s and ϵ, it halts and |P(s,ϵ)−U(s)|<ϵ Second, even your weaker definition fails here. Let ϵ=12. Then, there is no program that computes U within accuracy ϵ, because for every n, U(0n20ω)=34 while U(0ω)=0. Therefore, determining the value of U(s) within ϵ requires looking at infinitely many elements of the sequence. Any program would that outputs 0 on 0ω has to halt after reading some finite m symbols, in which case it would output 0 on 0m20ω as well.

The Learning-Theoretic Agenda: Status 2023

Frank_R2yΩ230

I have a question about the conjecture at the end of Direction 17.5. Let $U_{1}$ be a utility function with values in $[0, 1]$ and let $f : [0, 1] \to [0, 1]$ be a strictly monotonous function. Then $U_{1}$ and $U_{2} = f \circ U_{1}$ have the same maxima. $f$ can be non-linear, e.g. $f (x) = x^{2}$ . Therefore, I wonder if the condition $u (y) = α v (y) + β$ should be weaker.

Moreover, I ask myself if it is possible to modify $U_{1}$ by a small amount at a place far away from the optimal policy such that $π$ is still optimal fo... (read more)

3Vanessa Kosoy2y

No, because it changes the expected value of the utility function under various distributions. Good catch, the conjecture as stated is obviously false. Because, we can e.g. take U2 to be the same as U1 everywhere except after some action which π∗ doesn't actually take, in which case make it identically 0. Some possible ways to fix it: * Require the utility function to be of the form U:Oω→[0,1] (i.e. not depend on actions). * Use (strictly) instrumental reward functions. * Weaken the conclusion so that we're only comparing U1 and U2 on-policy (but this might be insufficient for superimitation). * Require π∗ to be optimal off-policy (but it's unclear how can this generalize to finite g).

Infra-Exercises, Part 1

Frank_R3y10

I have two questions that may be slightly off-topic and a minor remark:

Is a list of open and tractable problems related to Infra-Bayesianism somewhere available?
Do you plan to publish the results of the Infra-Bayesianism series in a peer-reviewed journal? I understand that there are certain downsides; mostly that it requires a lot of work, that the whole series may be too long for a journal article and that the peer review process takes much time. However, if your work is citeable, it could attract more researchers, who are able to contribute.
On page 22, you should include the condition a(bv) = (ab)v into the definition of a vector space.

Frank_R3y10

Thank you for the link. This clarifies a lot.

Frank_R3y10

I am starting to learn theoretical stuff about AI alignment and have a question. Some of the quantities in your post contain the Kolmogorov complexity of U. Since it is not possible to compute the Kolmogorov complexity of a given function or to write down a list of all function whose complexity is below a certain bound, I wonder how it would be possible to implement the PreDCA protocol on a physical computer.

3Tamsin Leake3y

Like all the other uncomputable or untractable logic in the post, the AI is to make increasingly informed guesses about them using something like logical induction, where one can estimate the likelihood of a logical statement without having to determine its truth value for sure.

LESSWRONG
LW

All of Frank_R's Comments + Replies