All of Jan's Comments + Replies

Ah, yes, definitely doesn’t apply in that situation in full generality! :) Thanks for engaging!

Oh yeah, should have added a reference for that!

The intuition is that the defender (model provider) has to prepare against all possible attacks, while the defender can take the defense as given and only has to find one attack that works. And in many cases that actually formalises into an exponential-linear relationship. There was a Redwood paper where reducing the probability of generating a jailbreak randomly by an order of magnitude only increases the time it takes contractors to discover one by a constant amount. I also worked out some theory here but that was quite messy.

Nostalgebraist’s new essay on… many things? AI ontology? AI soul magic?

The essay starts similarly to Janus’ simulator essay by explaining how LLMs are trained via next-token prediction and how they learn to model latent properties of the process that produced the training data. Nostalgebraist then applies this lens to today’s helpful assistant AI. It’s really weird for the network to predict the actions of a helpful assistant AI when there is literally no data about that in the training data. The behavior of the AI is fundamentally underspecified and only ... (read more)

Neuroscience and Natural Abstractions

Similarities in structure and function abound in biology; individual neurons that activate exclusively to particular oriented stimuli exist in animals from drosophila (Strother et al. 2017) via pigeons (Li et al. 2007) and turtles (Ammermueller et al. 1995) to macaques (De Valois et al. 1982). The universality of major functional response classes in biology suggests that the neural systems underlying information processing in biology might be highly stereotyped (Van Hooser, 2007, Scholl et al. 2013). In line with this h... (read more)

Thanks for sharing your thoughts Shos! :)

Hmmm good point. I originally made that decision because loading the image from the server was actually kind of slow. But then I figured out asynchronicity, so could totally change it... I'll see if I find some time later today to push an update! (to make an 'all vs all' mode in addition to the 'King of the hill')

Hi Jennifer!

Awesome, thank you for the thoughtful comment! The links are super interesting, reminds me of some of the research in empirical aesthetics I read forever ago.

On the topic of circular preferences: It turns out that the type of reward model I am training here handles non-transitive preferences in a "sensible" fashion. In particular, if you're "non-circular on average" (i.e. you only make accidental "mistakes" in your rating) then the model averages that out. And if you consitently have a loopy utility function, then the reward model will map all ... (read more)

Hi Erik! Thank you for the careful read, this is awesome!

Regarding proposition 1 - I think you're right, that counter-example disproves the proposition. The proposition we were actually going for was ${lim}_{B\to \infty }P\left[\right(s_a,s_1,\dots ,s_B\left)\right]=0.$ ,  i.e. the probability without the end of the bridge! I'll fix this in the post.

Regarding proposition II - janus had the same intuition and I tried to explain it with the following argument: When the distance between tokens becomes large enough, then eventually all bridges between the first token and an arbitrary second... (read more)

Uhhh exciting! Thanks for sharing!

Huh, thanks for spotting that! Yes, should totally be ELK 😀 Fixed it.

Hey Ben! :) Thanks for the comment and the careful reading!

Yes, we only added the missing arx.iv papers after clustering, but then we repeat the dimensionality reduction and show that the original clustering still holds up even with the new papers (Figure 4 bottom right). I think that's pretty neat (especially since the dimensionality reduction doesn't "know" about the clustering) but of course the clusters might look slightly different if we also re-run k-means on the extended dataset.

There's an important caveat here:

The visual stimuli are presented 8 degrees over the visual field for 100ms followed by a 100ms grey mask as in a standard rapid serial visual presentation (RSVP) task.

I'd be willing to bet that if you give the macaque more than 100ms they'll get it right - That's at least how it is for humans!

(Not trying to shift the goalpost, it's a cool result! Just pointing at the next step.)

Great points, thanks for the comment! :) I agree that there are potentially some very low-hanging fruits. I could even imagine that some of these methods work better in artificial networks than in biological networks (less noise, more controlled environment).

But I believe one of the major bottlenecks might be that the weights and activations of an artificial neural network are just so difficult to access? Putting the weights and activations of a large model like GPT-3 under the microscope requires impressive hardware (running forward passes, storing the ac... (read more)

Great point! And thanks for the references :) 

I'll change your background to Computational Cognitive Science in the table! (unless you object or think a different field is even more appropriate)

Thank you for the comment and the questions! :)

This is not clear from how we wrote the paper but we actually do the clustering in the full 768-dimensional space! If you look closely as the clustering plot you can see that the clusters are slightly overlapping - that would be impossible with k-means in 2D, since in that setting membership is determined by distance from the 2D centroid.

Oh true, I completely overlooked that! (if I keep collecting mistakes like this I'll soon have enough for a "My mistakes" page)

Yes, good point! I had that in an earlier draft and then removed it for simplicity and for the other argument you're making!

This sounds right to me! In particular, I just (re-)discovered this old post by Yudkowsky and this newer post by Alex Flint that both go a lot deeper on the topic. I think the optimal control perspective is a nice complement to those posts and if I find the time to look more into this then that work is probably the right direction.

Interesting, I added a note to the text highlighting this! I was not aware of that part of the story at all. That makes it more of a Moloch-example than a "mistaking adversarial for random"-example.

Yes, that's a pretty fair interpretation! The macroscopic/folk psychology notion of "surprise" of course doesn't map super cleanly onto the information-theoretic notion. But I tend to think of it as: there is a certain "expected surprise" about what future possible states might look like if everything evolves "as usual", $I_p\left(\right[x_1,\dots ,x_N\left]\right)$. And then there is the (usually larger) "additional surprise" about the states that the AI might steer us into, $I_{\xi }\left(\right[x_1,\dots ,x_N\left]\right)$. The delta between those two is the "excess surprise" that the AI needs to be able to bri... (read more)

Thank you for your comment! You are right, these things are not clear from this post at all and I did not do a good job at clarifying that. I'm a bit low on time atm, but hopefully, I'll be able to make some edits to the post to set the expectations for the reader more carefully.

The short answer to your question is: Yep, X is the space of events. In Vanessa's post it has to be compact and metric, I'm simplifying this to an interval in R. And $P_{+}/P_{-}$ can be derived from $P_g^H$ by plugging in g=0 and replacing the measure $m\left(A\right)$ by the... (read more)

Cool paper, great to see the project worked out! (:

One question: How do you know the contractors weren't just answering randomly (or were confused about the task) in your "quality after filtering" experiments (Table 4)? Is there agreement across contractors about the quality of completions (in case they saw the same completions)?

Fascinating! Thanks for sharing!

Cool experiment! I could imagine that the tokenizer handicaps GPT's performance here (reversing the characters leads to completely different tokens). With a character-level tokenizer GPT should/might be able to handle that task better!

Interesting, thank you! I guess I was thinking of deception as characterized by Evan Hubinger, with mesa-optimizers, bells, whistles, and all. But I can see how a sufficiently large competence-vs-performance gap could also count as deception.

Thanks for the comment! I'm curious about the Anthropic Codex code-vulnerability prompting, is this written up somewhere? The closest I could find is this, but. I don't think that's what you're referencing?

I was not aware of this, thanks for pointing this out! I made a note in the text. I guess this is not an example of "advanced AI with an unfortunately misspecified goal" but rather just an example of the much larger class of "system with an unfortunately misspecified goal".

Thanks for the comment, I did not know this! I'll put a note in the essay to highlight this comment.

Iiinteresting! Thanks for sharing! Yes, the choice of how to measure this affects the outcome a lot..

Hmm, fair, I think you might get along fine with my coworker from footnote 6 :) I'm not even sure there is a better way to write these titles - but they can still be very intimidating for an outsider.

Yes, I agree, a model can really push intuition to the next level! There is a failure mode where people just throw everything into a model and hope that the result will make sense. In my experience that just produces a mess, and you need some intuition for how to properly set up the model.

Hi! :) Thanks for the comment! Yes, that's on purpose, the idea is that a lot of the shorthand in molecular neuroscience are very hard to digest. So since the exact letters don't matter I intentionally garbled them with a Glitch Text Generator. But perhaps that isn't very clear without explanation, I'll add something.

This word Ǫ̵͎͊G̶̦̉̇l̶͉͇̝̽͆̚i̷͔̓̏͌c̷̱̙̍̂͜k̷̠͍͌l̷̢̍͗̃n̷̖͇̏̆å̴̤c̵̲̼̫͑̎̆ f.e. a garbled version of O-GLicklnac, which in term is the phonetic version of "O-GlcNAc"

Theory #4 appears very natural to me, especially in the light of papers like Chen et al 2006 or Cuntz et al 2012. And another supporting intuition from developmental neuroscience is that development is a huge mess and that figuring out where to put a long-range connection is really involved.  And there can be a bunch of circuit remodeling on a local scale, once you established a long-range connection, there is little hope of substantially rewiring it.

In case you want to dive deeper into this (and you don't want to read all those papers), I'd be happy ... (read more)

I've been meaning to dive into this for-e-ver and only now find the time for it! This is really neat stuff, haven't enjoyed a framework this much since logical induction. Thank you for writing this!

Yep, I agree, SLIDE is probably a dud. Thanks for the references! And my inside view is also that current trends will probably continue and most interesting stuff will happen on AI-specialized hardware.

Thank you for the comment! You are right, that should be a ReLu in the illustration, I'll fix it :)

Great explanation, I feel substantially less confused now. And thank you for adding two new shoulder advisors to my repertorie :D

Thank you for the thoughtful reply!

3. I agree with your point, especially that $u\left(\text{chocolate ice cream and vanilla ice cream}\right)\ne u\left(\text{chocolate ice cream}\right)+u\left(\text{vanilla ice cream}\right)$ should be true.

But I think I can salvage my point by making a further distinction. When I write $u\left(\text{chocolate ice cream}\right)$ I actually mean $u\left(\text{emb}\right(\text{chocolate ice cream}\left)\right)$ where $\text{emb}$ is a semantic embedding that takes sentences to vectors. Already at the level of the embedding we probably have $\text{emb}\left(\text{chocolate ice cream and vanilla ice cream}\right)\ne \text{emb}(\text{chocolate ice}$... (read more)

Awesome, thanks for the feedback Eric! And glad to hear you enjoyed the post!

I'm confused why you're using a neural network

Good point, for the example post it was total overkill. The reason I went with a NN was to demonstrate the link with the usual setting in which preference learning is applied. And in general, NNs generalize better than the table-based approach ( see also my response to Charlie Steiner ).

happy to chat about that

I definitely plan to write a follow-up to this post, will come back to your offer when that follow-up reaches the front of my q... (read more)

Thanks for the comment! (:

1. True, fixed it! I was confused there for a bit.
2. This is also true. I wrote it like this because the proof sketch on Wikipedia included that step. And I guess if step 3 can't be executed (complicated), then it's nice to have the sorted list as a next-best-thing.
3. Those are interesting points and I'm not sure I have a good answer (because the underlying problems are quite deep, I think). My statement about linearity in semantic embeddings is motivated by something like the famous "King – Man + Woman = Queen" from word2vec. Regarding li

Hey Charlie! 

Good comment, gave me a feeling "oh, ups, why didn't I?" for a while. I think having the Elo-like algorithm as a baseline to compare to would have been a good thing to have in any case. But there is something that the NN can do that the Elo-like algorithm can't; generalization. Every "new" element (or even an interpolation of older elements) will get the "initial score" (like 1500 in chess) in Elo, while the NN can exploit similarities between the new element and older elements.

Yes please, would be excited to see that!