All of domenicrosati's Comments + Replies

Thanks for giving this great work - I certainly agree with you on the limits of unlearning as it’s currently conceived for safety. I do wonder if a paradigm of “preventing learning” is a way to get around these limits.

Thanks for reaching out this is all great feedback.

That we will defenitly address. I will dm you for the vaccine implementation as we are currently working on this as well and to see what would be useful for code sharing since we are wee bit aways from having shareable replication of the whole paper.

Some answers

1. Oh woops it should be clearer this is the mutual information measure. If there is something more specific you are looking for here let me know as we do mention it several times (I think!). In case it helps mutual information is always an abstract

Thanks for pointing this out - I think its a critical point.

Im not imagining anything in particular (and ya in that paper we do very much do "baby" sized attacks)

Generally ya this is a problem we need to work out: what is the relationship between a defence strength and the budget an attacker would take to overcome this AND for large groups that have budget for training from scratch would defences here even make an impact.

I think your right in that the large budget groups who can just train from scratch would just not be impacted by defences of this nature.... (read more)

I think people underestimate formal study of research methods like reading texts / taking a course on research methodology for improving research abilities.

There are many concepts within control, experimental design, validitiy, and reliability like construct validity or conclusion validity that you would learn from a research methods textbook that are super helpful for improving the quality of research. I think many researchers implicitly learn these things without ever knowing what they exactly are but that is usually through trial and errors (peer review... (read more)

Thanks for the pointer! Yes RL has a lot of research of this kind - as an empirical research I just get stuck sometimes in translation

For my own clarity: What is the difference between mathematical approaches to alignment and other technical approaches like mechanistic interpretability work?

I imagine the focus is on in principal arguments or proofs regarding the capabilities of a given system rather than empirical or behavioural analysis but you mention RL so just wanted to get some colour on this.

Any clarification here would be helpful!

If someone did this - it would be nice to collect preference data over answers that are helpful to alignment and not helpful to alignment… that could be a dataset that is interesting for a variety of reasons like analyzing current models abilities to help with alignment, gaps in being helpful w.r.t alignment and of course providing a mechanism for making models better at alignment… a model like this could also maybe work as a specialized type of Constitutional AI to collect feedback from the models preferences that are more “alignment-aware” so to speak… n... (read more)

Im struggling to understand how this is is different from “we will build aligned ai to align ai”. specifically: Can someone explain to me how human-like and AGI are different? Can someone explain to me why human-like AI avoids typical x-risk scenarios (given those human-likes could say clone themselves, speed up themselves and rewrite their own software and easily become unbounded)? Why isnt an emulated cognitive system a real cognitive system… i don’t understand how you can emulate a human-like intelligence and it not be the same as fully human-like. ... (read more)

What are your thoughts on prompt tuning as a mechanism for discovering optimal simulation strategies?

I know you mention condition generation as something to touch on in future posts but I’d be eager to hear about where you think prompt tuning comes in considering continuous prompts are differentiable and so can be learned/optimized for specific simulation behaviour.