Research associate at the University of Lancaster, studying AI safety.
My current views on AI:
I repeatedly came across some strange behaviour while working on a project last night where the chatgpt app revealed the actual o1 CoT. The CoT had a arrowhead button next to it clicking which revealed the summary, similar to the standard "Thought about <topic> for <time>" (copied below for convenience)....
Epistemic status: These are first positive results. I have not yet run extensive tests to verify repeatability, so take them with a grain of salt. This post is meant to disseminate early results and collect ideas for further experiments to concretise these findings. Tldr: I study whether LLMs understand their...
I stumbled upon some bizarre behaviour by GPT-4 Turbo 128k, released via API access on November 6, 2023. Since the model is in its preview phase, some imperfections are expected. Nonetheless, I found the behaviour intriguing enough to share here in hopes that someone might conduct a more detailed investigation...
Not an expert, but I think the main problem is that we want our probes to classify eval / not eval, which means we want to train them on data that differs only along that axis. In reality, however, we never really have data that differs along just that axis. Our evals are different in a lot of ways to real world interactions, and therefore, we are not confident that probes trained on these datasets are capturing just that difference.
An n-person meeting requires n updates, each with n people listening, resulting in n^2 minutes of time spent in the meeting.
This seems incorrect. Each of the n people having 1:1 meetings requires (n^2)x minutes spent in the meeting, but an n-person meeting only requires nx minutes since the speakers only speak once to all people. Of course, this is excluding back and forth, which will scale with the number of people, but not quadratically, since many people will have the same questions/comments.
Thanks for the detailed breakdown!
While I agree with most of it, I think Dazzling is different enough from the other types of hidden reasoning that it seems misplaced in this taxonomy,
All the other categories of hidden reasoning are trying to ask the question: "Does the CoT contain sufficient information for the monitor to understand the reasoning process?" Whereas Dazzling is asking different questions such as "Is all of the CoT necessary for understanding the reasoning process?" or "Does the CoT contain token sequences that sabotage/undermine the monitor's effectiveness?".
it’s the kind of terminal value that i expect for most people would be different; guaranteed continuation in 5% of instances is much better than 5% chance of continuing in all instances; in the first case, you don’t die!
And even in the case where we are assigning negative utility to death, most people are really considering counterfactual utility from being alive, and 95% of that (expected) counterfactual utility is lost whether 95% of the "instances of you" die or whether there is a 95% chance that "you" die.
I claim that the negative utility due to stopping to exist is just not there
But we are not talking about negative utility due to stopping to exist. We are talking about avoiding counterfactual negative utility by committing suicide, which still exists!
guaranteed continuation in 5% of instances is much better than 5% chance of continuing in all instances; in the first case, you don’t die!
I think this is an artifact of thinking of all of the copies having a shared utility (i.e. you) rather than separate utilities that add up (i.e. so many yous will suffer if you don't commit suicide). If they have separate utilities, we should think of them as separate instances of yourself.
Whats the difference between fewer instances and fewer copies, and why is that load bearing for the expected utility calculation?
Yes, but the number of copies of you still reduces (or the probability that you are alive in standard probability theory, or the number of branches in many worlds). Why are these not equivalent in terms of the expected utility calculus?
You get a pretty similar dynamic just from having a universe that is large enough to contain many almost-identical copies of you.
Again, not sure why a large universe is needed. The expected utility ends up the same either way, whether you have some fraction of branches in which you remain alive or some probability of remaining alive.
Regarding the expected utility calculus. I agree with everything you said but i don’t see how any of it allows you to disregard the counterfactual suffering from not committing suicide in your expected value calculation.
Maybe the crux is whether we consider the utility of each “you” (i.e. you in each branch) individually, and add it up... (read more)
I don’t think quantum immortality changes anything. You can rephrame this in terms of standard probability theory and condition on them continuing to have subjective experience, and still get to the same calculus.
However, only considering the branches in which you survive, or conditioning on having subjective experience after the suicide attempt, ignores the counterfactual suffering prevented in all the branches (or probability mass) in which you did die, which may be less unpleasant than the branches in which you survived, but are many many more in number! Ignoring those branches biases the reasoning toward rare survival tails that don’t dominate the actual expected utility.
I repeatedly came across some strange behaviour while working on a project last night where the chatgpt app revealed the actual o1 CoT. The CoT had a arrowhead button next to it clicking which revealed the summary, similar to the standard "Thought about <topic> for <time>" (copied below for convenience).
I have attached three screenshots showing the CoT, its generated summary and a line or two of the query and response below.
Searching on twitter, I found one tweet perplexed by the same behaviour.
Epistemic status: These are first positive results. I have not yet run extensive tests to verify repeatability, so take them with a grain of salt. This post is meant to disseminate early results and collect ideas for further experiments to concretise these findings.
I study whether LLMs understand their training data and can use that understanding to make inferences about later training data. Specifically, I measure whether LLMs can infer which declarative facts in their training data are relevant to the current context and then leverage them. I show that finetuning LLMs on declarative data describing different personas reduces the number of iterative finetuning steps (a proxy for reinforcement learning) required to display... (read 3989 more words →)
I stumbled upon some bizarre behaviour by GPT-4 Turbo 128k, released via API access on November 6, 2023. Since the model is in its preview phase, some imperfections are expected. Nonetheless, I found the behaviour intriguing enough to share here in hopes that someone might conduct a more detailed investigation than I have.
These range from:
They define incoherence as the fraction of error explained by variance rather than bias, and then they find that on more complex tasks, a larger proportion of errors are incoherent i.e., caused by variance rather than bias.
But isn't this trivially obvious? On more complex tasks, models (and humans, monkeys, etc.) make more mistakes. So, unless models take more coherently misaligned actions on more complex tasks, so that coherent misalignment (bias) also increases with task complexity, the proportion of error caused by mistakes (variance) will increase.
Mistakes are increasing because of task complexity increasing. There is no reason to expect coherent misalignment to increase with task complexity. Therefore, their measure of incoherence will increase with task complexity.