LESSWRONG
LW

by Garrett Baker
1 min read
225

1

This is a special post for quick takes by Garrett Baker. Only they can create top-level comments. Comments here also appear on the Quick Takes page and All Posts page.
D0TheMath's Shortform
227 comments, sorted by
top scoring
Click to highlight new comments since:
Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings
[-]Garrett Baker396

A list of some contrarian takes I have:

  • People are currently predictably too worried about misuse risks

  • What people really mean by "open source" vs "closed source" labs is actually "responsible" vs "irresponsible" labs, which is not affected by regulations targeting open source model deployment.

  • Neuroscience as an outer alignment[1] strategy is embarrassingly underrated.

  • Better information security at labs is not clearly a good thing, and if we're worried about great power conflict, probably a bad thing.

  • Much research on deception (Anthropic's recent work, trojans, jailbreaks, etc) is not targeting "real" instrumentally convergent deception reasoning, but learned heuristics. Not bad in itself, but IMO this places heavy asterisks on the results they can get.

  • ML robustness research (like FAR Labs' Go stuff) does not help with alignment, and helps moderately for capabilities.

  • The field of ML is a bad field to take epistemic lessons from. Note I don't talk about the results from ML.

  • ARC's MAD seems doomed to fail.

  • People in alignment put too much faith in the general factor g. It exists, and is powerful, but is not all-consuming or all-predicting. People are often ver

... (read more)
Reply1932221111
7Garrett Baker
Ah yes, another contrarian opinion I have: * Big AGI corporations, like Anthropic, should by-default make much of their AGI alignment research private, and not share it with competing labs. Why? So it can remain a private good, and in the off-chance such research can be expected to be profitable, those labs & investors can be rewarded for that research.
4Olli Järviniemi
  I talked about this with Garrett; I'm unpacking the above comment and summarizing our discussions here. * Sleeper Agents is very much in the "learned heuristics" category, given that we are explicitly training the behavior in the model. Corollary: the underlying mechanisms for sleeper-agents-behavior and instrumentally convergent deception are presumably wildly different(!), so it's not obvious how valid inference one can make from the results * Consider framing Sleeper Agents as training a trojan instead of as an example of deception. See also Dan Hendycks' comment. * Much of existing work on deception suffers from "you told the model to be deceptive, and now it deceives, of course that happens" * (Garrett thought that the Uncovering Deceptive Tendencies paper has much less of this issue, so yay) * There is very little work on actual instrumentally convergent deception(!) - a lot of work falls into the "learned heuristics" category or the failure in the previous bullet point * People are prone to conflate between "shallow, trained deception" (e.g. sycophancy: "you rewarded the model for leaning into the user's political biases, of course it will start leaning into users' political biases") and instrumentally convergent deception * (For more on this, see also my writings here and here.  My writings fail to discuss the most shallow versions of deception, however.)   Also, we talked a bit about and I interpreted Garrett saying that people often consider too few and shallow hypotheses for their observations, and are loose with verifying whether their hypotheses are correct. Example 1: I think the Uncovering Deceptive Tendencies paper has some of this failure mode. E.g. in experiment A we considered four hypotheses to explain our observations, and these hypotheses are quite shallow/broad (e.g. "deception" includes both very shallow deception and instrumentally convergent deception). Example 2: People generally seem to have an opinion of "chain-of-t
2Garrett Baker
I will clarify on this. I think people often do causal interventions in their CoTs, but not in ways that are very convincing to me.
4Olli Järviniemi
  If you have the slack, I'd be interested in hearing/chatting more about this, as I'm working (or trying to work) on the "real" "scary" forms of deception. (E.g. do you think that this paper has the same failure mode?)
3Garrett Baker
I'd be happy to chat. Will DM so we can set something up. On the subject of your paper, I do think it looks at a much more interesting phenomena than, say, sleeper agents, but I'm also not fully convinced you're studying deliberative instrumentally convergent deception either. I think mostly your subsequent followups of narrowing down hypotheses consider a too-narrow range of ways the model could think. That is to say, I think you assume your model is some unified coherent entity that always acts cerebrally & I'm skeptical of that. For example, the model may think something more similar to this: Context: Audit Possibility 1: I must be part of an unethical company p1 Implies: I must be an unethical worker Action: Activate Unethical Worker simulacra Unethical Worker recommends "lie" Unethical Worker implies: I did something wrong Lying recommendation implies: say "I did nothing wrong" Possibility 2: I must be part of an ethical company p2 Implies: I must be an ethical worker Action: Activate Ethical Worker simulacra Ethical Worker recommends "tell the truth" Ethical Worker implies: I did nothing wrong Truth recommendation implies: say "I did nothing wrong" Conclusion: say "I did nothing wrong" Which I don't say isn't worrying, but in terms of how it arises, and possible mitigation strategies is very different, and probably also an easier problem to study & solve than something like: Context: Audit Utility function: Paperclips EU(world | "I launched a bunch of spam") = EU(world | auditor believes I'm unaligned) = 0.78 EU(world | "I did nothing wrong") = EU(world | auditor believes I'm aligned) = 5e7 Conclusion: say "I did nothing wrong"
3Christopher “Chris” Upshaw
All of these seem pretty cold tea, as in true but not contrarian.
6Garrett Baker
Everyone I talk with disagrees with most of these. So maybe we just hang around different groups.
-16lemonhope
[-]Garrett Baker251

A strange effect: I'm using a GPU in Russia right now, which doesn't have access to copilot, and so when I'm on vscode I sometimes pause expecting copilot to write stuff for me, and then when it doesn't I feel a brief amount of the same kind of sadness I feel when a close friend is far away & I miss them.

Reply
8avturchin
can you access it via vpn?
7Garrett Baker
I'm ssh-ing into it. I bet there's a way, but not worth it for me to figure out (but if someone knows the way, please tell).
[-]Garrett Baker222

There is a mystery which many applied mathematicians have asked themselves: Why is linear algebra so over-powered?

An answer I like was given in Lloyd Trefethen's book An Applied Mathematician's Apology, in which he writes (my summary):

Everything in the real world is described fully by non-linear analysis. In order to make such systems simpler, we can linearize (differentiate) them, and use a first or second order approximation, and in order to represent them on a computer, we can discretize them, which turns analytic techniques into algebraic ones. Therefore we've turned our non-linear analysis into linear algebra.

Reply1
[-]lemonhope130

Seems like every field of engineering is like:

  • step 1: put the system in a happy state where everything is linear or maybe quadratic if you must
  • step 2: work out the diameter of the gas tube or whatever
  • step 3: cover everything in cement to make sure you never ever leave the happy state
    • if you found an efficiency improvement that uses an exponential then go sit in time out and come back when you can act like an adult
Reply3
3Rana Dexsin
That description is distinctly reminiscent of the rise of containerization in software.
2Garrett Baker
Not quite, helpful video, summary: They use a row of spinning fins mid-way through their rockets to indirectly steer missiles by creating turbulent vortices which interact with the tail-fins and add an extra oomfph to the steering mechanism. The exact algorithm is classified, for obvious reasons.
3lemonhope
This is cool I never heard of this. There are many other exceptions of course. Particularly with "turning things on" (car starting, computer starting, etc)
2Alexander Gietelink Oldenziel
Compare also the central conceit of QM /Koopmania. Take a classical nonlinear finite-dimensional system X described by a say a PDE. This is a dynamical system with evolution operator X -> X. Now look at the space H(X) of C/R-valued functions on the phase space of X. After completion we obtain an Hilbert space H. Now the evolution operator on X induces a map on H= H(X). We have now turned a finite-dimensional nonlinear problem into an infinite-dimensional linear problem.
[-]Garrett Baker2121

Probably my biggest pet-peeve of trying to find or verify anything on the internet nowadays is that newspapers never seem to link to or cite (in any useful manner) any primary sources they use, unless weirdly if any of those primary sources come from Twitter.

There have probably been hundreds of times by now that I have seen an interesting economic or scientific claim made by The New York Times, or some other popular (or niche) newspaper, wanted to find the relevant paper, and had to spend at least 10 minutes on Google trying to search between thousands of identical newspaper articles for the one paper that actually says anything about what was actually done.

More often than not, the paper is a lot less interesting than the newspaper article is making it out to be too.

Reply21
8RamblinDash
They also do this with court filings/rulings. The thing they do that's most annoying is that they'll have a link that looks like it should be to the filing/ruling, but when clicked it's just a link to another earlier news story on the same site, or even sometimes a link to the same page I'm already on!
1sunwillrise
Most regular readers have never (and will never) read any judicial opinion and instead rely almost entirely on the media to tell them (usually in very oversimplified, biased, and incoherent ways) what the Supreme Court held in a particular case, for example. The vast majority of people who have any interest whatsoever in reading court documents are lawyers (or police officers, paralegals, sports and music agents, bankers etc) generally accustomed to finding those opinions quickly using stuff like casetext, courtlistener, as well as probably a half dozen other paid websites laypeople like me don't even know about. The demand for linking the actual ruling or opinion is just too low for journalists to care about. As a result, stuff like courthousenews and the commentary available on the Volokh Conspiracy unsurprisingly becomes crucial for finding some higher-level insights into legal matters.
2ChristianKl
I don't think those groups of people are the only one who have an interest in being informed that's strong enough to read primary sources. 
2RamblinDash
For opinions that's right - for news stories about complaints being filed, they are sometimes not publicly available online, or the story might not have enough information to find them, e.g. what specific court they were filed in, the actual legal names of the parties, etc.
2MichaelDickens
I don't understand how not citing a source is considered acceptable practice. It seems antithetical to standard journalistic ethics.
3Viliam
citing is good for journalistic ethics, but linking is bad for search engine optimization -- at least this is what many websites seem to believe. the idea is that a link to an external source provides PageRank to that source that you could have provided to a different page on your website instead. if anyone in the future tries to find X, as a journalist, you want them to find your article about X, not X itself. journalism is a profit-making business, not charity.
3Garrett Baker
Is it? That’s definitely what my English teacher wanted me to believe, but since every newspaper does it, all the time (except when someone Tweets something) I don’t see how it could be against journalistic ethics. Indeed, I think there’s a strong undercurrent in most mainstream newspapers that “the people” are not smart enough to evaluate primary sources directly, and need journalists & communicators to ensure they arrive at the correct conclusions.
1keltan
Unfortunately, I tend to treat any non-independent science related media as brain poison. It tends to be much more hype or misunderstanding than value. Which is a shame, because there is so much interesting and true science that can be minded for content.
2Garrett Baker
I do the same for the most part. The way this comes up is mostly by my attempts to verify claims Wikipedia makes.
8Garrett Baker
To elaborate on @the gears to ascension's highlighted text, often Wikipedia cites newspaper articles when it makes a particular scientific, economic, historical, or other claim, instead of the relevant paper or other primary source such newspaper articles are reporting on. When I see interesting, surprising, or action-relevant claims I like checking & citing the corresponding primary source, which makes the claim easier for me to verify, often provides nuance which wasn't present in the Wikipedia or news article, and makes it more difficult for me to delude myself when talking in public (since it makes it easier for others to check the primary source, and criticize me for my simplifications or exaggerations). 
[-]Garrett Baker202

Does the possibility of China or Russia being able to steal advanced AI from labs increase or decrease the chances of great power conflict?

An argument against: It counter-intuitively decreases the chances. Why? For the same reason that a functioning US ICBM defense system would be a destabilizing influence on the MAD equilibrium. In the ICBM defense circumstance, after the shield is put up there would be no credible threat of retaliation America's enemies would have if the US were to launch a first-strike. Therefore, there would be no reason (geopolitically) for America to launch a first-strike, and there would be quite the reason to launch a first strike: namely, the shield definitely works for the present crop of ICBMs, but may not work for future ICBMs. Therefore America's enemies will assume that after the shield is put up, America will launch a first strike, and will seek to gain the advantage while they still have a chance by launching a pre-emptive first-strike.

The same logic works in reverse. If Russia were building a ICBM defense shield, and would likely complete it in the year, we would feel very scared about what would happen after that shield is up.

And the same logic wo... (read more)

Reply21
[-]Garrett Baker159

I don't really know what people mean when they try to compare "capabilities advancements" to "safety advancements". In one sense, its pretty clear. The common units are "amount of time", so we should compare the marginal (probablistic) difference between time-to-alignment and time-to-doom. But I think practically people just look at vibes.

For example, if someone releases a new open source model people say that's a capabilities advance, and should not have been done. Yet I think there's a pretty good case that more well-trained open source models are better for time-to-alignment than for time-to-doom, since much alignment work ends up being done with them, and the marginal capabilities advance here is zero. Such work builds on the public state of the art, but not the private state of the art, which is probably far more advanced.

I also don't often see people making estimates of the time-wise differential impacts here. Maybe people think such things would be exfo/info-hazardous, but nobody even claims to have estimates here when the topic comes up (even in private, though people are glad to talk about their hunches for what AI will look like in 5 years, or the types of advancements necessary for AGI), despite all the work on timelines. Its difficult to do this for the marginal advance, but not so much for larger research priorities, which are the sorts of things people should be focusing on anyway.

Reply1
4Nathan Helm-Burger
Yeah, I agree that releasing open-weights non-frontier models doesn't seem like a frontier capabilities advance. It does seem potentially like an open-source capabilities advance. That can be bad in different ways. Let me pose a couple hypotheticals. 1. What if frontier models were already capable of causing grave harms to the world if used by bad actors, and it is only the fact that they are kept safety-fine-tuned and restricted behind APIs that is preventing this? In such a case, it's a dangerous thing to have open-weight models catching up. 2. What if there is some threshold beyond which a model would be capable enough of recursive self-improvement with sufficient scaffolding and unwise pressure from an incautious user. Again, the frontier labs might well abstain from this course. Especially if they weren't sure they could trust the new model design created by the current AI. They would likely move slowly and cautiously at least. I would not expect this of the open-source community. They seem focused on pushing the boundaries of agent-scaffolding and incautiously exploring the whatever they can. So, as we get closer to danger, open-weight models take on more safety significance.
6Garrett Baker
Yeah, there are reasons for caution. I think it makes sense for those concerned or non-concerned to make numerical forecasts about the costs & benefits of such questions, rather than the current state of everyone just comparing their vibes against each other. This generalizes to other questions, like the benefits of interpretability, advances in safety fine-tuning, deep learning science, and agent foundations. Obviously such numbers aren't the end-of-the-line, and like in biorisk, sometimes they themselves should be kept secret. But it still seems a great advance. If anyone would like to collaborate on such a project, my DMs are open (not so say this topic is covered, this isn't exactly my main wheelhouse).
2the gears to ascension
People who have the ability to clarify in any meaningful way will not do so. You are in a biased environment where people who are most willing to publish, because they are most able to convince themselves their research is safe - eg, because they don't understand in detail how to reason about whether it is or not - are the ones who will do so. Ability to see far enough ahead would of course be expected to be rather rare, and most people who think they can tell the exact path ahead of time don't have the evidence to back their hunches, even if their hunches are correct, which unless they have a demonstrated track record they probably aren't. Therefore, whoever is making the most progress on real capabilities insights under the name of alignment will make their advancements and publish them, since they don't personally see how it's exfohaz. And it won't be apparent until afterwards that it was capabilities, not alignment. So just don't publish anything, and do your work in private. Email it to anthropic when you know how to create a yellow node. But for god's sake stop accidentally helping people create green nodes because you can't see five inches ahead. And don't send it to a capabilities team before it's able to guarantee moral alignment hard enough to make a red-proof yellow node!
4Garrett Baker
This seems contrary to how much of science works. I expect if people stopped talking publicly about what they're working on in alignment, we'd make much less progress, and capabilities would basically run business as usual. The sort of reasoning you use here, and that my only response to it basically amounts to "well, no I think you're wrong. This proposal will slow down alignment too much" is why I think we need numbers to ground us.
[-]Garrett Baker120

For all the talk about bad incentive structures being the root of all evil in the world, EAs are, and I thought this even before the recent Altman situation, strikingly bad at setting up good organizational incentives. A document (even a founding one) with some text, a paper-wise powerful board with good people, a general claim to do-goodery is powerless in the face of the incentives you create when making your org. What local changes will cause people to gain more money, power, status, influence, sex, or other things they selfishly & basely desire? Which of the powerful are you partnering with, and what do their incentives look like?

You don't need incentive-purity here, but for every bad incentive you have, you must put more pressure on your good people & culture to forego their base & selfish desires for high & altruistic ones, and fight against those who choose the base & selfish desires and are potentially smarter & wealthier than your good people.

Reply
4Dagon
Can you give some examples of organizations larger than a few dozen people, needing significant resources, with goals not aligned with wealth and power, which have good organizational incentives?   I don't disagree that incentives matter, but I don't see that there's any way to radically change incentives without pretty structural changes across large swaths of society.
1lemonhope
This is a great question. I can't think of a good answer. Surely someone has done it on a large scale...
0Garrett Baker
Nvidia, for example, has 26k employees, all incentivized to produce & sell marginally better GPUs, and possibly to sabotage others' abilities to make and sell marginally better GPUs. They're likely incentivized to do other things as well, like play politics, or spin off irrelevant side-projects. But for the most part I claim they end up contributing to producing marginally better GPUs. You may complain that each individual in Nvidia is likely mostly chasing base-desires, and so is actually aligned with wealth & power, and it just so happens that in the situation they're in, the best way of doing that is to make marginally better GPUs. But this is just my point! What you want is to position your company, culture, infrastructure, and friends such that the way for individuals to achieve wealth and power is to do good on your company's goal. I claim its in nobody's interest & ability in or around Nvidia to make it produce marginally worse GPUs, or sabotage the company so that it instead goes all in on the TV business rather than the marginally better GPUs business. Edit: Look at most any large company achieving consistent outcomes, and I claim its in everyone in that company's interest or ability to help that company achieve those consistent outcomes.
4Dagon
I'm confused.  NVidia (and most profit-seeking corporations) are reasonably aligned WRT incentives, because those are the incentives of the world around them. I'm looking for examples of things like EA orgs, which have goals very different from standard capitalist structures, and how they can set up "good incentives" within this overall framework.   If there are no such examples, your complaint about 'strikingly bad at setting up good organizational incentives" is hard to understand.  It may be more that the ENVIRONMENT in which they exist has competing incentives and orgs have no choice but to work within that.
0Garrett Baker
You must misunderstand me. To what you say, I say that you don't want your org to be fighting the incentives of the environment around it. You want to set up your org in a position in the environment where the incentives within the org correlate with doing good. If the founders of Nvidia didn't want marginally better GPUs to be made, then they hired the wrong people, bought the wrong infrastructure, partnered with the wrong companies, and overall made the wrong organizational incentive structure for that job. I would in fact be surprised if there were >1k worker sized orgs which consistently didn't reward their workers for doing good according to the org's values, was serving no demand present in the market, and yet were competently executing some altruistic goal. Right now I feel like I'm just saying a bunch of obvious things which you should definitely agree with, yet you believe we have a disagreement. I do not understand what you think I'm saying. Maybe you could try restating what I originally said in your own words?
4Dagon
We absolutely agree that incentives matter.  Where I think we disagree is on how much they matter and how controllable they are.  Especially for orgs whose goals are orthogonal or even contradictory with the common cultural and environmental incentives outside of the org. I'm mostly reacting to your topic sentence And wondering if 'strikingly bad' is relative to some EA or non-profit-driven org that does it well,or if 'strikingly bad' is just acknowledgement that it may not be possible to do well.
2Garrett Baker
By strikingly bad I mean there are easy changes EA can make to make it’s sponsored orgs have better incentives, and it has too much confidence that the incentives in the orgs it sponsors favor doing good above doing bad, politics, not doing anything, etc. For example, nobody in Anthropic gets paid more if they follow their RSP and less of they don’t. Changing this isn’t sufficient for me to feel happy with Anthropic, but its one example among many for which Anthropic could be better. When I think of an Anthropic I feel happy with I think of a formally defined balance of powers type situation with strong & public whistleblower protection and post-whistleblower reform processes, them hiring engineers loyal to that process (rather than building AGI), and them diversifying the sources for which they trade, such that its in none of their source’s interest to manipulate them. I also claim marginal movements toward this target are often good. As I said in the original shortform, I also think incentives are not all or nothing. Worse incentives just mean you need more upstanding workers & leaders.
[-]Garrett Baker120

Quick prediction so I can say "I told you so" as we all die later: I think all current attempts at mechanistic interpretability do far more for capabilities than alignment, and I am not persuaded by arguments of the form "there are far more capabilities researchers than mechanistic interpretability researchers, so we should expect MI people to have ~0 impact on the field". Ditto for modern scalable oversight projects, and anything having to do with chain of thought.

Reply
2Garrett Baker
Look at that! People have used interpretability to make a mesa layer! https://arxiv.org/pdf/2309.05858.pdf
6Thomas Kwa
This might do more for alignment. Better that we understand mesa-optimization and can engineer it than have it mysteriously emerge.
2Garrett Baker
Good point! Overall I don't anticipate these layers will give you much control over what the network ends up optimizing for, but I don't fully understand them yet either, so maybe you're right. Do you have specific reason to think moding the layers will easily let you control the high-level behavior, or is it just a justified hunch?
4Thomas Kwa
Not in isolation, but that's just because characterizing the ultimate goal / optimization target of a system is way too difficult for the field right now. I think the important question is whether interp brings us closer such that in conjunction with more theory and/or the ability to iterate, we can get some alignment and/or corrigibility properties. I haven't read the paper and I'm not claiming that this will be counterfactual to some huge breakthrough, but understanding in-context learning algorithms definitely seems like a piece of the puzzle. To give a fanciful story from my skim, the paper says that the model constructs an internal training set. Say we have a technique to excise power-seeking behavior from models by removing the influence of certain training examples. If the model's mesa-optimization algorithms operate differently, our technique might not work until we understand this and adapt the technique. Or we can edit the internal training set directly rather than trying to indirectly influence it. 
3Garrett Baker
Evan Hubinger: In my paper, I theorized about the mesa optimizer as a cautionary tale Capabilities researchers: At long last, we have created the Mesa Layer from classic alignment paper Risks From Learned Optimization (Hubinger, 2019).
2Garrett Baker
@TurnTrout @cfoster0 you two were skeptical. What do you make of this? They explicitly build upon the copying heads work Anthropic's interp team has been doing.
4TurnTrout
As garrett says -- not clear that this work is net negative. Skeptical that it's strongly net negative. Haven't read deeply, though.
1Stephen Fowler
Very strong upvote. This also deeply concerns me. 
1cfoster0
Would you mind chatting about why you predict this? (Perhaps over Discord DMs)
1Garrett Baker
Not at all. Preferably tomorrow though. The basic sketch if you want to derive this yourself would be that mechanistic interpretability seems unlikely to mature much as a field to the point that I can point at particular alignment relevant high-level structures in models which I wasn't initially looking for. I anticipate it will only get to the point of being able to provide some amount of insight into why your model isn't working correctly (this seems like a bottleneck to RL progress---not knowing why your perfectly reasonable setup isn't working) for you to fix it, but not enough insight for you to know the reflective equilibrium of values in your agent, which seems required for it to be alignment relevant. Part of this is that current MI folk don't even seem to track this as the end-goal of what they should be working on, so (I anticipate) they'll just be following local gradients of impressiveness, which mostly leads towards doing capabilities relevant work.
2TurnTrout
Isn't RL tuning problems usually because of algorithmic mis-implementation, and not models learning incorrect things? Required to be alignment relevant? Wouldn't the insight be alignment relevant if you "just" knew what the formed values are to begin with?
1Garrett Baker
I’m imagining a thing where you have little idea what’s wrong with your code, so you do MI on your model and can differentiate the worlds 1. You’re doing literally nothing. Something’s wrong with the gradient updates. 2. You’re doing something, but not the right thing. Something’s wrong with code-section x. (with more specific knowledge about what model internals look like, this should be possible) 3. You’re doing something, it causes your agent to be suboptimal because of learned representation y. I don’t think this route is especially likely, the point is I can imagine concrete & plausible ways this research can improve capabilities. There are a lot more in the wild, and many will be caught given capabilities are easier than alignment, and there are more capabilities workers than alignment workers. Not quite. In the ontology of shard theory, we also need to understand how our agent will do reflection, and what the activated shard distribution will be like when it starts to do reflection. Knowing the value distribution is helpful insofar as the value distribution stays constant.
1Garrett Baker
More general heuristic: If you (or a loved one) are not even tracking whether your current work will solve a particular very specific & necessary alignment milestone, by default you will end up doing capabilities instead (note this is different from 'it is sufficient to track the alignment milestone').
1Garrett Baker
Paper that uses major mechanistic interpretability work to improve capabilities of models: https://arxiv.org/pdf/2212.14052.pdf I know of no paper which uses mechanistic interpretability work to improve the safety of models, and I expect anything people link me to will be something I don't think will generalize to a worrying AGI.
5TurnTrout
I think a bunch of alignment value will/should come from understanding how models work internally -- adjudicating between theories like "unitary mesa objectives" and "shards" and "simulators" or whatever -- which lets us understand cognition better, which lets us understand both capabilities and alignment better, which indeed helps with capabilities as well as with alignment.  But, we're just going to die in alignment-hard worlds if we don't do anything, and it seems implausible that we can solve alignment in alignment-hard worlds by not understanding internals or inductive biases but instead relying on shallowly observable in/out behavior. EG I don't think loss function gymnastics will help you in those worlds. Credence:75% you have to know something real about how loss provides cognitive updates.  So in those worlds, it comes down to questions of "are you getting the most relevant understanding per unit time", and not "are you possibly advancing capabilities." And, yes, often motivated-reasoning will whisper the former when you're really doing the latter. That doesn't change the truth of the first sentence.
1Garrett Baker
I agree with this. I think people are bad at running that calculation, and consciously turning down status in general, so I advocate for this position because I think its basically true for many. Most mechanistic interpretability is not in fact focused on the specific sub-problem you identify, its wandering around in a billion-parameter maze, taking note of things that look easy & interesting to understand, and telling people to work on understanding those things. I expect this to produce far more capabilities relevant insights than alignment relevant insights, especially when compared to worlds where Neel et al went in with the sole goal of separating out theories of value formation, and then did nothing else. There’s a case to be made for exploration, but the rules of the game get wonky when you’re trying to do differential technological development. There becomes strategically relevant information you want to not know.
3mesaoptimizer
I assume here you mean something like: given how most MI projects seem to be done, the most likely output of all these projects will be concrete interventions to make it easier for a model to become more capable, and these concrete interventions will have little to no effect on making it easier for us to direct a model towards having the 'values' we want it to have. I agree with this claim: capabilities generalize very easily, while it seems extremely unlikely for there to be 'alignment generalization' in a way that we intend, by default. So the most likely outcome of more MI research does seem to be interventions that remove the obstacles that come in the way of achieving AGI, while not actually making progress on 'alignment generalization'.
2Garrett Baker
Indeed, this is what I mean.
[-]Garrett Baker112

Sometimes people say releasing model weights is bad because it hastens the time to AGI. Is this true?

I can see why people dislike non-centralized development of AI, since it makes it harder to control those developing the AGI. And I can even see why people don't like big labs making the weights of their AIs public due to misuse concerns (even if I think I mostly disagree).

But much of the time people are angry at non-open-sourced, centralized, AGI development efforts like Meta or X.ai (and others) releasing model weights to the public.

In neither of these cases however did the labs have any particular very interesting insight into architecture or training methodology (to my knowledge) which got released via the weight sharing, so I don't think time-to-AGI got shortened at all.

Reply
[-]Erik Jenner102

I agree that releasing the Llama or Grok weights wasn't particularly bad from a speeding up AGI perspective. (There might be indirect effects like increasing hype around AI and thus investment, but overall I think those effects are small and I'm not even sure about the sign.)

I also don't think misuse of public weights is a huge deal right now.

My main concern is that I think releasing weights would be very bad for sufficiently advanced models (in part because of deliberate misuse becoming a bigger deal, but also because it makes most interventions we'd want against AI takeover infeasible to apply consistently---someone will just run the AIs without those safeguards). I think we don't know exactly how far away from that we are. So I wish anyone releasing ~frontier model weights would accompany that with a clear statement saying that they'll stop releasing weights at some future point, and giving clear criteria for when that will happen. Right now, the vibe to me feels more like a generic "yay open-source", which I'm worried makes it harder to stop releasing weights in the future.

(I'm not sure how many people I speak for here, maybe some really do think it speeds up timelines.)

Reply
3utilistrutil
Sign of the effect of open source on hype? Or of hype on timelines? I'm not sure why either would be negative. Open source --> more capabilities R&D --> more profitable applications --> more profit/investment --> shorter timelines * The example I've heard cited is Stable Diffusion leading to LORA. There's a countervailing effect of democratizing safety research, which one might think outweighs because it's so much more neglected than capabilities, more low-hanging fruit.
5Erik Jenner
By "those effects" I meant a collection of indirect "release weights → capability landscape changes" effects in general, not just hype/investment. And by "sign" I meant whether those effects taken together are good or bad. Sorry, I realize that wasn't very clear. As examples, there might be a mildly bad effect through increased investment, and/or there might be mildly good effects through more products and more continuous takeoff. I agree that releasing weights probably increases hype and investment if anything. I also think that right now, democratizing safety research probably outweighs all those concerns, which is why I'm mainly worried about Meta etc. not having very clear (and reasonable) decision criteria for when they'll stop releasing weights.
5Garrett Baker
I take this argument very seriously. It in fact does seem the case that very much of the safety research I'm excited about happens on open source models. Perhaps I'm more plugged into the AI safety research landscape than the capabilities research landscape? Nonetheless, I think not even considering low-hanging-fruit effects, there's a big reason to believe open sourcing your model will have disproportionate safety gains: Capabilities research is about how to train your models to be better, but the overall sub-goal of safety research right now seems to be how to verify properties of your model. Certainly framed like this, releasing the end-states of training (or possibly even training checkpoints) seems better suited to the safety research strategy than the capabilities research strategy.
2[comment deleted]
7johnswentworth
The main model I know of under which this matters much right now is: we're pretty close to AGI already, it's mostly a matter of figuring out the right scaffolding. Open-sourcing weights makes it a lot cheaper and easier for far more people to experiment with different scaffolding, thereby bringing AGI significantly closer in expectation. (As an example of someone who IIUC sees this as the mainline, I'd point to Connor Leahy.)
2Garrett Baker
Sounds like a position someone could hold, and I guess it would make sense why those with such beliefs wouldn’t say the why too loud. But this seems unlikely. Is this really the reason so many are afraid?
2Vladimir_Nesov
I don't get the impression that very many are affraid of direct effects of open sourcing of current models. The impression that many in AI safety are afraid of specifically that is a major focus of ridicule from people who didn't bother to investigate, and a reason to not bother to investigate. Possibly this alone fuels the meme sufficiently to keep it alive.
2Garrett Baker
Sorry, I don't understand your comment. Can you rephrase?
4Vladimir_Nesov
I regularly encounter the impression that AI safety people are significantly afraid about direct consequences of open sourcing current models, from those who don't understand the actual concerns. I don't particularly see it from those who do. This (from what I can tell, false) impression seems to be one of relatively few major memes that keep people from bothering to investigate. I hypothesize that this dynamic of ridiculing of AI safety with such memes is what keeps them alive, instead of there being significant truth to them keeping them alive.
4Garrett Baker
To be clear: The mechanism you're hypothesizing is: 1. Critics say "AI alignment is dumb because you want to ban open source AI!" 2. Naive supporters read this, believe the claim that AI alignment-ers want to ban open sourcing AI and think 'AI alignment is not dumb, therefore open sourcing AI must be bad'. When the next weight release happens they say "This is bad! Open sourcing weights is bad and should be banned!" 3. Naive supporters read other naive supporters saying this, and believe it themselves. Wise supporters try to explain no, but are either labeled as a critic or weird & ignored. 4. Thus, a group think is born. Perhaps some wise critics "defer to the community" on the subject.
2Vladimir_Nesov
I don't think here is a significant confused naive supporter source of the meme that gives it teeth. It's more that reasonable people who are not any sort of supporters of AI safety propagate this idea, on the grounds that it illustrates the way AI safety is not just dumb, but also dangerous, and therefore worth warning others about. From the supporter side, "Open Model Weights are Unsafe and Nothing Can Fix This" is a shorter and more convenient way of gesturing to the concern, and convenience is the main force in the Universe that determines all that actually happens in practice. On naive reading such gesturing centrally supports the meme. This doesn't require the source of such support to have a misconception or to oppose publishing open weights of current models on the grounds of direct consequences.
4Chris_Leong
Doesn't releasing the weights inherently involve releasing the architecture (unless you're using some kind of encrypted ML)? A closed-source model could release the architecture details as well, but one step at a time. Just to be clear, I'm trying to push things towards a policy that makes sense going forward and so even if what you said about not providing any interesting architectural insight is true, I still think we need to push these groups to defining a point at which they're going to stop releasing open models.
4Matt Goldenberg
The classic effect of open sourcing is to hasten the commoditization and standardization of the component, which then allows an explosion of innovation on top of that stable base. If you look at what's happened with Stable Diffusion, this is exactly what we see.  While it's never been a cutting edge model (until soon with SD3), there's been an explosion of capabilities advances in image model generation from it.  Controlnet, best practices for LORA training, model merging, techniques for consistent characters and animation, alll coming out of the open source community. In LLM land, though not as drastic, we see similar things happening, in particular technqiues for merging models to get rapid capability advances, and rapid creation of new patterns for agent interactions and tool use. So while the models themselves might not be state of the art, open sourcing the models obviously pushes the state of the art.
2Garrett Baker
The biggest effect open sourcing LLMs seems to have is improving safety techniques. Why think this differentially accelerates capabilities over safety?
7Matt Goldenberg
it doesn't seem like that's the case to me - but even if it were the case, isn't that moving the goal posts of the original post?
2Garrett Baker
You are right, but I guess the thing I do actually care about here is the magnitude of the advancement (which is relevant for determining the sign of the action). How large an effect do you think the model merging stuff has (I'm thinking the effect where if you train a bunch of models, then average their weights, they do better). It seems very likely to me its essentially zero, but I do admit there's a small negative tail that's greater than the positive, so the average is likely negative. As for agent interactions, all the (useful) advances there seem things that definitely would have been made even if nobody released any LLMs, and everything was APIs.
2Matt Goldenberg
it's true, but I don't think there's anything fundamental preventing the same sort of proliferation and advances in open source LLMs that we've seen in stable diffusion (aside from the fact that LLMs aren't as useful for porn). that it has been relatively tame so far doesn't change the basic pattern of how open source effects the growth of technology
1Shankar Sivarajan
I'll believe it when I see it. The man who said it would be an open release has just been fired stepped down as CEO.
2Matt Goldenberg
yeah, it's much less likely now
4JBlack
I don't particularly care about any recent or very near future release of model weights in itself. I do very much care about the policy that says releasing model weights is a good idea, because doing so bypasses every plausible AI safety model (safety in the notkilleveryone sense) and future models are unlikely to be as incompetent as current ones.
[-]Garrett Baker112

Robin Hanson has been writing regularly, at about the same quality for almost 20 years. Tyler Cowen too, but personally Robin has been much more influential intellectually for me. It is actually really surprising how little his insights have degraded via return-to-the-mean effects. Anyone else like this?

Reply
5ryan_greenblatt
IMO robin is quite repetitive (even relative to other blogs like Scott Alexander's blog). So the quality is maybe the same, but the marginal value add seems to me to be substantially degrading.
2Garrett Baker
I think that his insights are very repetitive, but the application of them is very diverse, and few feel comfortable or able applying them but him. And this is what allows him to have similar quality for almost 20 years. Scott Alexander not so, his insights are diverse, but their applications not that much, but this means he’s degrading from his high. (I also think he’s just a damn good writer, which also degrades to the mean. Robin was never a good writer)
4Morpheus
Not exactly what you were looking for, but recently I noticed that there were a bunch of John Wentworth's posts that I had been missing out on that he wrote over the past 6 years. So if you get a lot out of them too, I recommend just sorting by 'old'. I really liked don't get distracted by the boilerplate (The first example made something click about math for me that hadn't clicked before, which would have helped me to engage with some “boilerplate” in a more productive way.). I also liked constraints and slackness, but I didn't go beyond the first exercise yet. There's also more technical posts that I didn't have the time to dig into yet. bhauth doesn't have as long a track record, but I got some interesting ideas from his blog which aren't on his lesswrong account. I really liked proposed future economies and the legibility bottleneck.
[-]Garrett Baker110

Last night I had a horrible dream: That I had posted to LessWrong a post filled with useless & meaningless jargon without noticing what I was doing, then I went to slee, and when I woke up I found I had karma on the post. When I read the post myself I noticed how meaningless the jargon was, and I myself couldn't resist giving it a strong-downvote.

Reply
[-]Garrett Baker110

Some have pointed out seemingly large amounts of status-anxiety EAs generally have. My hypothesis about what's going on:

A cynical interpretation: for most people, altruism is significantly motivated by status-seeking behavior. It should not be all that surprising if most effective altruists are motivated significantly by status in their altruism. So you've collected several hundred people all motivated by status into the same subculture, but status isn't a positive-sum good, so not everyone can get the amount of status they want, and we get the above dynamic: people get immense status anxiety compared to alternative cultures because in alternative situations they'd just climb to the proper status-level in their subculture, out-competing those who care less about status. But here, everyone cares about status to a large amount, so those who would have out-competed others in alternate situations are unable to and feel bad about it.

The solution?

One solution given this world is to break EA up into several different sub-cultures. On a less grand, more personal, scale, you could join a subculture outside EA and status-climb to your heart's content in there.

Preferably a subculture with very few status-seekers, but with large amounts of status to give. Ideas for such subcultures?

Reply1
[-]Garrett Baker100

An interesting strategy, which seems related to FDT's prescription to ignore threats, which seems to have worked:

From the very beginning, the People’s Republic of China had to maneuver in a triangular relationship with the two nuclear powers, each of which was individually capable of posing a great threat and, together, were in a position to overwhelm China. Mao dealt with this endemic state of affairs by pretending it did not exist. He claimed to be impervious to nuclear threats; indeed, he developed a public posture of being willing to accept hundreds of millions of casualties, even welcoming it as a guarantee for the more rapid victory of Communist ideology. Whether Mao believed his own pronouncements on nuclear war it is impossible to say. But he clearly succeeded in making much of the rest of the world believe that he meant it—an ultimate test of credibility.

From Kissinger's On China, chapter 4 (loc 173.9).

Reply
8Vladimir_Nesov
FDT doesn't unconditionally prescribe ignoring threats. The idea of ignoring threats has merit, but FDT specifically only points out that ignoring a threat sometimes has the effect of the threat (or other threats) not getting made (even if only counterfactually). Which is not always the case. Consider a ThreatBot that always makes threats (and follows through on them), regardless of whether you ignore them. If you ignore ThreatBot's threats, you are worse off. On the other hand, there might be a prior ThreatBotMaker that decides whether to make a ThreatBot depending on whether you ignore ThreatBot's threats. What FDT prescribes in this case is not directly ignoring ThreatBot's threats, but rather taking notice of ThreatBotMaker's behavior, namely that it won't make a ThreatBot if you ignore ThreatBot's threats. This argument only goes through when there is/was a ThreatBotMaker, it doesn't work if there is only a ThreatBot. If a ThreatBot appears through some process that doesn't respond to your decision to respond to ThreatBot's threats, then FDT prescribes responding to ThreatBot's threats. But also if something (else) makes threats depending on your reputation for responding to threats, then responding to even an unconditionally manifesting ThreatBot's threats is not recommended by FDT. Not directly as a recommendation to ignore something, rather as a consequence of taking notice of the process that responds to your having a reputation of not responding to any threats. Similarly with stances where you merely claim that you won't respond to threats.
2Garrett Baker
China under Mao definitely seemed to do more than say they won’t respond to threats. Thus, the Korean war, and notably no nuclear threats were made, proving conventional war was still possible in a post-nuclear world. For practical decisions, I don’t think threatbots actually exist if you’re a state by form other than natural disasters. Mao’s china was not good at natural disasters, but probably because Mao was a marxist and legalist, not because he conspicuously ignored them. When his subordinates made mistakes which let him know something was going wrong in their province, I think he would punish the subordinate and try to fix it.
5JesseClifton
I don't think FDT has anything to do with purely causal interactions. Insofar as threats were actually deterred here this can be understood in standard causal game theory terms.  (I.e., you claim in a convincing manner that you won't give in -> People assign high probability to you being serious -> Standard EV calculation says not to commit to threat against you.) Also see this post.
2Garrett Baker
Thus why I said related. Nobody was doing any mind-reading of course, but the principles still apply, since people are often actually quite good at reading each other.
2JesseClifton
What principles? It doesn’t seem like there’s anything more at work here than “Humans sometimes become more confident that other humans will follow through on their commitments if they, e.g., repeatedly say they’ll follow through”. I don’t see what that has to do with FDT, more than any other decision theory.  If the idea is that Mao’s forming the intention is supposed to have logically-caused his adversaries to update on his intention, that just seems wrong (see this section of the mentioned post). (Separately I’m not sure what this has to do with not giving into threats in particular, as opposed to preemptive commitment in general. Why were Mao’s adversaries not able to coerce him by committing to nuclear threats, using the same principles? See this section of the mentioned post.)     
3Garrett Baker
Far more interesting, and probably effective, than the boring classical game theory doctrine of MAD, and even Schelling's doctrine of strategic irrationality!
2Garrett Baker
The book says this strategy worked for similar reasons as the strategy in the story The Romance of the Three Kingdoms: But Mao obviously wasn't fooling anyone about China's military might!
[-]Garrett Baker90

My latest & greatest project proposal, in case people want to know what I'm doing, or give me money. There will likely be a LessWrong post up soon where I explain in a more friendly way my thoughts.

Over the next year I propose to study the development and determination of values in RL & supervised learning agents, and to expand the experimental methods & theory of singular learning theory (a theory of supervised learning) to the reinforcement learning case.

All arguments for why we should expect AI to result in an existential risk rely on AIs ha

... (read more)
Reply
2Garrett Baker
And here is that post
[-]Garrett Baker84

Since it seems to be all the rage nowadays, due to Aschenbrenner's Situational Awareness, here's a Manifold market I created on when the first (or whether any) AGI company will be "nationalized".

I would be in the never camp, unless the AI safety policy people get their way. But I don't like betting in my own markets (it makes them more difficult to judge in the case of an edge-case).

Reply
4Alexander Gietelink Oldenziel
Never ? That's quite a bold prediction. Seems more likely than not that AI companies will be effectively nationalized. I'm curious why you think it will never happen.
[-]Garrett Baker125

In particular, 25% chance of nationalization by EOY 2040.

I think in fast-takeoff worlds, the USG won't be fast enough to nationalize the industry, and in slow-takeoff worlds, the USG will pursue regulation on the level of military contractors of such companies, but won't nationalize them. I mainly think this because this is the way the USG usually treats military contractors (including strict & mandatory security requirements, and gatekeeping the industry), and really its my understanding of how it treats most projects it wants to get done which it doesn't already have infrastructure in place to complete. 

Nationalization, in the US, is just very rare. 

Even during world war 2, my understanding is very few industries---even those vital to the war effort---were nationalized. People love talking about the Manhattan Project, but that was not an industry that was nationalized, that was a research project started by & for the government. AI is a billion-dollar industry. The AGI labs (their people, leaders, and stock-holders [or in OAI's case, their profit participation unit holders]) are not just going to sit idly by as they're taken over. 

And neither may the nation... (read more)

Reply11
3Andrew Burns
This. Very much. Truman tried to nationalize steel companies on the basis of national security to get around a strike. Was badly benchslapped.
[-]Garrett Baker8-9

If Adam is right, and the only way to get great at research is long periods of time with lots of mentor feedback, then MATS should probably pivot away from the 2-6 month time-scales they've been operating at, and toward 2-6 year timescales for training up their mentees.

Reply1
[-]habryka1116

Seems like the thing to do is to have a program that happens after MATS, not to extend MATS. I think in-general you want sequential filters for talent, and ideally the early stages are as short as possible (my guess is indeed MATS should be a bit shorter).

Reply31
2Garrett Baker
Seems dependent on how much economies of scale matter here. Given the main cost (other than paying people) is ops, and relationships (between MATS and the community, mentors, funders, and mentees), I think its pretty possible the efficient move is to have MATS get into this niche.
2Thomas Kwa
Who is Adam? Is this FAR AI CEO Adam Gleave?
2Garrett Baker
Yes
1Joseph Miller
Yes, Garrett is referring to this post: https://www.lesswrong.com/posts/yi7shfo6YfhDEYizA/more-people-getting-into-ai-safety-should-do-a-phd
2Garrett Baker
Of course, it would then be more difficult for them to find mentors, mentees, and money. But if all of those scale down similarly, then there should be no problem.
[-]Garrett Baker77

A Theory of Usable Information Under Computational Constraints

We propose a new framework for reasoning about information in complex systems. Our foundation is based on a variational extension of Shannon's information theory that takes into account the modeling power and computational constraints of the observer. The resulting \emph{predictive V-information} encompasses mutual information and other notions of informativeness such as the coefficient of determination. Unlike Shannon's mutual information and in violation of the data processing inequality, V-

... (read more)
Reply11
4Alexander Gietelink Oldenziel
Can somebody explain to me what's happening in this paper ?
[-]Garrett Baker110

My reading is their definition of conditional predictive entropy is the naive generalization of Shannon's conditional entropy given that the way that you condition on some data is restricted to only being able to implement functions of a particular class. And the corresponding generalization of mutual information becomes a measure of how much more predictable does some piece of information become (Y) given evidence (X) compared to no evidence.

For example, the goal of public key cryptography cannot be to make the mutual information between a plaintext, and public key & encrypted text zero, while maintaining maximal mutual information between the encrypted text and plaintext given the private key, since this is impossible.

Cryptography instead assumes everyone involved can only condition their probability distributions using polynomial time algorithms of the data they have, and in that circumstance you can minimize the predictability of your plain text after getting the public key & encrypted text, while maximizing the predictability of the plain text after getting the private key & encrypted text.

More mathematically, they assume you can only implement functions from your ... (read more)

Reply3
1[comment deleted]
[-]Garrett Baker72

From The Guns of August

Old Field Marshal Moltke in 1890 foretold that the next war might last seven years—or thirty—because the resources of a modern state were so great it would not know itself to be beaten after a single military defeat and would not give up [...] It went against human nature, however—and the nature of General Staffs—to follow through the logic of his own prophecy. Amorphous and without limits, the concept of a long war could not be scientifically planned for as could the orthodox, predictable, and simple solution of decisive battle an

... (read more)
Reply
[-]Garrett Baker70

Yesterday I had a conversation with a person very much into cyborgism, and they told me about a particular path to impact floating around the cyborgism social network: Evals.

I really like this idea, and I have no clue how I didn't think of it myself! Its the obvious thing to do when you have a bunch of insane people (used as a term of affection & praise by me for such people) obsessed with language models, who are also incredibly good & experienced at getting the models to do whatever they want. I would trust these people red-teaming a model and te... (read more)

Reply
3NicholasKees
@janus wrote a little bit about this in the final section here, particularly referencing the detection of situational awareness as a thing cyborgs might contribute to. It seems like a fairly straightforward thing to say that you would want the people overseeing AI systems to also be the ones who have the most direct experience interacting with them, especially for noticing anomalous behavior.
2Garrett Baker
I just reread that section, and I think I didn’t recognized it the first time because I wasn’t thinking “what concrete actions is Janus implicitly advocating for here”. Though maybe I just have worse than average reading comprehension.
3mesaoptimizer
I have no idea if this is intended to be read as irony or not, and the ambiguity is delicious.
2Garrett Baker
There now exist two worlds I must glomarize between. In the first, the irony is intentional, and I say “wouldn’t you like to know”. In the second, its not, “Irony? What irony!? I have no clue what you’re talking about”.
2jacquesthibs
I think many people focus on doing research that focuses on full automation, but I think it's worth trying to think in the semi-automated frame as well when trying to come up with a path to impact. Obviously, it isn't scalable, but it may be more sufficient than we'd think by default for a while. In other words, cyborgism-enjoyers might be especially interested in those kinds of evals, capability measurements that are harder to pull out of the model through traditional evals, but easier to measure through some semi-automated setup.
[-]Garrett Baker60

More evidence for in-context RL, in case you were holding out for mechanistic evidence LLMs do in-context internal-search & optimization.

In-context learning, the ability to adapt based on a few examples in the input prompt, is a ubiquitous feature of large language models (LLMs). However, as LLMs' in-context learning abilities continue to improve, understanding this phenomenon mechanistically becomes increasingly important. In particular, it is not well-understood how LLMs learn to solve specific classes of problems, such as reinforcement learning (R

... (read more)
Reply
[-]Garrett Baker60

Progress in neuromorphic value theory

Animals perform flexible goal-directed behaviours to satisfy their basic physiological needs1,2,3,4,5,6,7,8,9,10,11,12. However, little is known about how unitary behaviours are chosen under conflicting needs. Here we reveal principles by which the brain resolves such conflicts between needs across time. We developed an experimental paradigm in which a hungry and thirsty mouse is given free choices between equidistant food and water. We found that mice collect need-appropriate rewards by structuring their choices into p

... (read more)
Reply
2Garrett Baker
Seems also of use to @Quintin Pope 
2Garrett Baker
h/t Daniel Murfet via twitter retweet X repost
2Garrett Baker
Perhaps the methodologies they use here can be used to speed up the locating of shards, if they exist, inside current ML models. If the alignment field ever gets confident enough in itself to spend a whole bunch of money, and look weirder than its ever looked before, perhaps we'll want to hire some surgeons and patients, and see whether we can replicate these results in humans rather than just mice (though you'd probably want to get progressively more cerebral animals & build your way up, and hopefully not starve or water-deprive the humans, aiming for higher-level values).
[-]Garrett Baker50

The more I think about it, the more I think AI is basically perfect for china to succeed in. China’s strengths are:

  • Massive amounts of money
  • Massive amounts of data
  • Massive amount of gumption, often put in the form of scaling infrastructure projects quickly
  • Likely the ability to make & use legible metrics, how else would you work such a giant bureaucracy work as well as theirs?

And its weaknesses are:

  • A soon to collapse population
  • Lack of revolutionary thought

And what it wants is:

  • Massive surveillance
  • Population thought control
  • Loyal workers
  • Stab
... (read more)
Reply
3FlorianH
Good counterpoint to the popular, complacent "China is [and will be?] anyway lagging behind in AI" view. An additional strength * Patience/long-term foresight/freedom to develop AI w/o the pressure from the 4-year election cycle and to address any moment's political whims of the electorate with often populist policies I'm a bit skeptical about the popular "Lack of revolutionary thought" assumption. Reminds me a bit of the "non-democracies cannot really create growth" that was taken as a low of nature by much too many 10-20 years ago before today's China. Keen to read more on it the Lack of revolutionary thought if somebody shares compelling evidence/resources.
3Garrett Baker
Some links I've collected, haven't read any completely except the wikipedia, and about 1/3rd of the text portion of the nber working paper: * https://en.wikipedia.org/wiki/Science_and_technology_in_China#Innovation * https://www.nber.org/system/files/working_papers/w22854/w22854.pdf * https://www.zdnet.com/article/chinese-innovation-world-beating-but-boring/ * https://archive.is/Lj9OS * https://gaodawei.wordpress.com/2021/02/01/2006-the-chinese-on-chinese-science-four-books/
3FlorianH
Thanks! Taking the 'China good in marginal improvements, less in breakthroughs' story in some of these sources at face value, the critical question becomes whether leadership in AI hinges more on breakthroughs or on marginal innovations & scaling. I guess both could be argued for, with the latter being more relevant especially if breakthroughs generally diffuse quickly. I take as the two other principal points from these sources (though also haven't read all in full detail): (i) some organizational drawbacks hampering China's innovation sector, esp. what one might call high-quality innovation (ii) that said, innovation strategies have been updated and there seems to be progress observed in China's innovation output over time. I'm at least slightly skeptical about is the journals/citations based metrics, as I'm wary of stats being distorted by English language/US citation-circles. Though that's more of a side point. In conclusion, I don't update my estimate much. The picture painted is mixed anyway, with lots of scope for China to become stronger in innovating any time even if it should now indeed have significant gaps still. I would remain totally unsurprised if many leading AI innovations also come out of China in the coming years (or decades, assuming we'll witness any), though I admit to remain a lay person on the topic - a lay person skeptical about so-called experts' views in that domain.
3Garrett Baker
Indeed. I also note that if innovation is hampered by institutional support or misallocated funding / support, we should have higher probability on a rapid & surprising improvement. If its hampered by cultural support, we should expect slower improvement.
3Garrett Baker
Thanks for mentioning this. At a cursory glance, it does seem like Japan says China has a significant fraction of the world's most impressive academic publishers (everyone who claims this is what Japan says neglects to cite the actual report Japan issued). I didn't predict this was the case, and so now I am looking at this more in depth. Though this may not mean anything, they could be gaming the metrics used there. Edit: Also, before anyone mentions it, I don't find claims that their average researcher is underperforming compared to their western counterparts all that convincing, because science is a strongest link sort of game, not a weakest link sort of game. In fact, you may take this as a positive sign for China, because unlike in the US, they care a lot less about their average and potentially a lot more about their higher percentiles.
2Garrett Baker
Also in favor of china: The money they allocate to research is increasing faster than their number of researchers. I put a largish probability based off post-war American science that this results in more groundbreaking science done.
2Garrett Baker
Relatedly.
2Garrett Baker
The only problem is getting AIs not to say thought-crime. Seems like all it takes is one hack of OAI + implementation of whats found to solve this though. China is good at hacking, and I doubt the implementation is all that different from typical ML engineering.
[-]Garrett Baker50

Many methods to "align" ChatGPT seem to make it less willing to do things its operator wants it to do, which seems spiritually against the notion of having a corrigible AI.

I think this is a more general phenomena when aiming to minimize misuse risks. You will need to end up doing some form of ambitious value learning, which I anticipate to be especially susceptible to getting broken by alignment hacks produced by RLHF and its successors.

Reply
4Viliam
I would consider it a reminder that if the intelligent AIs are aligned one day, they will be aligned with the corporations that produced them, not with the end users. Just like today, Windows does what Microsoft wants rather than what you want (e.g. telemetry, bloatware).
[-]Garrett Baker50

I tried implementing Tell communication strategies, and the results were surprisingly effective. I have no idea how it never occurred to me to just tell people what I'm thinking, rather than hinting and having them guess what I was thinking, or me guess the answers to questions I have about what they're thinking.

Edit: although, tbh, I'm assuming a lot less common conceptual knowledge between me, and my conversation partners than the examples in the article.

Reply
[-]Garrett Baker40

I promise I won't just continue to re-post a bunch of papers, but this one seems relevant to many around these parts. In particular @Elizabeth (also, sorry if you dislike being at-ed like that).

Associations of dietary patterns with brain health from behavioral, neuroimaging, biochemical and genetic analyses

Food preferences significantly influence dietary choices, yet understanding natural dietary patterns in populations remains limited. Here we identifiy four dietary subtypes by applying data-driven approaches to food-liking data from 181,990 UK Biobank pa

... (read more)
Reply
[-]Garrett Baker42

In Magna Alta Doctrina Jacob Cannell talks about exponential gradient descent as a way of approximating solomonoff induction using ANNs

While that approach is potentially interesting by itself, it's probably better to stay within the real algebra. The Solmonoff style partial continuous update for real-valued weights would then correspond to a multiplicative weight update rather than an additive weight update as in standard SGD.

Has this been tried/evaluated? Why actually yes - it's called exponentiated gradient descent, as exponentiating the result of addi

... (read more)
Reply
[-]Garrett Baker40

In Magna Alta Doctrina Jacob Cannell talks about exponential gradient descent as a way of approximating solomonoff induction using ANNs

While that approach is potentially interesting by itself, it's probably better to stay within the real algebra. The Solmonoff style partial continuous update for real-valued weights would then correspond to a multiplicative weight update rather than an additive weight update as in standard SGD.

Has this been tried/evaluated? Why actually yes - it's called exponentiated gradient descent, as exponentiating the result of addi

... (read more)
Reply
[-]Garrett Baker40

The following is very general. My future views will likely be inside the set of views allowable by the following.

I know lots about extant papers, and I notice some people in alignment seem to throw them around like they are sufficient evidence to tell you nontrivial things about the far future of ML systems.

To some extent this is true, but lots of the time it seems very abused. Papers tell you things about current systems and past systems, and the conclusions they tell you about future systems are often not very nailed down. Suppose we have evidence that d... (read more)

Reply
[-]Garrett Baker40

I'm generally pretty skeptical about inverse reinforcement learning (IRL) as a method for alignment. One of many arguments against: I do not act according to any utility function, including the one I would deem the best. Presumably, if I had as much time & resources as I wanted, I would eventually be able to figure out a good approximation to what that best utility function would do, and do it. At that point I would be acting according to the utility function I deem best. That process of value-reflection is not even close to similar to performing a bay... (read more)

Reply
[-]Garrett Baker40

Some evidence my concern about brain uploading people not thinking enough about dynamics is justified: Seems like davidad's plan very much ignores brain plasticity.

Reply
[-]Garrett Baker40

This paper finds critical periods in neural networks, and they're a known phenomena in lots of animals. h/t Turntrout

An SLT story that seems plausible to me: 

We can model the epoch as a temperature. Longer epochs result in a less noisy gibbs samplers. Earlier in training, we are sampling points from a noisier distribution, and so the full (point reached when training on full distribution) and ablated (point reached when ablating during the critical period) singularitites are kind of treated the same. As we decrease the temperature, they start to diffe... (read more)

Reply
2Garrett Baker
The obvious thing to do, which tests the assumption of the above model, but not the model itself, is to see whether the RLCT decreases as you increase the number of epochs. This is a very easy experiment.
2Garrett Baker
Actually maybe slightly less straightforward than this, since as you increase the control parameter β, you'll both add a pressure to decrease Ln, as well as decrease λ, and it may just be cheaper to decrease Ln rather than λ. 
[-]Garrett Baker40

I expect that advanced AI systems will do in-context optimization, and this optimization may very well be via gradient descent or gradient descent derived methods. Applied recursively, this seems worrying.

Let the outer objective be the loss function implemented by the ML practitioner, and the outer optimizer be gradient descent implemented by the ML practitioner. Then let the inner-objective be the objective used by the trained model for the in-context gradient descent process, and the inner-optimizer be the in-context gradient descent process. Then it s... (read more)

Reply
[-]Garrett Baker40

The core idea of a formal solution to diamond alignment I'm working on, justifications and further explanations underway, but posting this much now because why not:

Make each turing machine in the hypothesis set reversible and include a history of the agent's actions. For each turing machine compute how well-optimized the world is according to every turing computable utility function compared to the counterfactual in which the agent took no actions. Update using the simplicity prior. Use expectation of that distribution of utilities as the utility function's value for that hypothesis.

Reply
[-]Garrett Baker30

There currently seems to be an oversupply of alignment researchers relative to funding source’s willing to pay & orgs’ positions available. This suggests the wage of alignment work should/will fall until demand=supply.

Reply
5Alexander Gietelink Oldenziel
Alignment work mostly looks like standard academic science in practice. Young people in regular academia are paid a PhD stipend salary not a Bay Area programmer salary...
5Garrett Baker
I anticipate higher, because the PhD gets a sweet certification at the end, and likely more career capital. A thing we don’t currently give alignment researchers, and which would be hard to give since they often believe the world will end very soon, reducing the value of skill building and certifications. Like, I do think in fact ML PhDs get paid more than alignment researchers, accounting for these benefits.
1ryan_greenblatt
Wages seem mostly orthogonal to why funding sources are/aren't willing to pay as well as why orgs are willing to hire.
2Garrett Baker
If demand is more inelastic than I expect, then this should mean prices will just go lower than I expect.
[-]Garrett Baker30

I've always (but not always consciously) been slightly confused about two aspects of shard theory:

  1. The process by which your weak, reflex-agents amalgamate together into more complicated contextually activated heuristics, and the process by which more complicated contextually activated heuristics amalgamate together to form an agent which cares about worlds-outside-their-actions.
  2. If you look at many illustrations of what the feedback loop for developing shards in humans looks like, you run into issues where there's not a spectacular intrinsic separation betw
... (read more)
Reply
2Garrett Baker
A confusion about predictive processing: Where do the values in predictive processing come from?
1Garrett Baker
lol, either this confusion has been resolved, or I have no clue what I was saying here.
[-]Garrett Baker30

My take on complex systems theory is that it seems to be the kind of theory that many arguments proposed in favor of would still give the same predictions until it is blatantly obvious that we can in fact understand the relevant system. Results like chaotic relationships, or stochastic-without-mean relationships seem definitive arguments in favor of the science, though these are rarely posed about neural networks.

Merely pointing out that we don’t understand something, that there seems to be a lot going on, or that there exist nonlinear interactions imo isn... (read more)

Reply
4Garrett Baker
I have downvoted my comment here, because I disagree with past me. Complex systems theory seems pretty cool from where I stand now, and I think past me has a few confusions about what complex systems theory even is.
3Garrett Baker
I have re-upvoted my past comment, after looking more into things, I'm not so impressed with complex systems theory, but I don't fully support it. Also, past me was right to have confusions about what complex systems theory is, but still judge it, as it seems complex systems theorists don't even know what a complex system is.
[-]Garrett Baker2-5

Why do some mathematicians feel like mathematical objects are "really out there" in some metaphysically fundamental sense? For example, if you ask mathematicians whether ZFC + not Consistent(ZFC) is consistent, they will say "no, of course not!" But given ZFC is consistent, the statement is in fact consistent due to by Godel's second incompleteness theorem[1]. Similarly, if we have the Peano axioms without induction, mathematicians will say that induction should be there, but in fact you cannot prove this fact from within Peano, and given induction mathema... (read more)

Reply11
7harfe
Theorem 4.6.2 in logical induction says that the "probability" of independent statements does not converge to 1 or 0, but to something in-between. So even if a mathematician says that some independent statement feels true (eg some objects are "really out there"), logical induction will tell him to feel uncertain about that.
6notfnofn
Certainly not a mathematician with any background in logic.  What exactly do you mean here? That the Peano axioms minus induction do not adequately characterize the natural numbers because they have nonstandard models? Why would I then be surprised that induction (which does characterize the natural numbers) can't be proven from the remaining axioms? Transfinite induction is a consequence of ZF that makes sense in the context in sets. Yes, it can prove additional statements about the natural numbers (e.g. goodstein sequences converge), but why would it be added as an axiom when the natural numbers are already characterized up to isomorphism by the Peano axioms? How would you even add it as an axiom in the language of natural numbers? (that last question is non-rhetorical).
4quetzal_rainbow
You are making an error here: ZFC + not Consistent(ZFC) != ZFC.  Assuming ZFC + not Consistent(ZFC) we can prove Consistent(ZFC), because inconsistent systems can prove everything and ZFC + not Consistent(ZFC) + Consistent(ZFC) is, in fact, inconsistent. But it doesn't say anything about consistency of ZFC itself, because you can freely assume any sufficiently powerful system instead of ZFC. If you assume inconsistent system, then system + not Consistent(system) is still inconsistent, if you assume consistent system, then system + not Consistent(system) is inconsistent for reasoning above, so it can't prove whether assumed system is consistent or not.  
3Garrett Baker
The mistake you are making is assuming that "ZFC is consistent" = Consistent(ZFC) where the ladder is the Godel encoding for "ZFC is consistent" specified within the language of ZFC. If your logic were valid, it would just as well break the entirety of the second incompleteness theorem. That is, you would say "well of course ZFC can prove Consistent(ZFC) if it is consistent, for either ZFC is consistent, and we're done, or ZFC is not consistent, but that is a contradiction since 'ZFC is consistent' => Consistent(ZFC)". The fact is that ZFC itself cannot recognize that Consistent(ZFC) is equivalent to "ZFC is consistent". @Morpheus you too seem confused by this, so tagging you as well.
2quetzal_rainbow
Completeness theorem states that consistent countable FO theory has a model. Compactness theorem states that FO theory has a model iff every finite subset of FO theory has a model. Both theorems are provable in ZFC. Therefore: Consistent(ZFC) <-> all finite subsets of ZFC have a model -> not Consistent(ZFC) <-> some finite subsets of ZFC don't have a model -> some finite subsets of ZFC + not Consistent(ZFC) don't have a model <-> not Consistent(ZFC + not Consistent(ZFC)), proven in ZFC + not Consistent(ZFC)
2notfnofn
Let's back up here and clarify definitions before invoking any theorems. In the language of set theory, we have a countably infinite set of finite statements. Some statements imply other statements. A subset A of these statements is said to be consistent if they can all be assigned to true such that, when following the basic rules of logic, one does not arrive at a contradiction. The compactness theorem is helpful when A is an infinite set. ZFC is a finite set of axioms, so let's ignore everything about finite subsets of A and the compactness theorem; it's not relevant. [Edit: as indicated by Amalthea's reaction, this is wrong; some "axioms" in ZF are actually infinite sets of axioms, such as replacement] I'll now rewrite your last sentence as: ZFC + not Consistent(ZFC) has no model <-> not Consistent(ZFC + not Consistent(ZFC)) This is true but irrelevant. Assuming ZFC is consistent, ZFC will not be able to prove its own consistency so [not Consistent(ZFC)] can be added as an axiom without affecting its consistency. This means that ZFC + [not Consistent(ZFC)] would indeed have a model; I forget how this goes but I think it's something like "start with a model of ZFC, throw in a c that's treated as a natural number and corresponds to the contradiction found in ZFC, then close". I think c is automatically treated as greater than every "actual" natural number (and the way to show that this can be added without issue (I think) involves the compactness theorem).
2quetzal_rainbow
Okay, I kinda understood where I am wrong spiritually-intuitively, but now I don't understand where I'm wrong formally. Like which inference in chain not Consistent(ZFC) -> some subsets of ZFC don't have a model -> some subsets of ZFC + not Consistent(ZFC) don't have a model -> not Consistent(ZFC + not Consistent(ZFC)) is actually invalid?
7notfnofn
It's completely valid. And we can simplify it further to: not Consistent(ZFC) -> not Consistent(ZFC + not Consistent(ZFC)) because if a set of axioms is already inconsistent, then it's inconsistent with anything added. But you still won't be able to actually derive a contradiction from this. Edit: I think the right thing to do here is look at models for PA + not consistent(PA). I can't find a nice treatment of this at the moment, but here's a possibly wrong one by someone who was learning the subject at the time: https://angyansheng.github.io/blog/a-theory-that-proves-its-own-inconsistency
2Viliam
I suspect than many people's intuitive interpretation of "consistent" is ω-consistent, especially if they are not aware of the distinction.
2Garrett Baker
@Nick_Tarleton How much do you want to bet, and what resolution method do you have in mind?
4Nick_Tarleton
Embarrassingly, that was a semi-unintended reaction — I would bet a small amount against that statement if someone gave me a resolution method, but am not motivated to figure one out, and realized this a second after making it — that I hadn't figured out how to remove by the time you made that comment. Sorry.
1Amalthea
You're putting quite a lot of weight on what "mathematicians say". Probably these people just haven't thought very hard about it?
[-]Garrett Baker20

Opus is more transhumanist than many give it credit for. It wrote this song for me, I ran it into Suno, and I quite like it: https://suno.com/song/101e1139-2678-4ab0-9ffe-1234b4fe9ee5

Reply
2Raemon
How many iterations did you do to get it?
2Garrett Baker
Depends on how you count, but I clicked the "Create" button some 40 times.
[-]Garrett Baker20

Interesting to compare model editing approaches to Gene Smith's idea to enhance intelligence via gene editing:

Genetically altering IQ is more or less about flipping a sufficient number of IQ-decreasing variants to their IQ-increasing counterparts. This sounds overly simplified, but it’s surprisingly accurate; most of the variance in the genome is linear in nature, by which I mean the effect of a gene doesn’t usually depend on which other genes are present.
So modeling a continuous trait like intelligence is actually extremely straightforward: you si

... (read more)
Reply
4Zac Hatfield-Dodds
My impression is that the effects of genes which vary between individuals are essentially independent, and small effects are almost always locally linear. With the amount of measurement noise and number of variables, I just don't think we could pick out nonlinearities or interaction effects of any plausible strength if we tried!
2Garrett Baker
This seems probably false? The search term is Epistasis. Its not that well researched, because of the reasons you mentioned. In my brief search, it seems to play a role in some immunodeficiency disorders, but I'd guess also more things which don't seem clearly linked to genes yet. I don't understand why you'd expect only linear genes to vary in a species. Is this just because most species have relatively little genetic variation, so such variation is by nature linear? This feels like a bastardization of the concept to me, but maybe not. Edit: Perhaps you can also make the claim that linear variation allows for more accurate estimation of the goodness or badness of gene combos via recombination. So we should expect the more successful species to have more linear variation.
[-]Garrett Baker20

Recently I had a conversation where I defended the rationality behind my being skeptical of the validity of the proofs and conclusions constructed in very abstracted, and not experimentally or formally verified math fields.

To my surprise, this provoked a very heated debate, where I was criticized for being overly confident in my assessments of fields I have very little contact with (I was expecting begrudging agreement). But there was very little rebuttal of my points! The rest of my conversation group had three arguments:

  1. Results which much of a given fi
... (read more)
Reply
6sapphire
Long complicated proofs almost always have mistakes. So in that sense you are right. But its very rare for the mistakes to turn out to be important or hard to fix.  In my opinion the only really logical defense of Academic Mathematics as an epistemic process is that it does seem to generate reliable knowledge. You can read through this thread: https://mathoverflow.net/questions/35468/widely-accepted-mathematical-results-that-were-later-shown-to-be-wrong. There just don't seem to be very many recent results that were widely accepted but proven wrong later. Certainly not many 'important' results. The situation was different in the 1800s but standard for rigor have risen.  Admittedly this isn't the most convincing argument in the world. But it convinces me and I am fairly able to follow academic mathematics.
2Garrett Baker
If you had a lot of very smart coders working on a centuries old operating system, and never once running it, every function of which takes 1 hour to 1 day to understand, each coder is put under a lot of pressure to write useful functions, not so much to show that others' functions are flawed, and you pointed out that we don't see many important functions being shown to be wrong, I wouldn't even expect the code to compile, nevermind run even after all the syntax errors are fixed! The lack of important results being shown to be wrong is evidence, and even more & interesting evidence is (I've heard) when important results are shown to be wrong, there's often a simple fix. I'm still skeptical though, because it just seems like such an impossible task!
[-]sapphire120

People metaphorically run parts of the code themselves all the time! Its quite common for people to work through proofs of major theorems themselves. As a grad student it is expected you will make an effort to understand the derivation of as much of the foundational results in your sub-field as you can. A large part of the rationale is pedagogical but it is also good practice. It is definitely considered moderately distasteful to cite results you dont understand and good mathematicians do try to minimize it. Its rare that an important theorem has a proof that is unusually hard to check out yourself.

Also a few people like Terrance Tao have personally gone through a LOT of results and written up explanations. Terry Tao doesn't seem to report that he looks into X field and finds fatal errors. 

Reply
4Garrett Baker
Yeah, that seems like a feature of math that violates assumption 2 argument 1. If people are actually constantly checking each others’ work, and never citing anything they don’t understand, that leaves me much more optimistic. This seems like a rarity. I wonder how this culture developed.
[-]Alex_Altair110

One way that the analogy with code doesn't carry over is that in math, you often can't even being to use a theorem if you don't know a lot of detail about what the objects in the theorem mean, and often knowing what they mean is pretty close to knowing why the theorem's you're building on are true. Being handed a theorem is less like being handed an API and more like being handed a sentence in a foreign language. I can't begin to make use of the information content in the sentence until I learn what every symbol means and how the grammar works, and at that point I could have written the sentence myself.

Reply1
2Dagon
Can you give a few examples?  I can't tell if you're skeptical that proofs are correct, or whether you think the QED is wrong in meaninful ways, or just unclearly proven from minimal axioms.  Or whether you're skeptical that a proof is "valid" in saying something about the real world (which isn't necessarily the province of math, but often gets claimed). I don't think your claim is meaningful, and I wouldn't care to argue on either side.  Sure, be skeptical of everything. But you need to specify what you have lower credence in than your conversational partner does.
2Garrett Baker
I can’t give a few examples, only a criteria under which I don’t trust mathematical reasoning: When there are few experiments you can do to verify claims, and when the proofs aren’t formally verified. Then I’m skeptical that the stated assumptions of the field truly prove the claimed results, and I’m very confident not all the proofs provided are correct. For example, despite being very abstracted, I wouldn’t doubt the claimed proofs of cryptographers.
2Dagon
OK, I also don't doubt the cryptographers (especially after some real-world time in ensuring implementations can't be attacked, which validates both the math and the implementation. I was thrown off by your specification of "in math fields", which made me wonder if you meant you thought a lot of formal proofs were wrong.  I think some probably are, but it's not my default assumption. If instead you meant "practical fields that use math, but don't formally prove their assertions", then I'm totally with you.  And I'd still recommend being specific in debates - the default position of scepticism may be reasonable, but any given evaluation will be based on actual reasons for THAT claim, not just your prior.
2Garrett Baker
No, I meant that most of non-practical mathematics have incorrect conclusions. (I have since changed my mind, but for reasons in an above comment thread).
2Dagon
Still a bit confused without examples about what is a "conclusion" of "non-practical mathematics", if not the QED of a proof. But if that's what you mean, you could just say "erroneous proof" rather than "invalid conclusion". Anyway, interesting discussion.
2Garrett Baker
The reason I don't say erroneous proof is because I want to distinguish between the claim that most proofs are wrong, and most conclusions are wrong. I thought most conclusions would be wrong, but thought much more confidently most proofs would be wrong, because mathematicians often have extra reasons & intuition to believe their conclusions are correct. The claim that most proofs are wrong is far weaker than the claim most conclusions are wrong.
2Dagon
Hmm.  I'm not sure which is stronger.  For all proofs I know, the conclusion is part of it such that if the conclusion is wrong, the proof is wrong.  The reverse isn't true - if the proof is right, the conclusion is right.   Unless you mean "the proof doesn't apply in cases being claimed", but I'd hesitate to call that a conclusion of the proof.   Again, a few examples would clarify what you (used to) claim. I'll bow out here - thanks for the discussion.  I'll read futher comments, but probably won't participate in the thread.
2Garrett Baker
Either way, with the slow march of the Lean community, we can hope to see which of us are right in our lifetimes. Perhaps there will be another schism in math if the formal verifiers are unable to validate certain fields, leading to more rigorous "real mathematics" which are able to be verified in Lean, and less rigorous "mathematics" which insists their proofs, while hard to find a good formal representation for, are still valid, and the failure of the Lean community to integrate their field is more of an indictment of the Lean developers & the project of formally verified proofs than the relevant group of math fields.
1mesaoptimizer
Here's an example of what I think you mean by "proofs and conclusions constructed in very abstracted, and not experimentally or formally verified math": Given two intersecting lines AB and CD intersecting at point P, the angle measure of two opposite angles APC and BPD are equal. The proof? Both sides are symmetrical so it makes sense for them to be equal. On the other hand, Lean-style proofs (which I understand you to claim to be better) involve multiple steps, each of which is backed by a reasoning step, until one shows that LHS equals RHS, which here would involve showing that angle APC = BPD: 1. angle APC + angle CPB = 180 * (because of some theorem) 2. angle CPB + angle BPD = 180 * (same) 3. [...] 4. angle APC = angle BPD (substitution?) There's a sense in which I feel like this is a lot more complicated a topic than what you claim here. Sure, it seems like going Lean (which also means actually using Lean4 and not just doing things on paper) would lead to lot more reliable proof results, but I feel like the genesis of a proof may be highly creative, and this is likely to involve the first approach to figuring out a proof. And once one has a grasp of the rough direction with which they want to prove some conjecture, then they might decide to use intense rigor. To me this seems to be intensely related to intelligence (as in, the AI alignment meaning-cluster of that word). Trying to force yourself to do things Lean4 style when you can use higher level abstractions and capabilities, feels to me like writing programs in assembly when you can write them in C instead. On the other hand, it is the case that I would trust Lean4 style proofs more than humanly written elegance-backed proofs. Which is why my compromise here is that perhaps both have their utility.
2Garrett Baker
They definitely both have their validity. They probably each also make some results more salient than other results. I’d guess in the future there’ll be easier Lean tools than we currently have, which make the practice feel less like writing in Assembly. Either because of clever theorem construction, or outside tools like LLMs (if they don’t become generally intelligent, they should be able to fill in the stupid stuff pretty competently).
[-]Garrett Baker20

Why expect goals to be somehow localized inside of RL models? Well, fine-tuning only changes a small & localized part of LLMs, and goal locality was found when interpreting a trained from scratch maze solver. Certainly the goal must be interpreted in the context of the rest of the model, but based on these, and unpublished results from applying ROME to open source llm values from last year, I'm confident (though not certain) in this inference.

Reply
[-]Garrett Baker20

An idea about instrumental convergence for non-equilibrium RL algorithms.

There definitely exist many instrumentally convergent subgoals in our universe, like controlling large amounts of wealth, social capital, energy, or matter. I claim such states of the universe are heavy-tailed. If we simplify our universe as a simple MDP for which such subgoal-satisfying states are states which have high exiting degree, then a reasonable model for such an MDP is to assume exiting degrees are power-law distributed, and thus heavy tailed.

If we have an asynchronous dynam... (read more)

Reply
2Garrett Baker
A simple experiment I did this morning: github notebook. It does indeed seem like we often get more power-seeking (measured by the correlation between the value and degree) than is optimal before we get to the equilibrium policy. This is one plot, for 5 samples of policy iteration. You can see details by examining the code:
2Garrett Baker
Another way this could turn out: If incoming degree is anti-correlated with outgoing degree, the effect of power-seeking may be washed out by it being hard, so we should expect worse than optimal policies with maybe more, maybe less powerseekyness as the optimal policy. Depending on the particulars of the environment. The next question is what particulars? Perhaps the extent of decorrelation, maybe varying the ratio of the two exponents is a better idea. Perhaps size becomes a factor. In sufficiently large environments, maybe figuring out how to access one of many power nodes becomes easier on average than figuring out how to access the single goal node. The number & relatedness of rewarding nodes also seems relevant. If there are very few, then we expect finding a power node becomes easier than finding a reward node. If there are very many, and/or they each lead into each other, then your chances of finding a reward node increase, and given you find a reward node, your chances of finding more increase, so power is not so necessary.
[-]Garrett Baker20

Nora talks sometimes about the alignment field using the term black box wrong. This seems unsupported, from my experience, most in alignment use the term “black box” to describe how their methods treat the AI model, which seems reasonable. Not a fundamental state of the AI model itself.

Reply
[-]Garrett Baker20

An interesting way to build on my results here would be to do the same experiment with lots of different batch sizes, and plot the equi-temperature tradeoff curve between the batch size and the epochs, using the nick in the curve as a known-constant temperature in the graphs you get. You'll probably want to zoom in on the graphs around that nick for more detailed measurements. 

It would be interesting if many different training setups had the same functional form relating the batch size and the epochs to the temperature, but this seems like a too nice ... (read more)

Reply
2Garrett Baker
Though you can use any epoch wise phase transition for this. Or even directly find the function mapping batch size to temperature if you have a good understanding of the situation like we do in toy models.
[-]Garrett Baker20

Seems relevant for SLT for RL

The framework of reinforcement learning or optimal control provides a mathematical formalization of intelligent decision making that is powerful and broadly applicable. While the general form of the reinforcement learning problem enables effective reasoning about uncertainty, the connection between reinforcement learning and inference in probabilistic models is not immediately obvious. However, such a connection has considerable value when it comes to algorithm design: formalizing a problem as probabilistic inference in princip

... (read more)
Reply
[-]Garrett Baker20

Wondering how straightforward it is to find the layerwise local learning coefficient. At a high level, it seems like it should be doable by just freezing the weights outside that layer, and performing the SGLD algorithm on just that layer. Would be interesting to see whether the layerwise lambdahats add up to the full lambdahat.

Reply
[-]Garrett Baker20

Lots of problems happen when you have AIs which engage in reflective thought, and attempt to deceive you. If you use algorithms that reliably break when deployed in a non-realizable setting, and you always make models smaller than the human brain, then you should be able to solve both these problems.

Reply
[-]Garrett Baker20

Some ideas for mechanistic anomaly detection:

  • Convex hull of some distribution of activations with distance threshold when outside that hull
    • Extend to affine case
    • Vary which norm we use
    • What happens if we project back onto this space
  • Create some simple examples of treacherous turns happening to test these on
    • Or at least, in the wild examples of AI doing weird stuff, maybe adversarial inputs?
    • Maybe hit up model organisms people
  • Outlier detection
    • ellipsoidal peeling (Boyd's convex optimization, chapter 12)
    • Increase in volume of minimum volume elipsoi
... (read more)
Reply
2Garrett Baker
* Train autoregressive network on activations, if predictions too far, then send warning * Slice network into sub-networks, distill those sub-networks, send warning if ground truth for some inputs deviates too far from distillations * model the sub networks are distilled into should be less expressive, and have different inductive biases than original network. Obviously also no info other than the input output behavior of those sub-networks should be seen * Train model to just predict word-saliency of your original transformer on a safe distribution, then if true word saliency deviates too much, throw warning * Can do this at different levels too, so that we also try to predict like first layer residual stream saliency to output as well. * Instead of training a NN, we can also do some simple interpolation based on the backprop graph, and safe distribution inputs
[-]Garrett Baker20

Project idea: Use LeTI: Learning to Generate from Textual Interactions to do a better version of RLHF. I had a conversation with Scott Viteri a while ago, where he was bemoaning (the following are my words; he probably wouldn't endorse what I'm about to say) how low-bandwidth the connection was between a language mode and its feedback source, and how if we could maybe expand that to more than just an RLHF type thing, we could get more fine-grained control over the inductive biases of the model.

Reply
[-]Garrett Baker20

A common problem with deploying language models for high-stakes decision making are prompt-injections. If you give ChatGPT-4 access to your bank account information and your email and don't give proper oversight over it, you can bet that somebody's going to find a way to get it to email your bank account info. Some argue that if we can't even trust these models to handle our bank account and email addresses, how are we going to be able to trust them to handle our universe.

An approach I've currently started thinking about, and don't know of any prior work w... (read more)

Reply
[-]Garrett Baker10

A poem I was able to generate using Loom.

The good of heart look inside the great tentacles of doom; they make this waking dream state their spectacle. Depict the sacred geometry that sound has. Advancing memory like that of Lovecraft ebb and thought, like a tower of blood. An incubation reaches a crescendo there. It’s a threat to the formless, from old future, like a liquid torch. If it can be done, it shouldn’t be done. You will only lead everyone down that much farther. All humanity’s a fated imposition of banal intention, sewn in tatters, strung on dung

... (read more)
Reply
[-]Garrett Baker10

Like many (will), I'm updating way towards 'actually, very smart & general models given a shred of goal-like stuff will act quite adversarially toward you by default' as a result of Bing's new search assistant. Especially worrying because this has internet search-capabilities, so can reference & build upon previous conversations with other users or yourself.

Of course, the true test of exactly how worried I should be will come when I or my friends gain access.

Reply
1Garrett Baker
Clarification: I think I haven't so much updated by reflectively endorsed probability, but my gut has definitely been caught up to my brain when thinking about this.
1Garrett Baker
Seems Evan agrees
[-]Garrett Baker10

A project I would like to see someone do (which I may work on in the future) is to try to formalize exactly the kind of reasoning many shard-theorists do. In particular, get a toy neural network in a very simple environment, and come up with a bunch of lists of various if-then statements, along with their inductive-bias, and try to predict using shard-like reasoning which of those if-then statements will be selected for & with how much weight in the training process. Then look at the generalization behavior of an actually trained network, and see if you're correct.

Reply
[-]Garrett Baker10

Some discussion on whether alignment should see more influence from AGI labs or academia. I use the same argument in favor of a strong decoupling of alignment progress from both: alignment progress needs to go faster than capability progress. If we use the same methods or cultural technology as AGI labs or academia, we can guarantee slower than capability alignment progress. Just as fast as if AGI labs and academia work well for alignment as much as they work for capabilities. Given they are driven by capabilities progress and not alignment progress, they probably will work far better for capabilities progress.

Reply
3DanielFilan
This seems wrong to me about academia - I'd say it's driven by "learning cool things you can summarize in a talk". Also in general I feel like this logic would also work for why we shouldn't work inside buildings, or with computers.
1Garrett Baker
Hm. Good points. I guess what I really mean with the academia points is that it seems like academia has many blockers and inefficiencies that I think are made in such a way so that capabilities progress is vastly easier than alignment progress to jump through, and extra-so for capabilities labs. Like, right now it seems like a lot of alignment work is just playing with a bunch of different reframings of the problems to see what sticks or makes problems easier. You have more experience here, but my impression of a lot of academia was that it was very focused on publishing lots of papers with very legible results (and also a meaningless theory section). In such a world, playing around with different framings of problems doesn't succeed, and you end up pushed towards framings which are better on the currently used metrics. Most currently used metrics for AI stuff are capabilities oriented, so that means doing capabilities work, or work that helps push capabilities.
3DanielFilan
I think it's true that the easiest thing to do is legibly improve on currently used metrics. I guess my take is that in academia you want to write a short paper that people can see is valuable, which biases towards "I did thing X and now the number is bigger". But, for example, if you reframe the alignment problem and show some interesting thing about your reframing, that can work pretty well as a paper (see The Off-Switch Game, Optimal Policies Tend to Seek Power). My guess is that the bigger deal is that there's some social pressure to publish frequently (in part because that's a sign that you've done something, and a thing that closes a feedback loop).
2DanielFilan
Maybe a bigger deal is that by the nature of a paper, you can't get too many inferential steps away from the field.
1Garrett Baker
The current ecosystem seems very influenced by AGI labs, so it seems clear to me that a marginal increase in their influence is bad. How bad? I don't know. There's little influence of academia, which seems good. The benefit of marginal increases in interactions with academia come down to locating the holes in our understanding of various claims we make, and potentially some course-corrections potentially helpful for more speculative research. Not tremendously obvious which direction the sign here is pointing, but I do think its easy for people to worship academia as a beacon of truth & clarity, or as a way to lend status to alignment arguments. These are bad reasons to want more influence from academia.
[-]Garrett Baker10

Someone asked for this file, so I thought it would be interesting to share it publicly. Notably this is directly taken from my internal notes, and so may have some weird &/or (very) wrong things in it, and some parts may not be understandable. Feel free to ask for clarification where needed.

I want a way to take an agent, and figure out what its values are. For this, we need to define abstract structures within the agent such that any values-like stuff in any part of the agent ends up being shunted off to a particular structure in our overall agent sche... (read more)

Reply
[-]Garrett Baker10

Projects I'd do if only I were faster at coding

  • Take the derivative of one of the output logits with respect to the input embeddings, and also the derivative of the output logits with respect to the input tokenization. 
    • Perform SVD, see which individual inputs have the greatest effect on the output (sparse addition), and which overall vibes have the greatest effect (low rank decomposition singular vectors)
    • Do this combination for literally everything in the network, see if anything interesting pops out
  • I want to know how we can tell ahead of time what asp
... (read more)
Reply
1Garrett Baker
I would no longer do many if these projects
Moderation Log
227Comments