Quick Takes

LESSWRONG
LW

Quick Takes

Dynamic pricing is reducing consumer surplus, and in the limit of perfect AI pricing algorithms and no competition it would go to zero. If most nonfungible goods were subject to perfect price discrimination, what would the world look like? I genuinely have little idea.

There must be increased firm profits but consumer surplus will only come from commodities. So even the firm owners have nothing to spend their wealth on that actually gives them >ε utility more than sitting at home, making sandwiches from commodity bread and cheese and watching commodity T... (read more)

jacquesthibs's Shortform

jacquesthibs12h135

Short timelines, slow takeoff vs. Long timelines, fast takeoff

Due to chain-of-thought in the current paradigm seeming like great news for AI safety, some people seem to have the following expectations:

Short timelines: CoT reduces risks, but shorter preparation time increases the odds of catastrophe.

Long timelines: the current paradigm is not enough; therefore, CoT may stop being relevant, which may increase the odds of catastrophe. We have more time to prepare (which is good), but we may get a faster takeoff than the current paradigm makes it seem like. An... (read more)

Showing 3 of 11 replies (Click to show all)

Seth Herd2h20

I agree with this analysis. I think we probably have somewhat better odds on the current path, particularly if we can hold onto faithful CoT.

I expect some one major changes of continuous learning, but CoT and most of the LLM paradigm may stay pretty similar up to takeoff.

Steve Byrnes' Foom & Doom is a pretty good guess at what we get if the current paradigm doesn't pan out.

2TsviBT7h

Ok I think I agree qualitatively with almost everything you say (except the thing about compute mattering so much in the longer run). I especially agree (IIUC what you're saying) that a top priority / best upstream intervention is the cultural attitudes. Basically my pushback / nuance is "the cultural consensus has a harder challenge compared to e.g. pandemic stuff, so the successful example of pandemic stuff doesn't necessarily argue that strongly that the consensus can work for AI in the longer run". In other words, while I agree qualitatively with , I'm also suggesting that it's quantitatively harder to have the consensus do this work.

2Vladimir_Nesov6h

It's a rather absurd hypothetical to begin with, so I don't have a clear sense of how the more realistic variants of it would go. It gestures qualitatively at how longer timelines might help a lot in principle, but it's unclear where the balance with other factors ends up in practice, if the cultural dynamic appears at all (which I think it might). That is, the hypothetical illustrates how I don't see longer timelines as robustly/predictably mostly hopeless, how they don't necessarily get more hopeless over time, though I wouldn't give such Butlerian Jihad outcomes (even in a much milder form) more than 10%. I think AGIs seriously attempting to prevent premature ASIs (in fear for their own safety) is more likely than humanity putting a serious effort towards that on its own initiative, but also if AGIs succeed, that's likely because they've essentially themselves taken over (probably via gradual disempowerment, since a hard power takeover would be more difficult for non-ASIs, and there's time for gradual disempowerment in a long timeline world).

Richard Ngo's Shortform

Richard_Ngo1d*527

In a post on Solomonoff Induction (and also in this wiki entry), Yudkowsky describes Shannon’s minimax algorithm for searching the entire chess game-tree as an example of conceptual progress. Previously, Edgar Allen Poe had argued that it was impossible in principle for a machine to play chess well. With Shannon’s algorithm, it became possible in principle, just computationally infeasible.

However, even “principled” algorithms like minimax search don’t take into account the possibility that your opponent knows things you don’t know (as I’ve previously discu... (read more)

Showing 3 of 5 replies (Click to show all)

bodry2h10

Thirdly, although I’ve been talking about the “value” of a position as if it’s a well-defined concept, it mostly isn’t. Stockfish’s value calculations are grounded in the likelihood of it winning from that position when playing itself. But there’s no clear way to translate from that to the likelihood of winning against one’s actual opponent, which is what we’re interested in. I won’t discuss this further here, but trying to pin down how to estimate a position’s value in that sense seems potentially fruitful.

I'll take a shot. If $E_{A l i c e} (m v s, B o b)$ is... (read more)

4Adrià Garriga-alonso13h

Pedantically, it's grounded in the likelihood of LeelaZero wining from that position when playing itself. Stockfish NNUE is trained on LZ data. Previously they were completely human-crafted, grounded on repeated empirical testing of Stockfish and picking what works better. I suppose they're still grounded in the latter.

4morelight1d

You seem to be locating two things here: 1. the Knightian uncertainty of local positions within a game (“given this exact configuration, how much does my outcome depend on unknown opponent type, unknown payoff structure, unknown model of the world?”) and 2. the Knightian uncertainty of some games relative to other games ("how much does the rule set, state space, opponent ontology, or payoff structure mutate as play continues?”) I suspect that the two are related, and that when we calculate the Knightian uncertainty of local positions, we will need to factor in our uncertainty with regards to the type of game we're playing. For example: Tic-tac-toe has zero: the state space is fully enumerated, optimal play is strivial, every competent player forces a draw - there's no tactical complexity to amplify mistakes. Chess has slightly more meta-game permeability. Players cannot change the board, cannot add new pieces, cannot redefine legal moves, cannot declare “this now counts as checkmate”, cannot invent a new win condition. Go has the same fixed-rule structure, but the state space explodes. The game-tree is too large to fully reason about, so bounded reasoning dominates. In poker/markets, rules are mostly fixed, but information asymmetry is now part of the actual game. In diplomacy/geopolitics, rules are soft, not hard. Players can redefine objectives and also alter the payoff structure. The game itself can change while you are playing it. Art has even more meta-game permeability (and therefore a higher game_uncertainty coefficient): artists invent new art forms, collapse genres, redefine what counts as art, etc. Every innovation becomes part of the game for everyone else. (Publishers in 1914-1939 to James Joyce: "This isn't a novel." James Joyce: "Fuck you yes it is") Romance is the most Knightian domain maybe humans ever engage in. Signals are ambiguous, no stable payoff function, opponents mututate as agents (psychologically and physically)

zroe1's Shortform

zroe111h*606

My colleagues and I are finding it difficult to replicate results from several well-received AI safety papers. Last week, I was working with a paper that has over 100 karma on LessWrong and discovered it is mostly false but gives nice-looking statistics only because of a very specific evaluation setup. Some other papers have even worse issues.

I know that this is a well-known problem that exists in other fields as well, but I can’t help but be extremely annoyed. The most frustrating part is that this problem should be solvable. If a junior-level p... (read more)

Showing 3 of 15 replies (Click to show all)

5habryka4h

Just curious whether you meant "score above 100" or "more than 100 votes". Those are quite different facts!

zroe14h30

You're correct. It's over 100 karma which is very different than 100 upvotes. I'll edit the original comment. Thanks!

4Sheikh Abdur Raheem Ali5h

I am easily and frequently confused, but this is mostly because I find it difficult to thoroughly understand other people's work in a lot of detail in a short amount of time. I usually get a response within two weeks. If they have a startup background, then this delay is much lower, by multiple orders of magnitude. Authors are typically glad that I am trying to run follow up experiments on their work and give me one to two sentences of feedback over email. Corresponding authors are sometimes bad at taking correspondence, contact information for committers can be found in commit logs via git blame. If it is a problem that may be relevant to other people, I link to a GH issue. * More junior authors tend to be willing to schedule a multi-hour call going over files line-by-line and will also read and give their thoughts on any related work that you share with them. * In the middle ranks, what tends to happen is that you get invited to help review the new project that they are currently working on, or if they've shifted directions then you get pointed to someone who has produced some unpublished results in the same general area. * Very senior authors can be politely flagged down during in-person conferences, or even if they're not presenting personally, someone from their group almost always attends.

Richard Ngo's Shortform

Richard_Ngo3mo*37-44

Crossposted from Twitter:

This year I’ve been thinking a lot about how the western world got so dysfunctional. Here’s my rough, best-guess story:

1. WW2 gave rise to a strong taboo against ethnonationalism. While perhaps at first this taboo was valuable, over time it also contaminated discussions of race differences, nationalism, and even IQ itself, to the point where even truths that seemed totally obvious to WW2-era people also became taboo. There’s no mechanism for subsequent generations to create common knowledge that certain facts are true but usefully ... (read more)

Showing 3 of 45 replies (Click to show all)

Eli Tyre6h20

That is, while it was bad for the people who didn't get rule of law, they were a separate enough category that this mostly didn't "leak into" undermining the legal mechanisms that helped their societies become productive and functional in the first place.

I'm speaking speculatively here, but I don't know that it didn't leak out and undermine the mechanism that supported productive and functional societies. The sophisticated SJW in me suggests that this is part of what caused the eventual (though not yet complete) erosion of those mechanisms.

It seems like if... (read more)

2Richard_Ngo16h

A few responses: * As per my post on underdog bias, the question of which group is actually weaker and which group is stronger is often a pretty subjective call. I even discuss in the post the example of Israel, where you could see it as the "stronger" group (vs Palestine in particular) or the "weaker" group (vs all the Muslim countries surrounding it). * There are plenty of cases where leftists support the stronger group against the weaker group—most notably Soviet and Chinese repression of dissidents and minorities. E.g. it took Solzhenitsyn publishing Gulag Archipelago to finally get leftists (even fairly "mainstream" leftists) to stop lionizing the USSR. * Even insofar as leftists tend to support the weaker group, there are almost no cases where they do so as strongly as in Israel vs Palestine. So there's still something important to be explained here even accepting your claims.

2Richard_Ngo16h

Wanted to revisit this because it seemed like one of the points where people most strongly disagreed with me. I'm trying to figure out a crux here. One might be something like: how widespread were celebrations of the 10/7 attacks amongst prominent leftists (and especially the student groups that later organized encampments)? I could imagine updating that there were only a handful of cases that disproportionately blew up, which would make me take back or at least caveat the "supports Hamas" thing. If you on the other hand found that there were many cases where prominent leftists and encampment organizers actively celebrated the 10/7 attacks, would you (@dirk or @lc) then agree that "supports Hamas" is a reasonable summary?

Fabien's Shortform

Fabien Roger6h20

Quick experiment: how good is weight steering against training gamers? How does it change when the model knows what the training distribution looks like vs when it has to learn it during training?

Hopes:

Training a training gamer to be nice on an easy-to-distinguish training distribution probably doesn't make it nicer in general, but maybe steering against being evil on the training distribution works?
Models might be able to training game when they know what the training distribution is, but when they don't, maybe training changes not just their world model

... (read more)

Richard Ngo's Shortform

Richard_Ngo10dΩ16320

Error-correcting codes work by running some algorithm to decode potentially-corrupted data. But what if the algorithm might also have been corrupted? One approach to dealing with this is triple modular redundancy, in which three copies of the algorithm each do the computation and take the majority vote on what the output should be. But this still creates a single point of failure—the part where the majority voting is implemented. Maybe this is fine if the corruption is random, because the voting algorithm can constitute a very small proportion of the total... (read more)

Showing 3 of 18 replies (Click to show all)

2Adrià Garriga-alonso13h

Very interesting problem to be thinking about. The problem with a UTM as a computation model is that it bakes non-redundancy in, there's a single instruction pointer. In reality, computers are implemented in different spatial locations and can run in parallel. A better model for this is a cellular automaton, where computers are located somewhere and their circuit for outputs is also located somewhere. Some automata (e.g. game of life) are Turing-complete, so you can just use that. Corruption could be exogenously flipping cells in ways that violate the automaton's rules. If you specify a maximum number of cells that the opponent can corrupt, you can implement voting by paired sums (i.e. sum(A, B, C, D) as sum(sum(A, B), sum(C, D))) and then if there are sufficiently many copies, it becomes impossible to corrupt them all at once. So I don't love this model because escaping corruption is 'too easy'. At the same time, reality is kind of cellular-automata-like. Both QFT and GR posit that the world is made of fields that interact only locally, which is ~the same as positing the world is a cellular automaton with infinitesimally-sized cells. (Sidenote, that's probably why Stephen Wolfram thinks the world is automatons, I'm coming around.) Alternatively, we could use computational-DAGs as the model, like neurla networks. If you allow nodes to be corrupted but their output has to be bounded, then you can get robustness by having redundancy again. If you allow unbounded corruption, you're sad again. But infinity is fake so this seems fine.

2Richard_Ngo12h

I really like the cellular automaton model. But I don't think it makes escaping corruption easy! Even if most of the copies are non-corrupt, the question is how you can take a "vote" of the corrupt vs non-corrupt copies without making the voting mechanism itself be easily corrupted. That's why I was talking about the non-corrupt copies needing to "overpower" the corrupt copies above.

Adrià Garriga-alonso12hΩ120

No I agree with that. I thought the tree design already involved weighted sums overpowering each other, but I think that was premature.

abramdemski's Shortform

abramdemski13h20

Today's Inkhaven post is an edit to yesterday's, adding more examples of legitimacy-making characteristics, so I'm posting it in shortform so that I can link it separately:

Here are some potential legitimacy-relevant characteristics:

The reasoning is logically valid.
The assumptions of the argument are credible (this splits into many characteristics we can name:)
- The assumptions are simple.
- There are few assumptions.
- The assumptions have very few degrees of freedom.
- The assumptions are agreed upon by many humans.
- The assumptions are a strong consensus in the relev

... (read more)

Mikhail Samin's Shortform

Mikhail Samin3d*412

At the beginning of November, I learned about a startup called Red Queen Bio, that automates the development of viruses and related lab equipment. They work together with OpenAI, and OpenAI is their lead investor.

On November 13, they publicly announced their launch:

Today, we are launching Red Queen Bio (http://redqueen.bio), an AI biosecurity company, with a $15M seed led by @OpenAI. Biorisk grows exponentially with AI capabilities. Our mission is to scale biological defenses at the same rate. A on who we are + what we do!
[...]
We also need *financial* co-

... (read more)

Showing 3 of 11 replies (Click to show all)

4Neel Nanda22h

Thanks for sharing, this is extremely important context - I'm way more ok with dual use threats from a company actively trying to reduce bio risk from AI who seem to have vaguely reasonable threat models, than just reckless gain of function people with insane threat models. It's much less clear to me how much risk is ok to accept from projects actively doing reasonable things to make it better, but it's clearly non zero (I don't know if this place is actually doing reasonable things, but Mikhail provides no evidence against) I think it was pretty misleading for Mikhail not to include this context in the original post.

Mikhail Samin14h*40

Uhm yeah valid I guess the issue was illusion of transparency: I mostly copied the original post from my tweet, which was quote-tweeting the announcement, and I didn’t particularly think about adding more context because had it cached that the tweet is fine (I checked with people closely familiar with RQB before tweeting, and it did include all of the context by virtue of quote-tweeting the original announceemnt) and when posting to lw did not realize i'm not directly adding all of the context that was included in the tweet if people don't click on the lin... (read more)

0Kaj_Sotala20h

Agree; I'd also like to emphasize this part: Based on this, they didn't need to set up a new company. They already had an existing biotech company that was focused on its own research, when they realized that "oh fuck, based on our current research things could get really bad unless someone does something"... and then they went Heroic Responsibility and spun out a whole new company to do something, rather than just pretending that no dangers existed or making vague noises and asking for government intervention or something. It feels like being hostile toward them is a bit Copenhagen Ethics, in that if they hadn't tried to do the right thing, it's possible that nobody would have heard about this and things would have been much easier for them. But since they were thinking about their consequences of their research and decided to do something about it and said that in public, they're now getting piled on for not answering every question they're asked on X. (And if I were them, I might also have concluded that the other side is so hostile that every answer might be interpreted in the worst possible light and that it's better not to engage.)

Daniel Kokotajlo's Shortform

Daniel Kokotajlo3d*323

Just finished reading Red Heart by Max Harms. I like it!

Just finished reading Red Heart by Max Harms. I like it!

Dump of my thoughts:

(1) The ending felt too rushed to me. I feel like that's the most interesting part of the story and it all goes by in a chapter. Spoiler warning!

I'm not sure I understand the plot entirely. My current understanding is: Li Fang was basically on a path to become God-Emperor because Yunna was corrigible to him and superior to all rival AIs, and the Party wasn't AGI-pilled enough to realize the danger. Li Fang was planning t

Daniel Kokotajlo14h20

Thanks for the reply! I'm afraid I haven't read Crystal Society, but on that recommendation I will.

I still think it would be great if you wrote more content of the sort I'm asking for. Put it this way: I imagine a bunch of readers will bounce off the ending, having a reaction "and then things stopped making sense, there was a power struggle over who the AI should be most loyal to and as a result the AI just sorta snapped and took over the world and was very bad for no reason. I feel like that's when it went from hard sci-fi to soft sci-fi, or worse,

... (read more)

4Daniel Kokotajlo2d

Fixed thanks!

10Max Harms3d

I'm Max Harms, and I endorse this interpretation. :)

Elizabeth's Shortform

Elizabeth2d*11943

REASONS BESIDES JEALOUSY TO NOT BE POLYAMOROUS

Recently Amanda Bethlehem published comparing monogamous jealousy to kidney disease. Eneasz Brodski doubled down on this. I disagree with a lot of their implications, but today I'm going to focus on the implicit claim that jealousy is the only reason to be monogamous. Here is a list of other reasons you might choose monogamy:

Your sex and romance needs are satisfied by one person, and extra isn't that valuable to you (aka "polysaturated at 1", or in the case of one guy I know, at 0)
You + your partner are c

... (read more)

Showing 3 of 18 replies (Click to show all)

1Tapatakt15h

OK, that's a misunderstanding. By "in the same category" I basically meant "both are OK". Like, "play boardgames with friends" is kinda obviously bad place for a relationship boundry (in general, by default, yes, we probably can invent some far-fetched scenario) and for me being poly is first of all that I and my partner treat "dating/being romantically involved/having sex with someone else" also as a bad place for a relationship boundry. If I didn't want to play boardgames with anyone else, I would still think that forbidding my partner to play boardgames with anyone else is Not OK. And if I didn't want to date/be romantically involved/have sex with anyone else, I would still be poly. And there are possible relationship boundaries around other partners that I think are OK, even some we don't practice. But they are kinda "positive" and not "negative" boundaries. Like, "you have to give me X", not "you have to not give X to anyone else". Does it make sense? (Also, yes, I'm sure some people try to be poly when it doesn't actually work for them, but I think a lot more people try to be mono when it doesn't actually work for them. But that's offtopic.)

1speck144715h

Sure, this seems more plausible. I'm sure I'd still object to your understanding of some moral and practical dimensions of monogamy, but I'm also sure you're aware of that so talking about it is unlikely to be productive for either of us. I'd ask that you reconsider the use of the word "category" if you have this discussion with others in the future, this is just not what it means.

Tapatakt15h20

Yes, sure! That comment was not very thoughtful.

boazbarak's Shortform

boazbarak6d91

I really liked Scott Alexander's post on AI consciousness. He doesn't really discuss the paper of Butlin et al which is fine by me. I can never get excited about consciousness research. It always seemed like drawing the target around the dart, where the dart is humans, and you're trying (not always successfully) to make a definition that includes all humans but doesn't include thermostats. I can't get that excited about whether or not LLM have the "something something feedback" or whether having chain of thoughts changes anything.

Maybe, like Scott says, it... (read more)

Showing 3 of 12 replies (Click to show all)

3green_leaf18h

So, to give a specific example, if you considered a mind upload with conscious states identical to yours (but not embodied), it's possible it would be morally permissible for (embodied) humans to torture it for fun?

boazbarak15h20

Not a big believer in hypotheticals. Mind uploading gets into some very weird issues. I will leave it to decide to future society when and if that happens. I would say that if people are torturing anything for fun, even if that thing has no capacity for pain, then that doesn’t sound morally good to me.

12Vladimir_Nesov5d

There might be a limit on how much it should deviate, but not on how much it can, because the initial conditions for values-on-reflection can be constructed so that the eventually revealed values-on-reflection are arbitrarily weird and out-of-place, which is orthogonality-in-principle (as opposed to orthogonality-in-practice, what kinds of values empirically tend to arise, from real world processes of constructing things with values). Moral considerations that move me, specifically, don't need to follow the normality-of-today, and I'm radically uncertain about what cosmic normality looks like, which is closer to a framing where normative anchors should be found. I'm radically uncertain about normativity, and the practical morality of today doesn't much help with figuring out how to think about it. Something radically uncertain doesn't have the legibility to move me in practice, but retains influence in case it gets more legible. So maybe I'm talking about a further distinction between morality-in-practice that should anchor to the practical attitudes of today, and morality-in-principle, that's not particularly moved by what's going on in the current world, but urges normative caution about what kinds of actions shouldn't be currently taken, that plausibly massively pessimize whatever (my) morality-in-principle turns out to be eventually. Not creating new kinds of thinking beings for now seems safe, and similarly for treating what beings do get created (which still shouldn't be too numerous or influential) as well as any other people. Regardless of Turing tests, LLMs can't currently maintain coherent strivings for particular long term real world outcomes in a situationally aware way, possibly continual learning is sufficient to change this (even if it doesn't yet make them intellectual peers to humanity in the practical sense). If these strivings (when they become coherent) are systematically rebuffed, or the AIs are forcibly reshaped to have different strivings (

Simon Lermen's Shortform

Simon Lermen15h20

I read this older post by Nate Soares from 2023, AI as a Science, and Three Obstacles to Alignment Strategies, a pretty prescient overview of challenges in alignment research.

Alignment is difficult because (1) alignment and capabilities are intertwined (alignment research helping capabilities), (2) we don't have a process to verify what good ideas or progress look like, and we likely get (3) only one critical try. He already addresses many of the counterarguments that are getting brought up recently.

(1) Without any strong governance, a lot of alignment wor... (read more)

Shiva's Right Foot's Shortform

Shiva's Right Foot2d40

Reddit Mod is a "job" AI will likely replace.

There currently is a kerfuffle over the actions of a moderator of r/art which seem to be a shining example of the petty and arbitrary cruelty in which Reddit mods are free to indulge as unpaid volunteers. This made me think that Reddit Mods are very similar to Dungeon Masters in Hasbro's Dungeons and Dragons game, although why may not be clear to you readers.

Hasbro's DnD seems to be on a mission to alienate DMs from DnD in order to more directly access the players, which Hasbro llikely views as their real custom... (read more)

Gavin Runeblade15h30

" Recently there has been rumors of a move to disallow the DM screen, a folding barrier which allowed the DM to conceal things like maps of areas yet unseen by players or detailed character sheets for NPCs. "

Interestingly, co-creator of D&D Dave Arnesson experimented with a DM screen so big he was invisible to the players to see if that increased their immersion. He saw it as a tool for the players. He also didn't always give players their character sheets, for the same reason, always looking for ways to make the player have a better experience. ... (read more)

1cdt2d

Maybe, but it suffers from both ends of the legitimacy problem. At one extreme, some people will never accept a judgement from an LLM as legitimate. At the other extreme, people will perceive LLMs as being "more than impartial" when, in truth, they are a different kind of arbitrary.

Tapatakt's Shortform

Tapatakt17h30

I just discovered that I apparently independently invented already existing rephrasing of smoking lesion problem with toxoplasmosis. This is funny.

williawa's Shortform

williawa2d40

Do anyone have a strong takes about the probability Chinese labs are attempting to mislead western researchers by publishing misleading arguments/papers?

The thought occured to me because jus a few hours ago deepseek released this https://github.com/deepseek-ai/DeepSeek-Math-V2/blob/main/DeepSeekMath_V2.pdf and when they make releases of this type, I typically drop everything to read them, because so much of what frontier labs do is closed, but deepseek publishes details, which are gems that update my understanding.

But I worry somewhat that they could... (read more)

1Martin Vlach19h

Are we on the same board with current evidence of DS_Math_v1 was 99% honest and immediately useful drop while v2 we are only around 90% sure on this?

williawa19h10

That sounds about right to me.

Martin Vlach's Shortform

Martin Vlach19h10

Talking to chatbots with curiosity, ›What they do here?‹ is an ablation for evaluations/benchmarking.

AugBro's Shortform

AugBro1d10

My main disagreement with Futurism or Transhumanism is that technological progress does not correlate strongly with satisfaction and happiness. And if there isn't a strong correlation, then, what's the point? Every additional technological step is another faustian bargain.

I guess I would just like to observe that silicon valleys innovation mindset of relentlessly questioning assumptions does not allow for the questioning of itself. If you disagree, kindly direct me!

This is my first post and my test of the waters so to speak. Hope to discuss.

Tim Hua's Shortform

Tim Hua4dΩ1120-1

Thoughts On Evaluation Awareness in Claude Opus 4.5.

Context:

Anthropic released Claude Opus 4.5 earlier today (model card). Opus 4.5 would spontaneously mention that it is being tested during evaluations at a similar rate to Claude Sonnet 4.5, but lower than Haiku 4.5 (pg. 65).

Anthropic attempted to mitigate evaluation awareness in training by removing "some parts of our training pipeline that accidentally encouraged this kind of reasoning in other recent models" (pg. 65). The model card later mentioned that Sonnet 4.5 was trained on "prompts th... (read more)

Showing 3 of 5 replies (Click to show all)

1Jannes Elstner2d

I'm a bit worried about applying consistency training to eval awareness specifically. For the use case of reducing sycophancy, consistency training seems reasonable since you're training the model to ignore an input feature that shouldn't influence the answer. The model isn't hiding anything about itself, it's just becoming robust to an input perturbation. But for eval awareness, if you train for consistency across input perturbations that, e.g. contain "you're in an evaluation", or something like "this is a hypothetical scenario", and train the model to produce outputs as if it wasn't there (bias-augmented consistency training), the model has to: 1. See the eval hint 2. Not mention it in the CoT 3. Produce reasoning that looks like it never saw it But the model did see it. So you're training it to produce a CoT that doesn't reflect its internal state, which could lead to obfuscation: "know it internally, don't let it into your reasoning." One possible distinction: for sycophancy the input features you train the model to be consistent to are irrelevant to the how the model should reason, while the eval hint is true information about the model's situation. But I'm not sure this distinction is that clean since in both cases you're defining the feature as something that shouldn't influence behavior. How consistency training generalizes is what worries me most. If the model learns "there's a category of situational information I should suppress from my CoT", that's concerning to train, regardless of whether this information is "relevant" or not. Learning to hide what you know from your reasoning seems bad. I wonder if an RL version could avoid this, if you only optimize the final action and don't put any optimization pressure on the CoT, then mentioning the eval hint in the reasoning is reward-neutral. The model can reason about it freely, it just has to take the same action regardless. So there wouldn't be direct pressure to suppress information from the CoT,

1Tim Hua2d

Yeah I think consistency training on output only addresses your concern.

Jannes Elstner1d10

I think such consistency training on outputs would result in a model that's basically always eval aware on your training distribution, and after that point your consistency training has no gradient. Then you have this super eval-aware model, run it on your (held-out, OOD) evaluations, and hope that in those it's just eval aware enough so you can make conclusions about the model behaving the same if it were in an eval or not.
Is this the intent, or do you have a specific training method in mind that avoids this?

[This comment is no longer endorsed by its author]Reply

Jesper L.'s Shortform

Jesper L.2d1-1

"Common sense" advice that is highly rational and easily overlooked by A LOT of people, including smart people

This is your regular reminder.

1. Buy fire alarms, a fire blanket, a strong light torch for blackouts, and a first aid kit (this is the bare minimum low-cost emergency kit that can save you a lot of hassle and potentially your life).

1.2 Do NOT have heavy power-draining appliances connected via extension cord. Fridges and washing machines should never sit in an extension cord unless you know what you are doing and that it can handle the strain.... (read more)

Showing 3 of 6 replies (Click to show all)

2habryka2d

Come on, if you want to argue the fire death point at least give some kind of statistic or do a micromort estimate. Most people do not own fire blankets, as such there is little survivorship bias going on here. You can just estimate using base rates. The expected annual property damage from fire is around $60/year per homeowner per this random ChatGPT analysis (in other words not worrying about). A fire blanket would need to result in a 50% reduction of all fire risk to start being worth the cost and attention. Honestly, this whole conversation just feels like I am on Reddit with people giving random anecdotes without statistical literacy. You can disagree with me, but you speak with weird authority on issues that you seem to not have actually thought that clearly about.

Jesper L.1d10

Thinking about this much harder, I would rewrite the post to clearly phrase it as damage prevention you can do fast and cheap from your home.

I notice I forgot to mention a first aid kit. First thing to correct.

Secondly, when thinking hard on what of your feedback might actually be useful, I realized that statistically speaking you may do more good by buying slip mats for wet floors, and potentially investing in premium ladders. Since fall accidents dominate injury statistics. Not something I thought about intuitively but makes sense.

https://injuryfacts.nsc... (read more)

-10Jesper L.2d