Since the post by Ryan Greenblatt I've been hoping to see similar, expanded work. I think it's an interesting thing to measure, since it can plausibly have large effects on the frequency of minor (though sometimes significant) mistakes.
It's also interesting how this seems to break down for certain types of reasoning and retrieval. As Greenblatt showed in another post, models are really bad at retrieving several facts required to compute something in one forward pass, even though the required tokens to turn the failures into successes are usually much lowe... (read more)
Since the post by Ryan Greenblatt I've been hoping to see similar, expanded work. I think it's an interesting thing to measure, since it can plausibly have large effects on the frequency of minor (though sometimes significant) mistakes.
It's also interesting how this seems to break down for certain types of reasoning and retrieval. As Greenblatt showed in another post, models are really bad at retrieving several facts required to compute something in one forward pass, even though the required tokens to turn the failures into successes are usually much lowe... (read more)