while I think most people aren't quite tackling this with the particular taste I'd apply, it does sure seem like everyone is working on "do stuff with LLMs" and it's not where the underpicked fruit is

I disagree, I think pretty much nobody is attempting anything useful with LLM-based interfaces. Almost all projects I've seen in the wild are terrible and there are tons of unpicked low-hanging fruits.

I'd been thinking, on and off, about ways to speed up agent-foundations research using LLMs. An LLM-powered exploratory medium for mathematics is one possibility.

A big part of highly theoretical research is flipping between different representations of the problem: viewing it in terms of information theory, in terms of Bayesian probability, in terms of linear algebra; jumping from algebraic expressions to the visualizations of functions or to the nodes-and-edges graphs of the interactions between variables; et cetera.

The key reason behind it is that research heuristics bind to representations. E. g., suppose you're staring at some graph-theory problem. Certain problems of this type are isomorphic to linear-algebra problems, and they may be trivial in linear-algebra terms. But unless you actually project the problem into the linear-algebra ontology, you're not necessarily going to see the trivial solution when staring at the graph-theory representation. (Perhaps the obvious solution is to find the eigenvectors of the adjacency matrix of the graph – but when you're staring at a bunch of nodes connected by edges, that idea isn't obvious in that representation at all.)

This is a bit of a simplified example – the graph theory/linear algebra connection is well-known, so experienced mathematicians may be able to translate between those representations instinctively – but I hope it's illustrative.^[1]

As a different concrete example, consider John Wentworth's Bayes Net Algebra. This is essentially an interface for working with factorizations of joint probability distributions. The nodes-and-edges representation is more intuitive and easy to tinker with than the "formulas" representation, which means that having concrete rules for tinkering with graph representations without committing errors would significantly speed up how quickly you can reason through related math problems. Imagine if the derivation of such frameworks was automated: if you could set up a joint PD in terms of formulas, automatically project the setup into graph terms, start tinkering with it by dragging nodes and edges around, and get errors if and only if back-projecting the changed "graph" representation into the "formulas" representations results in a setup that's non-isomorphic to the initial one.

(See also this video, and the article linked above.)

A related challenge are refactors. E. g., suppose you're staring at some complicated algebraic expression with an infinite sum. It may be the case that a certain no-loss-of-generality change of variables would easily collapse that expression into a Fourier series, or make some Obscure Theorem #418152/Weird Trick #3475 trivially applicable. But unless you happen to be looking at the problem through those lens, you're not going to be able to spot it. (Especially if you don't know the Obscure Theorem #418152/Weird Trick #3475.)

It's plausible that the above two tasks is what 90% of math research consists of (the "normal-science" part of it), in terms of time expenditure. Flipping between representations in search of a representation-chain where every step is trivial.

Those problems would be ameliorated by (1) reducing the friction costs of flipping between representations, and (2) being able to set up automated searches for simplifying refactors of the problem.

Can LLMs help with (1)? Maybe. They can write code and they can, more or less, reason mathematically, as long as you're not asking them for anything creative. One issue is that they're also really sloppy and deceptive when writing proofs... But that problem can potentially be ameliorated by fine-tuning e. g. r1 to justify all its conclusions using rigorous Lean code, which could be passed to automated proof-checkers before being shown to you.^[2]

Can LLMs help with (2)? Maybe. I'm thinking something like the Pantheon interface, where you're working through the problem on your own, and in a side window LLMs offer random ideas regarding how to simplify the problem.

LLMs have bad research taste, which would extend to figuring out what refactorings they should try. But they also have a superhuman breadth of knowledge regarding theorems/math results. A depths-first search might thus be productive here. Most of LLM suggestions would be trash, but as long as complete nonsense is screened off by proof-checkers, and the ideas are represented in a quickly-checkable manner (e. g., equipped with one-sentence summaries), and we're giving LLMs an open-ended task, some results may be useful.

I expect I'd pay $200-$500/month for a working, competently executed tool of this form; even more the more flexible it is. I expect plenty of research mathematicians (not only agent-foundations folks) would, as well. There's a lucrative startup opportunity there.

@johnswentworth, any thoughts?

1. ^{^}

   A more realistic example would concern ansatzes, i. e., various "weird tricks" for working through problems. They likewise bind to representations, such that the idea of using one would only occur to you if you're staring at a specific representation of the problem, and would fail to occur if you're staring at an isomorphic-but-shallowly-different representation.

2. ^{^}

   Or using o3 with a system prompt where you yell at it a lot to produce rigorous Lean code, with a proof-checker that returns errors if it ever uses a placeholder always-passes "sorry" expression. But I don't know whether you can yell at it loudly enough using just the system prompt, and this latest generation of LLMs seems really into Goodharting, so it might straight-up try to exploit bugs in your proof-checker.

Here's some problems I ran into the past week which feel LLM-UI-shaped, though I don't have a specific solution in mind. I make no claims about importance or goodness.

Context (in which I intentionally try to lean in the direction of too much detail rather than too little): I was reading a neuroscience textbook, and it painted a general picture in which neurotransmitters get released into the synaptic cleft, do their thing, and then get reabsorbed from the cleft into the neuron which released them. That seemed a bit suspicious to me, because I had previously done a calculation involving diffusion rate of serotonin, and I had cached that serotonin diffused pretty fast (and therefore would probably clear from the cleft mostly via diffusion). So I hit the LLMs + google to figure it out.

The LLMs just repeated basically the same words which were in the textbook. That's usual LLM behavior. So then I googled diffusion rates for all the major neurotransmitters (and helpfully found a paper which measured diffusion rate in the cleft itself, so I knew roughly how much slower it would be in the cleft compared to free solution). I also googled for some images of synapses to estimate their size (which was surprisingly tricky to find in text form - many sources repeated the distance between the two neurons, but few mentioned the radius of the synapse). Then I did the math, and concluded that, indeed, all of the small molecule neurotransmitters probably clear the cleft mainly via diffusion. (... In the CNS, anyway. Neurumuscular junctions are maybe different, because the synapse radius is bigger.)

This seems like the sort of thing where the right UI/scaffolding combo would make it a lot faster to get the correct info with LLM assistance. But if I just directly ask the question to an LLM in a standard chat interface, it parrots back the thing the textbook says (which is presumably blindly repeated all over the place).

Next up, having established that all the major neurotransmitters clear the cleft mainly via diffusion, I wanted to know how long it takes them to be reabsorbed from the extracellular space outside the cleft. This matters a lot because it determines how localized the signal is, for purposes of receptors outside the cleft itself. This was more difficult to figure out, because lots of papers would offhandedly claim that the reabsorption time was a millisecond or two - a number which happens to roughly match the time it takes small molecule neurotransmitters to diffuse out of the cleft... and therefore the number one would end up with if one incorrectly assumed that the transmitter is mostly cleared from the cleft via reabsorption, and tried to measure the reabsorption time via looking at the time it takes the cleft to clear. And because that number was uninformedly repeated in many papers, the LLM would of course also repeat it often. But when I dug into the papers, it was clearly wrong; proper measurements were scarce, but those I found suggested reabsorption on a slower timescale (and dominated by reabsorption into glia rather than neurons, in many cases).

So it's a similar pattern to the first half of this comment: there's some thing which people will often say, and therefore the LLM will say that thing in response to a question. But if one asks "why do people say this thing?", and looks for object-level evidence, the thing is clearly wrong. It seems like there should be some LLM UI/scaffolding which would make it a lot easier to get the real answers in those cases, but I don't know what that UI/scaffolding would look like.

Another idea I've been thinking about:

Consider the advantage prediction markets have over traditional news. If I want to keep track of some variable X, such as "the amount of investment going into Stargate", and all I have are traditional news, I have to constantly manually sift through all related news reports data in search of related information. With prediction markets, however, I can just bookmark this page and check it periodically.

An issue with prediction markets is that they're not well-organized. You have the tag system, but you don't know which outcomes feed into other events, you don't necessarily know what prompts specific market updates (unless someone mentions that in the comments), you don't have a high-level outline of the ontology of a given domain, etc. Traditional news reports offer some of that, at least: if competently written and truthful, they offer causal models and narratives behind the events.

It would be nice if we could fuse the two. An interface for engaging with the news that combines conciseness of prediction-market updates with an attempt at a model-based understanding offered by traditional news.

One obvious idea is to arrange it into the form of a Bayes net. People (perhaps the site's managers, perhaps anyone) could set up "causal models", in which specific variables are downstream of other variables. Other people (forecasters/experts hired by the project's managers, or anyone, like in prediction markets) could bet on which models are true^[1], and within the models, on the values of specific variables^[2]. (Relevant.)

Among other things, this would ensure built-in "consistency checks". If, within a given model, a variable X is downstream of outcomes A, B, C, such that X only happens if all of A, B, C happen, but the market-estimated P(X) isn't equal to P(ABC), this would suggest either that the prediction markets are screwing up, or that there's something wrong with the given model.

Furthermore, one way for this to gain notoriety/mainstream appeal is if specific high-status people or institutions set up their own "official" causal models. For example, an official AI 2027 causal model, or an official MIRI model of AI doom which avoids the multlple-stage fallacy and clearly shows how it's convergent.

Tons of ways this might not work out, but I think it's an interesting idea to try. (Though maybe it's something that should be lobbed off to Manifold Markets' leadership.)

1. ^{^}

   Or, perhaps in an even more fine-grained manner, which links between different variables are true.

2. ^{^}

   Ideally, with many variables shared between different models.

I think there's a possibility for ui people to make progress on the reputation tracking problem by virtue of tight feedback loops relative to people thinking more abstractly about it. The most rapid period of learning in this regard that I know of is early days at PayPal eBay where they were burning millions a day in fraud at certain points.

Secondly: the chat interface for llm is just bad for power users. Ai Labs is slightly better but still bad.

Edit: meant aistudio