LESSWRONG
LW

Oliver Sourbut
1023Ω120202380
Message
Dialogue
Subscribe

oliversourbut.net

  • Autonomous Systems @ UK AI Safety Institute (AISI)
  • DPhil AI Safety @ Oxford (Hertford college, CS dept, AIMS CDT)
  • Former senior data scientist and software engineer + SERI MATS

I'm particularly interested in sustainable collaboration and the long-term future of value. I'd love to contribute to a safer and more prosperous future with AI! Always interested in discussions about axiology, x-risks, s-risks.

I enjoy meeting new perspectives and growing my understanding of the world and the people in it. I also love to read - let me know your suggestions! In no particular order, here are some I've enjoyed recently

  • Ord - The Precipice
  • Pearl - The Book of Why
  • Bostrom - Superintelligence
  • McCall Smith - The No. 1 Ladies' Detective Agency (and series)
  • Melville - Moby-Dick
  • Abelson & Sussman - Structure and Interpretation of Computer Programs
  • Stross - Accelerando
  • Graeme - The Rosie Project (and trilogy)

Cooperative gaming is a relatively recent but fruitful interest for me. Here are some of my favourites

  • Hanabi (can't recommend enough; try it out!)
  • Pandemic (ironic at time of writing...)
  • Dungeons and Dragons (I DM a bit and it keeps me on my creative toes)
  • Overcooked (my partner and I enjoy the foody themes and frantic realtime coordination playing this)

People who've got to know me only recently are sometimes surprised to learn that I'm a pretty handy trumpeter and hornist.

Sequences

Posts

Sorted by New

Wikitag Contributions

Comments

Sorted by
Newest
Breaking Down Goal-Directed Behaviour
No wikitag contributions to display.
Hierarchical Agency: A Missing Piece in AI Alignment
Oliver Sourbut20d20

Nice! I'm very late to this. A few thoughts.

Focusing on 'full agents' might be misleading. Humans are fairly agenty as far as things go. Mobs and corporations and countries are a bit less agenty on the whole. So maybe you need some spectrum (or space) of agentness for things to be appropriately similarly-typed.

Game theory and MDPs and so on often treat the population as static. But we spawn (and dissolve) actors all the time. So a full theory here would need to build that in from the start. This has the nice property that it might be a foundation to describe a hierarchy too: mobs, corporations, and countries get spawned, dissolved, transformed, etc. all the time, after all. (Looser thought: could some agents-composed-of-parts be 'suspended' or 'transferred' to different substrate...?)

Reply
adamzerner's Shortform
Oliver Sourbut1mo20

(Maybe you're looking for the word 'seasoning'...? But maybe that includes other herbs in a way you didn't want.)

Reply
Will compute bottlenecks prevent a software intelligence explosion?
Oliver Sourbut1mo20

With that weighting, human experts have much less frontier-taste-relevant data than they have total data... AI can learn from all copies.

Ah, yep yep. (Though humans do in fact learn (with losses) from other copies, and within a given firm/lab/community the transfer is probably quite high.)

Hmm, I think we have basically the same model, with maybe a bit different parameters (about which we're both somewhat uncertain). But think that readers of the OP without some of that common model might be misled.

From a lot of convos and writing, I infer that many people conflate a lot of aspects of intelligence into one thing. With that view, 'intelligence explosion' is just a dynamic where a single variable, 'the intelligence' (or 'the algorithmic quality'), gets real big real fast. And then of course, because intelligence is the thing that gets you new technology, you can get all the new technology.

About this, you said,

There is then a further question, if that IE goes very far, of whether AI will generalize to chemistry etc

I'd have thought "yes" bc you achieve somewhat superhuman sample efficiency and can quickly get as much chemistry exp as a human expert

revealing that you correctly distinguish different factors of intelligence like 'sample efficiency' and 'chemistry knowledge' (I think I already knew this but it's good to have local confirmation), and that you don't think a software-only IE yields all of them.

Regarding the second sentence, it could be a misleading use of terms to call that 'generalisation'[1], but I agree that 'sample efficiency' is among the relevant aspects of intelligence (and is a candidate for one that could be mostly generalisably built up automated in silico), and a relevant complement is 'chemistry (frontier research) experience', and that a lot of each taken together may effectively get you chemistry research taste in addition (which can yield new chemistry knowledge if applied with suitable experimental apparatus).

I'm emphasising[2] (in my exploration post, in this thread, and the sister comment about frontier taste depreciation) that there's practically a wide gulf between 'hoover taste up from web data' and 'robotics or humans-with-headsets', in two ways. The first tops out somewhere (probably at sub-frontier) due to the depreciation of frontier research taste. The second cluster doesn't top out anywhere, but is slower and has more barriers to getting started. Is a year to exceed humanity's peak taste bold in most domains? Not sure! If a lot of in silico is possible, maybe it's doable. That might include cyber and software (and maybe narrow particular areas of chem/bio where simulation is especially good).

If you know for sure that those other bottlenecks proceed super fast, you don't need to necessarily clarify what intelligence factors you're talking about for practical purposes, but if you're not sure, I think it's worth being super clear about it where possible.


  1. I might instead prefer terminology like 'quickly implies ... (on assumptions...)' ↩︎

  2. Incidentally, the other thing I'm emphasising (but I think you'd agree?) is that on this view, R&Ds are always substantially driven by experimental throughput, with 'sample efficiency (of the combined workforce) at accruing research taste' being the main other rate-determining factor (because the steady state of research taste depends on this, and exploration quality * experimental throughput is progress). Throwing more labour at it can make your serial experimentation a bit faster, and can parallelise experimentation (with some parallelism discount), with presumably very diminishing returns. Throwing smarter labour (as in, better sample efficiency, and maybe faster-thinking, with diminishing returns), can increase the rate, by getting more insight per experiment and choosing better experiments. ↩︎

Reply
Will compute bottlenecks prevent a software intelligence explosion?
Oliver Sourbut1mo20

Yes more 'data', but not necessarily more frontier-taste-relevant experience, which requires either direct or indirect contact with frontier experiments. That might be a crux.

Reply
Will compute bottlenecks prevent a software intelligence explosion?
Oliver Sourbut1mo40

BTW I also think novelty taste depreciates quickly as the frontier of a domain moves, so I'm less bullish on hoovering up taste from existing data/interviews for a permanent taste boost. But it might take you some way past frontier, which may or may not be sizeably impactful in a given domain.

Reply
Will compute bottlenecks prevent a software intelligence explosion?
Oliver Sourbut1mo20

I think we're on the same page about what factors exist.

For the intelligence explosion I think all we need is experience with AI RnD?

Well, sort of, but precisely because intelligence isn't single-dimensional, we have to ask 'what is exploding here?'. And like I said, I think you straightforwardly get speed. And plausibly you get sample efficiency (in the inference-from-given-data sense). And maybe you get 'exploratory planning', which is hypothetically one, more transferrable, factor of general exploration efficiency.

But you don't get domain-specific novelty/interestingness taste, which is another critical factor, except for the domains you can easily get lots of data on. So either you need to hoover that up from existing data, or get it some other how. That might be interviews, it might be a bunch of robotics and other actuator/sensor equipment, it might be the Shulman-style humans-with-headsets-as-actuators thing and learning by experience. But then the question becomes whether there's enough data to hoover it up real fast with your speed+efficiency learning, or whether you hit a bunch of scaling up bottlenecks (and in which domains).

Reply
Human takeover might be worse than AI takeover
Oliver Sourbut1mo60

I am hopeful that one of the things we can do with just-before-the-brink AI will be to accelerate the design and deployment of such voluntary coordination contracts. Could we manage to use AI to speed-run the invention and deployment of such subsidiarity governance systems? I think the biggest challenge to this is how fast it would need to move in order to take effect in time. For a system that needs extremely broad buy-in from a large number of heterogenous actors, speed of implementation and adoption is a key weak point.

FYI FLF has a Fellowship on AI for Human Reasoning which centrally targets objectives like this (if I've understood)

Reply1
Will compute bottlenecks prevent a software intelligence explosion?
Oliver Sourbut1mo20

I wrote a bit about experimentation recently.

You seem very close to taking this position seriously when you talk about frontier experiments and experiments in general, but I think you also need to notice that experiments come in more dimensions than that. Like, you don't learn how to be better at chemistry just by playing around with GPUs.

Reply
Will compute bottlenecks prevent a software intelligence explosion?
Oliver Sourbut1mo40

It's quite clear that labs both want more high-quality researchers -- top talent has very high salaries, reflecting large marginal value-add.

Three objections, one obvious. I'll state them strongly, a bit devil's advocate; not sure where I actually land on these things.

Obvious: salaries aren't that high.

Also, I model a large part of the value to companies of legible credentialed talent being the marketing value to VCs and investors, who (even if lab leadership can) can't tell talent apart except by (rare) legible signs. This is actually a way to get more compute (and other capital). (The legible signs are rare because compute is a bottleneck! So a Matthew effect pertains.)

Finally, the utility of labs is very convex in the production of AI: the actual profit comes from time spent selling a non-commoditised frontier offering at large margin. So small AI production speed gains translate into large profit gains.

Reply
Will compute bottlenecks prevent a software intelligence explosion?
Oliver Sourbut1mo20

The best objection to an SIE

 

I think the compute bottleneck is a reasonable objection, but there is also the fairly straightforward objection that gaining skills takes experience, and experience takes interaction (or hoovering up from web data).

You can get experience in things like 'writing fast code' easily, so a speed explosion is fairly plausible (up to diminishing returns). But various R&Ds or human influence or whatever are much harder to get experience for. So our exploded SIs might be super fast and maybe super good at learning from experience, but out of the gate at best human expert level where relevant data are abundant, and at best human novice level where data aren't.

Reply
Load More
6Oliver Sourbut's Shortform
3y
3
20You Can’t Skip Exploration: Why understanding experimentation and taste is key to understanding AI
2mo
0
75FLF Fellowship on AI for Human Reasoning: $25-50k, 12 weeks
2mo
1
26Deceptive Alignment and Homuncularity
6mo
12
8Cooperation and Alignment in Delegation Games: You Need Both!
1y
0
17Terminology: <something>-ware for ML?
Q
2y
Q
27
6Alignment, conflict, powerseeking
2y
1
16Careless talk on US-China AI competition? (and criticism of CAIS coverage)
2y
3
12Invading Australia (Endless Formerlies Most Beautiful, or What I Learned On My Holiday)
2y
1
28Hertford, Sourbut (rationality lessons from University Challenge)
2y
7
26Un-unpluggability - can't we just unplug it?
2y
10
Load More