Note: We at PrincInt are launching a program teaching “Discernment” or “Taste” and generally spend a lot of our time in workshops and otherwise trying to teach academics how to think about AI such that their research ends up being useful. I had used a metaphor (building on Wentworth’s postmortem) multiple times to explain what we do and why, and thought that it could be useful more broadly. All views are my own, and strategy takes do not represent everyone at PrincInt etc. AI writing level 0 (no non-trivial use)

When I first joined the field, I told 80,000 hours that I have a background in Economics, almost a decade as a professor, and connections from my time in China. They said something like “all of that is useless - you have 10 years running NGOs, we need Ops people, do that”. At the time, they were right! In 2021-22 there was little going on in either Economics or China, but if I was joining now it would be completely different - I’d have orgs to join, or if I tried fundraising for something in those fields, it would now be much easier. I would not have to explain why China is important, or that Economics has something interesting to say, the path has been threaded, and light shined on an issue. This is what it feels like to watch a Streetlight get built over a problem area.

The Puzzle Planet

The way I imagine the alignment problem recently is thus: picture an obviously constructed planet with continent arrangement broadly representing our scientific fields - perhaps the economics part of it is closer to sociology and further from astrophysics. The planet is currently hurtling towards Earth, and we don’t know when exactly it’ll hit or if it’ll change behavior once it reaches us.

We send an exploratory mission to the surface, and we find that scattered over the planet are numerous locks and keys; some ornate, some plain, some open multiple locks, and others are themselves useless or open locks with nothing behind them. Some locks hide further keys, or hints to locations of keys and locks. 

There is a lot of discussion about what should be done, and which path to take. Some say we should look for keys which get us to the core of the planet, where we can set explosives; others say we should look for keys which allow us entrance to a “command room” - third still think neither of these places exist and we should nuke the planet on approach, despite its intriguing nature and possible incredible wealth hidden beneath the surface.

The debate on “What to do” and “Which keys give us what we want” are of the shape “These keys are red herrings, and these are True Keys”; discussion of which are which is beyond the scope of this story - but you can say “keys that you pursue aren’t True Keys in your path” or “I think this key is the True Key in most of paths we may want to take”. Our own area of expertise dictates which keys or locks we’re more able to find, and to recognise.

Bring the Light

The main issue with looking for keys is lighting. Space is dark, and it is hard looking for keys on the planet if there is no light, even within your area of expertise. Some Streetlights have already been built - either through funders who make them or through some parts of the planet already having them - and looking under the Streetlight is much easier, and that is where a lot of our effort as a field is concentrated. When field-building organizations bring people into the field, naturally, people tend to gravitate towards doing things in the Streetlights, and (in my view, unfortunately) organizations themselves tend to get built around well defined problems with lots of funding, i.e., around Streetlights.

Some groups give people Torches. Torches can be taken with you and allow you to look at the narrow part of terrain that you are on, but give lower light than a streetlight would. PrincInt has historically looked for explorers and armed them with Torches in our Fellowship Program and old Affiliate program. Torches are quite useful - they allow us to explore otherwise harder parts of the planet, and if something is found we can tell others and build Streetlights there. This process has happened with Simplex for example. Other organizations hand out Torches in particular parts of the field, AFFINE is doing it for Agent Foundations for example, and Iliad is doing it in Mathematics. PrincInt’s expertise is in doing it everywhere (where others are not), and I’d argue we’ve done a great job with statistical physics for example.

But there is another thing you can have instead. To be able to decide where to give out Torches, you need Nightvision Goggles. Nightvision allows you a super-low-res way of scanning vast swathes of land and saying “True Keys for no path are located in this part of the planet, despite having many Streetlights”; or “These bushes have many True Keys for Path A, someone should look into it with Torches”. They allow you to look for patterns in the planet, usually building up a pattern-matching which allows you to make predictions about fruitfulness of different areas.

This is essentially “Taste” or "Discernment" and in a lot of our scaling plans (more fellowships, more programs), the bottleneck we hit is having enough people with Nightvision. While Torches and Streetlights can be kind of “handed on” (i.e. move funding from one person to the next, or one area to the next), Nightvision Goggles are custom, and work only for one user. If they want to pass them on, they need to make new ones for the next person. This makes them scarce, and a field-wide bottleneck. We are far from first to identify it, and the difficulty of training it.

As an attempted solution, we plan to work on building the capacity up through our Discernment Development Project (to be announced), a sort of EMBA-style series of workshops and seminars aimed at developing Nightvision Goggles at some sort of scale (faster than the current “spend many years thinking about the problem and become enlightened”). We are uncertain that we have found the best style and are open to inputs.

If you look at Streetlights, you can look and say “Wow, so many people searching, finding so many keys”, but if you look at the whole planet, it is still very dark, and ominous, and our progress towards actually useful keys seems much more contained. I think this intuition matters a lot, and I wish more people had it, especially people with decision-making power.

So I would love it if Nightvision Goggles were possessed by important decision-makers, inside and outside of our field! Funders, AI Safety lab leaders, AI company leaders and employees, journal committees, government officials, religious leaders, investors, and anyone else with a lever on our search efforts or decisionmaking on where Streetlights get built - or indeed, with a foot on the gas controlling the speed of the puzzle planet flying towards earth - I want all of them equipped with this to both see how to direct others, and to see how much we still do not know. 

If you’ll allow for a bit of sentimentality, I wish that you too, the person reading this, had better Nightvision, if nothing else than to help others see better as well. I also wish it for myself, but alas, it is slow! Best as I can tell, Nightvision is intuitive (i.e. fuzzy/closer to wisdom than knowledge) in some ways and has to be built up over time. Most people whose taste/takes I trust developed it over a long amount of time, reading a lot, touching the problem, trying different things, until they connected enough different problem understandings to see the puzzle-planet patterns well enough, at least in a field of their expertise. If people have recommendations for faster ways, please reach out so that we can improve our efforts of constructing a curriculum to build up Nightvision!

So, how do we find True Keys?

In this metaphor, it is clear - people who are in charge of handing out Torches or Streetlamp building projects must have Nightvision Goggles, or work hard on making sure they have it enough to make good decisions. But the dark is fickle, and fear has big eyes, so sometimes we make Streetlamps around a false key or just a single True Key with nothing next to it; other times wisps distract us and we fail to find fruitful areas even as we pass them. Patterns appear that seem clear, and if only we looked further we would have seen them to be wrong. Locks seem important but we find that they only work once, or lead to dead ends.

A lot of my criticisms of the field are of the sort where people come with a research paper and say “Look, a key, isn’t this progress” but the painful objection is “is the key useful?” and often it is not. We often reward organizations that find keys without ensuring they are True Keys which makes for adverse incentives. Additionally, sending more people to look under Streetlights is really not great, it is getting crowded there, and the number of keys found there give false confidence and hope.

I also wish that more intentional effort went into building up new fields of research and making the process of building Streetlights easier - but people interpret that as a desire for “looking in the dark qua looking in the dark”. Instead I wish for people with Nightvision to be able to more freely give out Torches and to tighten the Torches-Streetlights-Digging loop. Funders, for their part, did diversify bets and we’re getting better at it, but the “build Nightvision” project is yet to come alive, Nightvision Goggles as a skill are underappreciated compared to “can find keys” skill, and our Streetlights, all neatly in a row blind us to how plentiful the locks and keys are in the dark.

So my advice to seekers of True Keys is thus: look for broader visions, understand where you are looking and what for, think about what locks your keys may open and whether those lead to many new fields to explore or simply to things that are easy to illuminate with existing lights. If you do work under someone’s Streetlight, try to learn more abstract things - why were Keys here? Why did they work? What patterns does this follow, and in what direction does that takes us? And have Agency to walk out of the light following your Taste if that is where evidence points to.

Think about whether your work is doing what is easy or popular or if it is doing the hard work. Think if you are building Streetlights in the wrong places. Think about people guiding you and their Nightvision Goggles power. Figures like Wentworth, Yudkowsky, Wei Dai, Soares, Tsvi and recently Ngo I see as pointing to the vastness of the dark areas and saying “There are important things here which are out of Spotlights and we need to spend time looking there with Nightvision Goggles, Torches and eventually build Spotlights in important areas.”

I hear the counter which was brought up before that “Streetlighting is just a good way to sort by tractability” but that’s like sorting your nutrition by how easy it is to swallow - unlikely to be the right way to feed yourself. I also understand that people doing work under Streetlights are building up experience which they can later use with Torches - cool if true, but often, like moths, we stay comfortably in the light. There is also the legitimate counter that many wrong things exist, and lots of the darkness do not contain anything useful - but I would say that Nightvision should help minimize the effort wasted, and I think it unlikely that all the needed work is already conveniently lit.

On our end, PrincInt is organizing Coordinated Research Projects - a kind of Nightvision-Torch-and-Streetlight building project with focus areas in different parts of the planet, that (best as we can see) connect patterns that seem likely to drive us in the right direction. Stay tuned for that, and our Nightvision Goggles manufacturing project!

With thanks to Joe Collman for listening to this metaphor over half a dozen times in the last few weeks, Lucas T. Lauren G., Mick Z. and Paul C. for constructive feedback, and everyone else who I spoke with in Singapore and Berkeley who added their thoughts.