Very interesting! I think this is one of the rare times where I feel like a post would benefit from an up-front Definition. What actually is Leakage, by intensional definition?

This is one technical point that younger people are often amazed to hear, that for a long time the overwhelming majority of TV broadcast was perfectly ephemeral, producing no records at all. Not just that the original copies were lost or never digitised or impossible to track down or whatever, but that nothing of the sort ever existed. The technology for capturing, broadcasting, and displaying a TV signal is so much easier than the tech for recording one, that there were several decades when the only recordings of TV came from someone setting up a literal film camera pointed at a TV and capturing the screen on photographic film, and that didn't happen much.

(This also meant that old TV was amazingly low latency. The camera sensor scanned through, producing the signal, which went through some analog circuitry and straight onto the air, into the circuitry of your TV and right onto the screen. The scanning of the electron beam across the screen was synchronised with the scanning of the camera sensor. At no point was even a single frame stored - I need to check the numbers but I think if you were close to the TV station, you're looking at the top of the frame before the bottom of the frame is even captured by the camera)

What about NMR or XRF? XRF can non-destructively tell you the elemental composition of a sample, which (if the sample is pure) can often pin down the compound, and NMR spectroscopy is also non destructive and can give you some info about chemical structure too

This is an interesting post!

I'm new to alignment research - any tips on how to prove what the inner goal actually is?

Haha! haaaaa 😢

Not least being the military implications. If you have widely available tech that lets you quickly and cheaply accelerate something car-sized to a velocity of Mach Fuck (they're meant to circle the earth in 4.2 hours, making them 2 or 3 times faster than a rifle bullet), that's certainly a dual use technology.

Covid was a big learning experience for me, but I'd like to think about more than one example. Covid is interesting because, compared to my examples of birth control and animal-free meat, it seems like with covid humanity smashed the technical problem out of the park, but still overall failed by my lights because of the political situation.

How likely does it seem that we could get full marks on solving alignment but still fail due to politics? I tend to think of building a properly aligned AGI as a straightforward win condition, but that's not a very deeply considered view. I guess we could solve it on a whiteboard somewhere but for political reasons it doesn't get implemented in time?

I think almost all of these are things that I'd only think after I'd already noticed confusion, and most are things I'd never say in my head anyway. A little way into the list I thought "Wait, did he just ask ChatGPT for different ways to say "I'm confused"?".

I expect there are things that pop up in my inner monologue when I'm confused about something, that I wouldn't notice, and it would be very useful to have a list of such phrases, but your list contains ~none of them.

Edit: Actually the last three are reasonable. Are they human written?

One way of framing the difficulty with the lanternflies thing is that the question straddles the is-ought gap. It decomposes pretty cleanly into two questions: "What states of the universe are likely to result from me killing vs not killing lanternflies" (about which Bayes Rule fully applies and is enormously useful), and "Which states of the universe do I prefer?", where the only evidence you have will come from things like introspection about your own moral intuitions and values. Your values are also a fact about the universe, because you are part of the universe, so Bayes still applies I guess, but it's quite a different question to think about.
If you have well defined values, for example some function from states (or histories) of the universe to real numbers, such that larger numbers represent universe states that you would always prefer over smaller numbers, then every "should I do X or Y" question has an answer in terms of those values. In practice we'll never have that, but still it's worth thinking separately about "What are the expected consequences of the proposed policy?" and "What consequences do I want", which a 'should' question implicitly mixes together.

I've always thought of it like, it doesn't rely on the universe being computable, just on the universe having a computable approximation. So if the universe is computable, SI does perfectly, if it's not, SI does as well as any algorithm could hope to.

A slightly surreal experience to read a post saying something I was just tweeting about, written by a username that could plausibly be mine.