The randomized control trial is a great example where a superintelligence actually could do better by using a non-random strategy. Ideally, an AI could take its whole prior into account and do a value of information calculation. Even if it had no useful prior, that would just mean that any method of choosing is equally "random" under the the AI's knowledge.

This doesn't require omniscience, or AI: people do this now (based on info they have). If you have more info, we know how to use it (there is theory). Why are we talking about AI, this is a math problem.

Slightly harshly worded suggestion (not to you specifically): maybe more reading, less invocation of robo-Jesus in vain.

How are humans supposed to generate cryptographic reward tokens in such a way that an AI could not duplicate the process?

It’s much better to write the piece that you want to read yourself, which usually means pitching the technical content at a level slightly higher than you were comfortable with when you started thinking about it.

Inapplicable when you're writing about natural sciences for layman audience.

Denotational equivalence is actually undecidable for any class of languages stronger than deterministic pushdown automata.

This doesn't mean that we can't obtain certain evidence that two languages/automata are equal/equivalent via some mechanism other than a decision algorithm, of course. It also doesn't mean that we can't assign a probability of equality in an entirely sensible way. In fact, in probabilistic programming, probabilistic extensional equality of random variables is trivial to model: the problem is that you're talking, there, about zero-free-parameter thunks rather than arbitrarily parameterized lambdas.

So we can't really decide the denotational equivalence of lambda expressions (or recurrent neural networks), but I think that decision algorithms aren't really useful, from a cognitive perspective, for more than obtaining a 100% likelihood of equivalence. That's powerful, when you can do it, but you should also be able to get non-100% likelihoods in other ways.

The various forms of probabilistic static analysis can probably handle that problem.

I get why homomorphisms between lambda-expressions are the main homomorphism problem for AGI in general. But for humans, I'd guess that homomorphisms between some drastically simpler subset of models would explain most or all of our abstraction capacity. For example, if humans can find morphisms between causal models, or between spatial models, or between some easily-modeled agents (e.g., agents similar to the human in question), or both, then it feels like that would be sufficient for most of the abstract reasoning which we actually perform.

I really like this as an example of incentive misalignment, and I think it should be in the community's core materials on that subject.

That said, the ideas about healthcare contracts to fix the alignment problem could use some fleshing out. So here's my attempt at fleshing them out.

General properties of the problem:

• We want an open market in healthcare ideas. Anybody should be able to invest in whatever idea they have.

• Ideas which do not work should cost the investor, ideas which do work should reward the investor.

• The ideas must be tried on some actual people. Those people, unavoidably, bear some risk, and must be incentivized to take that risk.

Solution: First, the cost to incentivize people to try ideas must be paid by someone. The investors are a natural choice here. So imagine that the investors offer payment to people to try their ideas. To avoid selection issues, the investors may specify that people must fulfill certain criteria in order to be eligible. Anyone who meets the criteria can sign up for whatever the investors are pushing. This would be sort of like an open market for clinical trials, except that it can scale to any number of people and to long times.

For their investment, the investors would get a contract. Under that contract, they get some regular payments for as long as their participants see the benefits the investors claimed and don't see serious problems. That payment ultimately needs to come from the participants; all participants pay to participate in the market as a whole. If the market is government-managed, then everyone is a participant and their payment comes from taxes.

As an example, consider a cure for prostate cancer. The investors put out their cure at some price, paying people to try their particular idea. Only those with prostate cancer are eligible. All participants get paid by the investors. As soon as a participant's cancer clears, the investors get paid a big chunk of money.Whenever the participant has health issues later in life, the investors have to pay some money back, regardless of what those later issues are. Presumably, more serious issues would be more expensive, and the original contract might specify an extra payment if the prostate cancer returns. (There would be a secondary market for investors to insure against normal later-life health risks, but investors would still need to handle more-than-normal problems, and would get paid for less-than-normal problems).

Now an even more interesting example: a new diet. The problem with monetizing a new diet is that, once word gets out, anyone can just try it without having to pay. Thus the prevalence of pills: pills can be patented and monetized. But with this market structure, the investors are PAYING people to participate. Anyone who's trying the diet will WANT to sign up, because they get paid. The investors can then reap the benefits (if any) over the person's lifetime.

Finally, a preventative example. Once again, we have a prostate cancer treatment, but this time it is preventative rather than a cure. This time, prostate cancer is not required for participation. The investors pay the participants for their effort, and make money for every prostate-cancer-free year of every participant's life. Again, there is more complexity around paying for any later problems, insuring those payments, etc... but that's all standardized stuff and the associated costs are known in advance.

If an AGI is based on a neural network, how can you tell from the logs whether or not the AI knows it's in a simulation?

After reading the story at the beginning, I thought "huh, this teacher seems rather low-level for a teacher". I also thought that back when I was first getting into LW, a depiction of that level as the highest level would not have encouraged me to explore further.

I was more pleased with the bit at the end.

At some point, it might be worth making a few versions of this story which illustrate some of the trickier techniques, with the urn lady trying exploit specific biases. If a short story like that illustrated a bias well enough to trick the reader right up until the reveal, and the reveal were written so that the reader could believably learn to think in such a way as to catch it, I think that would really help convince newcomers that there's material here worth studying.

Here's my explanation of it. Let me know if this helps with your concerns at all:

Imagine we have an AI design we want to test. Call this AI C3PO, and let its utility function be U(A) where A is a world-state from the set of all possible world-states. And let the super-unlikely-event-happening-at-the-specified-time described in the post be w such that w = true if it happens and w = false if it doesn't happen. Then let A* be a world state in the subset of all world-states A in which w = true. Basically, A* is A given that w happened (this is how we simulate a "false belief" by only allowing the AI to consider worlds in which w = true). Finally, let C be a constant.

The proposal is that we create a variant of C3PO, C3PO* that has the utility function:

U*(A) = P(!w) * C + P(w) * (U(A*))

If the AI is boxed such that it cannot affect the probability of w occurring and it cannot modify its own utility function, then maximizing U* is exactly the same as maximizing U once event w has occurred (ie. with false belief w). In this way, we are able to perfectly simulate C3P0 to find out what it would do if w were true, but we don't actually have to convince it that w is true.