Curated. Tackles thorny conceptual issues at the foundation of AI alignment while also revealing the weak spots of the abstractions used to do so.

I like the general strategy of trying to make progress on understanding the problem relying only on the concept of "basic agency" without having to work on the much harder problem of coming up with a useful formalism of a more full throated conception of agency, whether or not that turns out to be enough in the end.

The core point of the post: that certain kinds of goals only make sense at all given that there are certain kinds of patterns present in the environment, and that most of the problem of making sense of the alignment problem is identifying what those patterns are for the goal of "make aligned AGIs", is plausible and worthy of discussion. I also appreciate that this post elucidates the (according to me) canon-around-these-parts general patterns that render the specific goal of aligning AGIs sensible (eg, compression based analyses of optimization) and presents them as such explicitly.

The introductory examples of patterns that must be present in the general environment for certain simpler goals to make sense—especially how the absence of the pattern makes the goal not make sense—are clear and evocative. I would not be surprised if they helped someone notice that there are some ways that the canon-around-these-parts hypothesized patterns which render "align AGIs" a sensible goal are importantly flawed.

Curated. The problem of certain evidence is an old fundamental problem in Bayesian epistemology and this post makes a simple and powerful conceptual point tied to a standard way of trying to resolve that problem. Explaining how to think about certain evidence vs. something like Jefferey's conditionalization under the prediction market analogy of a Bayesian agent is itself valuable. Further pointing out both that:

1) You can think of evidence and hypotheses as objects of the same type signature using the analogy.

And 

2) The difference between them is revealed by the analogy to be a quantitative rather than qualitative difference.

Moves me much further in the direction of thinking that radical probabilism will be a fruitful research program. Unfruitful research programs rarely reveal deep underlying similarities between seemingly very different types of fundamental objects.

There is! It is now posted! Sorry about the delay.

Hello, last time a taught a class on the basics of Bayesian epistemology. This time I will teach a class that goes a bit further. I will explain what a proper scoring rule is and we will also do some calibration training. In particular, we will play a calibration training game called two lies, a truth, and a probability. I will do this at 7:30 the same place as last time. Come by to check it out.

Hello! Please note that I will be giving a class called the Bayesics in Eigen hall at 7:30. Heard of Bayes's theorem but don't fully understand what the fuss is about? Want to have an intuitive as well as formal understanding of what the Bayesian framework is? Want to learn how to do bayesian updates in your head? Come and learn the Bayesics.

Also, please note that I will be giving a class at 7:30 after the reading group called "The Bayesics" where I will teach you the basics of intuitive Bayesian epistemology and how to do Bayesian updates irl on the fly as a human. All attending the reading group are welcome to join for that as well.

I think you should still write it. I'd be happy to post it instead or bet with you on whether it ends up negative karma if you let me read it first.

AN APOLOGY ON BEHALF OF FOOLS FOR THE DETAIL ORIENTED 

Misfits, hooligans, and rabble rousers. 
Provocateurs and folk who don’t wear trousers. 
These are my allies and my constituents. 
Weak in number yet suffused with arcane power. 

I would never condone bullying in my administration. 
It is true we are at times moved by unkind motivations. 
But without us the pearl clutchers, hard asses, and busy bees would overrun you. 
You would lose an inch of slack per generation. 

Many among us appreciate your precision. 
I admit there are also those who look upon it with derision. 
Remember though that there are worse fates than being pranked. 
You might instead have to watch your friend be “reeducated”, degraded, and spanked 
On high broadband public broadcast television. 

We’re not so different really. 
We often share your indignation 
With those who despise copulation. 
Although our alliance might be uneasy 
We both oppose the soul’s ablation. 

So let us join as cats and dogs, paw in paw 
You will persistently catalog 
And we will joyously gnaw.

I didn’t figure out that the “bow” in “rainbow” referred to a bow like as in bow and arrow, and not a bow like a bow on a frilly dress, until five minutes ago. I was really pretty confused about this since I was like 8. Somebody could’ve explained but nobody did.