Talk of "density in thingspace" sweeps a lot of interesting problems under the rug. Thingspace is very high-dimensional. What distance metric are we supposed to use? EY writes:
I believe that in the field of statistical learning, for algorithms that actually do depend on distance metrics, the standard cheap trick is to "sphere" the space by making the standard deviation equal 1 in all directions.
An alternative (which I use at work) is to make the overall range 1 in all directions. Doubtless there are other, equally arbitrary choices available.
Today's post, Mutual Information, and Density in Thingspace was originally published on 23 February 2008. A summary (taken from the LW wiki):
Discuss the post here (rather than in the comments to the original post).
This post is part of the Rerunning the Sequences series, where we'll be going through Eliezer Yudkowsky's old posts in order so that people who are interested can (re-)read and discuss them. The previous post was Entropy, and Short Codes, and you can use the sequence_reruns tag or rss feed to follow the rest of the series.
Sequence reruns are a community-driven effort. You can participate by re-reading the sequence post, discussing it here, posting the next day's sequence reruns post, or summarizing forthcoming articles on the wiki. Go here for more details, or to have meta discussions about the Rerunning the Sequences series.