Daniel_Burfoot comments on Kevin T. Kelly's Ockham Efficiency Theorem - Less Wrong

30 Post author: Johnicholas 16 August 2010 04:46AM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (81)

You are viewing a single comment's thread. Show more comments above.

Comment author: cousin_it 19 August 2010 11:46:19AM *  5 points [-]

I don't think it's been thoroughly explained in the sequences, but why not start with the Wikipedia page? Or this post.

On second thought, this is actually a very simple idea that can be explained in a comment. Let me try:

The Kolmogorov complexity of a finite string of bytes is the size (in bytes) of the shortest Python program that outputs that string and then halts. Or you could use any other language in place of Python and get a different value of complexity - but it cannot differ by more than a constant as the string grows, because you could always implement a constant-size Python interpreter in the other language to "convert" one description into another. So complexity is not unique, but all definitions of complexity sorta "agree" as we go to infinity.

For example, if a string consists of "a" repeated a million times, its Kolmogorov complexity is quite small because you can write a very short program that outputs a million a's and then halts. But this is not the case for all possible strings. Here's a simple way to see why: there can't be more Python programs of length N than there are different byte strings of length N, because any program is itself a byte string. Therefore, you cannot encode all strings of length N using programs of length N/2 or something, because there's not enough of them. Actually this shows that most strings of length N must have complexity close to N. (Obviously it can't be much larger than N, because any string of length N can be output by a program of length N+c that "hardcodes" that specific string.)

Another important observation. Imagine you have a huge, random-looking string of bytes. How can you calculate its Kolmogorov complexity? Turns out that you can't, because there's no easier way to do that than, essentially, simulating all programs up to length N and seeing what they output. But this is impossible because any given program can (say) hang idly for a million years and only then start outputting symbols, and you have no way of learning that reliably from the program text (this is known as the "halting problem"). So K-complexity is largely a theoretical notion, I have no idea why people love it so.

Comment author: Daniel_Burfoot 19 August 2010 04:55:27PM 0 points [-]

So K-complexity is largely a theoretical notion, I have no idea why people love it so.

Imagine some Oracle told you that obtaining the ultimate theory of physics - the one that unifies relativity and quantum gravity, and all that - was impossible. Then following your logic you might conclude that it was pointless to study physics, since one can never obtain the perfect theory. But that would be wrong; it would still be highly worthwhile to attempt to obtain better and better physical theories.