What probability distribution over turing machines do you blindly pick it from? That's another instance of the same problem.

Pragmatically, if I non-blindly pick some representation of turing machines that looks simple to me (e.g. the one Turing used), I don't really doubt that it's within a few thousand bits of the "right" version of solomonoff, whatever that means.

Since we can't agree on a single ideal reference UTM, we instead approximate it by limiting ourselves to a class of "natural" UTMs which are mutually interpretable within a constant.

Even if you do that, you're left with an infinite number of cliques such that within any given clique the languages can write short interpreters for each other. Picking one of the cliques is just as arbitrary as picking a single language to begin with. i.e. for any given class of what-we-intuitively-think-of-as-complicated programs X, you can design a language that can concisely represent members of X and can concisely represent interpreters with this special case.

What I'm confused about is this constant penalty. Is it just "some constant" or is it knowable and small?

It's a function of the language you're writing an interpreter of and the language you're writing it in. "Constant" in that it doesn't depend on the programs you're going to run in the new language. i.e. for any given pair of languages there's a finite amount of disagreement between those two versions of the Solomonoff prior; but for any given number there are pairs of languages that disagree by more than that.

Is there a specific UTM for which we can always write a short compiler on any other UTM?

No. There's no law against having a gigabyte-long opcode for NAND, and using all the shorter opcodes for things-we-intuitively-think-of-as-complicated.

Solomonoff Induction is supposed to be a formalization of Occam’s Razor, and it’s confusing that the formalization has a free parameter in the form of a universal Turing machine that is used to define the notion of complexity.

I'm very confused about this myself, having just read this introductory paper on the subject.

My understanding is that an "ideal" reference UTM would be a universal turing machine with no bias towards any arbitrary string, but rigorously defining such a machine is an open problem. Based on our observation of UTMs, the more arbitrary simplifications a Turing Machine makes, the longer its compiler will have to be on other UTMs. This is called the Short Compiler Assumption. Since we can't agree on a single ideal reference UTM, we instead approximate it by limiting ourselves to a class of "natural" UTMs which are mutually interpretable within a constant. The smaller the constant, the less arbitrary simplification the UTMs in the class will tend to make.

This seems to mesh with the sequences post on Occam's Razor:

What if you don't like Turing machines? Then there's only a constant complexity penalty to design your own Universal Turing Machine that interprets whatever code you give it in whatever programming language you like.

What I'm confused about is this constant penalty. Is it just "some constant" or is it knowable and small? Is there a specific UTM for which we can always write a short compiler on any other UTM?

I'm getting out of my league here, but I'd guess that there is an upper bound on how complex you can make certain instructions across all UTMs because UTMs must (a) be finite, and (b) at the lowest level implement a minimal set of instructions, including a functionally full set of logical connectives. So for example, say I take as my "nonbiased" UTM a UTM that aside from the elementary operations of the machine on its tape, jump instructions, etc. has only a minimal number of instructions implementing a minimally complete operator set with less than two connectives: {NAND} or {NOR}. My understanding is that anything that's a Universal Turing Machine is going to have to itself have a small number of instructions that implement the basic machine instructions and a complete set of connectives somewhere in its instruction set, and converting between {NAND} or {NOR} and any other complete set of connectives can be done with a trivially short encoding.

1) How can I know whether this belief is true?

Expose it to tests. For example, you might stick your head out a window and look up. The theory that the sky is green strongly predicts that you should see green and only very weakly allows for you to see anything else (your eyes may occasionally play tricks on you, perhaps you are looking close to a sunset, etc.).

2) How can I assign a probability to it to test its degree of truthfulness? 3) How can I update this belief?

If you knew absolutely nothing about skies other than that they were some colour, you would start with a symmetrical prior distributed across a division of colour space. You would then update your belief every time you come into contact with entangled information. For example, every time you look up at the sky and see a colour, your posterior probability that the sky is that colour goes up at the expense of alternative colours. Observing other people describe what they see when they look at the sky and learning about how vision works and the chemical composition of the sky are also good examples of evidence you could use.

In practise, manually updating every belief you have all the time is far to arduous, and most people collect large amounts of beliefs and information prior to learning much about statistics anyway. Because of this, the prior probability you assign to your beliefs will often have to be a quick approximation.

Cue for noticing rationalization: In a live conversation, I notice that the time it takes to give the justification for a conclusion when prompted far exceeds the time it took to generate the conclusion to begin with.

I don't like how every time you send or receive a love letter from her you're adding conditions. Under this framework, that means you're changing your girlfriend! I'm not sure what the point of monogamy is if every time you correspond with your mate you switch to a new one.

It doesn't matter whether the meaning someone ascribes to a word seems stupid to you

Yes, it does.

SaidAchmiz even admits that for some of these, the usage he doesn't like is more common, which is a big hint.

Do you really think this is the case? How does this apply to "the exception that proves the rule", for instance?

Consider this hypothetical exchange:

Bob: All bears are either black or white.
Fred: Eh? But I saw a brown bear just yesterday.
Bob: Well, that's the exception that proves the rule.

Let's suppose that this usage is in fact more common than the two that I cited as "correct". It seems to be either false or meaningless. What is Bob saying here? How does Rationalist Taboo help us?

This really just seems to be linguistic pedantry/elitism disguised as trying to make a good point. Almost none of these situations ever lead to misunderstandings, and it seems like you're just annoyed that people use words differently than the dictionary definition tells them to.

I agree. That's the confusion that I was sorting out for myself.

The point I was making is that you attempt to maximise your utility function. Your utility function decreases when you learn of a bad thing happening. My point was that if you don't know, your utility function is the weighted average of the various things you could think could happen, so on average learning the state of the universe does not change your utility in that manner.