Actually: Maybe this is a decent chance to have Less Wrong folk help refine common LW concepts. Or at least a few of the ones that are featured in this article. Certainly most folk won't share my confusions, misgivings, or ignorance about most of Less Wrong's recurring concepts, but certainly a few must about a few, and could benefit from such a list! Consider this comment a test of the idea. I'll list some concepts from this post that I'm dissatisfied with and at least one reason why, and maybe others can point out a better way of thinking about the concept or a way to make it more precise or less confused. Here we go:

• Mathematical structure: I don't know what this means, and I don't think Wikipedia's definition is the relevant one. Can we give minimal examples of mathematical structures? What's simpler than the empty set? Is a single axiom a mathematical structure? (What if that axiom is ridiculously (infinitely?) long---how short does something have to be to be an axiom? Where are we getting the language we use to write out the axiom, and where did it get its axioms? ("Memetic evolution?! That's not even a... what is this i dont even"))
• Ensemble: There are a whole bunch of different ensemble formulations---see Standish's "Theory of Nothing" for a few. Besides being ensembles of different 'things'---mathematical structures, computations, whatever---are there other ways of differentiating or classifying them? Is the smallest ensemble that deserves its name just "anything computable", or is there a way to use e.g. (infinite and/or infinitely nested?) speed priors to get an interesting "smaller" ensemble? If they're not equivalent, is that an accident? Is any of this relevant to choosing what things some part of your decision theory is allowed to reason about, at least at first? (I'm thinking about dangerous Pascalian problems if infinities are allowed anywhere.)
• Weightings: Why does a universe-wide universal-prior weighting match the same universal prior probability distribution that is optimal for predicting inputs from environments of arbitrary size/complexity? (Presumably I'd better understand this if I actually understood the proofs, but in general I'm annoyed at that "within a constant!" trick even if it makes up like half of algorithmic probability theory.) And what does it mean for reality to have a chosen Turing language? What if it disagrees with your chosen Turing language; presumably Solomonoff induction would learn that too? (What are the answers to the more pointed questions I should be asking here?)
• computation: I sort of get what a computation is in its abstract form. But then I look at my desk. What computations are being executed? How do we analyze the physical world as being made up of infinite infinitely overlapping computations? What is approximate computation? What does it mean to execute a computation? Inquiry about computation could go on for a long time though, and deserves it's own bundle of posts.
• sweeping confusion into UDT: (I have been told that) in many decision scenarios it can be determined by a bookie what the UDT agent's implicit preferences and probabilities are even after they've been timestamped into a decision policy---they might not be explicitly separated and in some cases that gets around tricky problems but generally it's kind of misleading to say that it's all just preference, considering the processes that determined those preferences happened to care about this weird "probability"-ish thing called (relative) existence. Are any probabilistic-ish problems being obscured because of not explicitly distinguishing between beliefs and preferences even though implicitly they're there? (Is this line of inquiry fundamentally confused, and if so, how?)
• anthropics: Is there any reason to bother pondering anthropics directly, rather than thinking decision theoretically while taking into account that ones preferences seem like they may refer to something like probabilities? This is related to the previous bullet point.

If this is useful---if some answers to the above questions or confusions are useful---then I'll consider writing a much longer post in the same vein. Being annoyed or disturbed about imperfections of concepts or conceptual breakdowns is something I am automatically predisposed to be motivated to do, and if this works I can see myself as being more valuable to the LW community, especially the part of the community that is afraid to ask stupid questions with a loud voice.

My reasoning is this:

Consider the domain of bit streams - to avoid having to deal with infinity, let's take some large but finite length, say a trillion bits. Then there are 2^trillion possible bit streams. Now restrict our attention to just those that begin with a particular ordered pattern, say the text of Hamlet, and choose one of those at random. (We can run this experiment on a real computer by taking a copy of said text and appending enough random noise to bring the file size up to a trillion bits.) What can we say about the result?

Well, almost all bit streams that begin with the text of Hamlet, consist of just random noise thereafter, so almost certainly the one we choose will lapse into random noise as soon as the chosen text ends.

Suppose we go about things in a different way and instead of choosing a bit stream directly, we take our domain as that of programs a trillion bits long, and then take the first trillion bits of the program's output as our bit stream. Now restrict our attention to just those programs whose output begins with the text of Hamlet, and choose one of those at random. What can we say about the result this time?

It's possible that the program we chose consists of print "...", i.e. the output bit stream is just embedded in the program code. Then our conclusion from choosing a bit stream directly still applies.

But this is highly unlikely. The entropy of English text has been estimated at about one bit per character. In other words, there exist subroutines that will output the text of Hamlet, that are about eight times shorter than print "...". That in turn means that in our domain of programs a trillion bits long, exponentially more programs contain the compact subroutine than the literal print statement. Therefore our randomly selected program is almost certainly printing the text of Hamlet by using a compact subroutine, not a literal print statement.

But the compact subroutine will probably not lapse into random noise as soon as Hamlet is done. It will continue to generate... probably not a meaningful sequel, Hamlet doesn't constrain the universe enough for that, but at least something that bears some syntactic resemblance to English text.

For the constraint the output bit stream must begin with the text of Hamlet, substitute at least one observer must exist. We are then left with the conclusion that if the universe were randomly chosen from all bit streams, we should be Boltzmann brains, observing just enough order for an instance of observation, with an immediate lapse back into noise. As we observe continuing order, we may conclude that the universe was randomly chosen from programs (or mathematical descriptions, if the universe is not restricted to be computable) so that it was selected to contain a compact generator of the patterns that produced us, therefore a generator that will continue producing order.

Note that this is an independent derivation of Solomonoff's result that future data should be expected to be that which would be generated by the shortest program that would produce past data. (I actually figured out the above argument as a way to refute Hume's claim that induction is not logical, before I heard of Solomonoff induction.)

It also works equally well if output data is allowed to be infinite, or even uncountably infinite in size (e.g. continuum physics), because uncountably infinite data can still be generated by a formal definition of finite size.

What interests me about the Boltzmann brain (this is a bit of a tangent) is that it sharply poses the question of where the boundary of a subjective state lies. It doesn't seem that there's any part X of your mental state that couldn't be replaced by a mere "impression of X". E.g. an impression of having been to a party yesterday rather than a memory of the party. Or an impression that one is aware of two differently-coloured patches rather than the patches themselves together with their colours. Or an impression of 'difference' rather than an impression of differently coloured patches.

If we imagine "you" to be a circle drawn with magic marker around a bunch of miscellaneous odds and ends (ideas, memories etc. but perhaps also bits of the 'outside world', like the tattoos on the guy in Memento) then there seems to be no limit to how small we can draw the circle - how much of your mental state can be regarded as 'external'. But if only the 'interior' of the circle needs to be instantiated in order to have a copy of 'you', it seems like anything, no matter how random, can be regarded as a "Boltzmann brain".

a uniform weighting of mathematical structures in a Tegmark-like 'verse

I don't know what this is supposed to mean. There isn't any uniform distribution over a countably infinite set. We can have a continuous uniform distribution over certain special kinds of uncountable sets (for example, the real unit interval [0,1]), but somehow I doubt that this was the intended reading.

If it turns out that moon is made out of cheese, should you focus on optimizing your world, or the world of the philosopher considering the thought experiment?

It seems that from philosopher's point of view, your world has low measure (moral relevance), but it's unclear whether it's tempting to decide so just because it's not possible to affect the cheese world. From cheese world's point of view, it's even less clear, and the difficulty of control is similar.

I wonder how much of a role ability to control plays in these estimates, since decision theoretically, moral value (and so probability) of a fact one can't control is meaningless, and so one is tempted to assign arbitrary or contradictory values. (A related question is whether it makes sense to have control over probability (or moral value) of a fact without having control over the fact itself (I would guess so), in which case probability (moral value) isn't meaningless if it itself can be controlled, even if the fact it's probability (moral value) of can't.)

Every now and then I see a claim that if there were a uniform weighting of mathematical structures in a Tegmark-like 'verse---whatever that would mean even if we ignore the decision theoretic aspects which really can't be ignored but whatever---that would imply we should expect to find ourselves as Boltzmann mind-computations

The idea is this: Just as most N-bit binary strings have Kolmogorov complexity close to N, so most N-bit binary strings containing s as a substring have Kolmogorov complexity at least N - length(s) + K(s) - somethingsmall.

And now applying the analogy:

N-bit binary string <---> Possible universe

N-bit binary string containing substring s <---> Possible universe containing a being with 'your' subjective state. (Whatever the hell a 'subjective state' is.)

we get:

N-bit binary string containing substring s with Kolmogorov complexity >= N - length(s) + K(s) - O(1) <---> A Boltzmann brain universe.

We don't seem to be experiencing nonsensical chaos, therefore the argument concludes that a uniform weighting is inadequate and an Occamian weighting over structures is necessary

I've never seen 'the argument' finish with that conclusion. The whole point of the Boltzmann brain idea is that even though we're not experiencing nonsensical chaos, it still seems worryingly plausible that everything outside of one's instantaneous mental state is just nonsensical chaos.

What an 'Occamian' weighting buys us is not consistency with our experience of a structured universe (because a Boltzmann brain hypothesis already gives us that) but the ability to use science to decide what to believe - and thus what to do - rather than descend into a pit of nihilism and despair.

My response is that simpler universes would exist within more complicated ones, and thus you'd be more likely to be in a simpler universe. For example, the universe as we know it is much simpler than a Boltzmann brain. As such, you'd be more likely to find our universe somewhere within another universe than you would be to find a Boltzmann brain.

I'm generally against this weighting because of all the infinity paradoxes. For example, no matter how complex the universe is, we'd logically figure that it's almost definitely much more complex.