Lumifer, you are falling prey to several of the traps detailed in A Human's Guide to Words. So far I have basically parroted EY's 102 material.

Meditation: Taboo "Knowledge" and describe your relation with riding a bicycle.

Meditation: Taboo "Knowledge" and describe your relation with some field of science you are proficient in.

Meditation: Taboo "Knowledge" and describe a religious person's views on god.

...

...

You and me both know what 'knowledge' is in everyday speech. The problem is what constitues 'knowlege' in extreme situations.

The thing is that "Knowledge" is ambiguous in everyday speech. We misunderstood each other when I initially answered your question: I thought you were speaking about the tried and tired philosophical issue that have been discussed for ages.

The answer in the Philosophical Issue of Knowledge is: "You philosophers are all morons; you are using the same word to mean different things."

Plato has a famous definition of "Knowledge": Justified True Belief. Notice how he has moved the problem of explaining "Knowledge" into the problem of explaining "Justification." (And "True." And "Belief." Neither concepts were actually well explained when Plato was alive and kicking.)

"Knowledge" can also be a synonym for "Skill." Such as knowing how to ride a bicycle. Notice how the grammatical construction "knowing how to <verb phrase>." is different from "knowing <noun phrase> to be true." One could argue that they are the same thing, but I think they are not. So we have at least two types of everyday discussed knowledge: Procedural Knowledge (how to do stuff) and Object Knowledge (facts and stuff).

The distinction between the two is obvious when you really taboo it: Procedural knowledge is like a tool. It is a means to an end, an extension of your primitive action set. Having lots of procedural knowledge is a boon in Instrumental Rationality, but most skills are irrelevant to Epistemological Rationality. (Riding a cicycle will only very rarely tell you the secrets of the universe.)

Object Knowledge, or Facts, are thingies in your mental model of how the world works. This mental model is what you use when you want to predict how the world is going to behave in the future, so that you can make plans. (Because you have goals you want to attain.)

Your world model is updated automatically by processes which you do not control. A sufficiently advanced agent might be able to excercise some control, at least at the design level, of it's updating algorithms. In short, you take in sensory data and crunch some numbers and out comes a bayesian-esque update.

So my standing viewpoint is: I don't care what you call it; "knowledge" or "hunch" or "divine inspiration." I care about what your probability distribution over future events is. I don't care what you call it "skills" or "knowledge" or "talent." I care about what sort of planning algorithm you implement.

And on the topic of subjectivity: If I have trained skills or observed evidence different from you, then yes we have subjectively different "knowledge." I for instance know 12 programming languages and intimate facts about my significant other.

But the thing is that there is only One Correct Way of updating on evidence: Bayes Theorem. If you deviate from that you will have less than optimal predictive power.

I really suggest you go and read some of the core sequences to refresh this.

I have no idea what these algorithms might be are and neither do you. Accordingly I don't see any basis for speculating what will they allow.

Well, let's think about whether we have a proof of concept. What's an example of a generalization about high-complexity algorithms that might show most of them to be easily usefully compressed, for an observer living inside one? At this point it's OK if we don't know that the generalization holds; I just want to know what it could even look like to discover that a universe that looks like ours (as opposed to, say, one that looks like a patchwork or a Boltzmann Braintopia) is the norm for high-complexity sapience-permitting worlds.

ETA: Since most conceivable universes are very very complicated, I'd agree that we probably live in a very very complicated universe, if it could be shown that our empirical data doesn't strongly support nomic simplicity.

The default rule is that "man is the measure of all things" so presumably you are using these words in the context of what is short and simple for the human brain.

No, I'm saying it's short and simple relative to the number of ways a universe could be, and short and simple relative to the number of ways a life-bearing universe could be. There's no upper bound on how complicated a universe could in principle be, but there is a lower bound, and our physics is, even in human terms, not far off from that lower bound.

I don't think that most high-complexity algorithms for building a life-permitting observable universe would allow a theory as simple as human physics to approximate the algorithm as well as our observable universe does.

Do you think the observable universe is a lot more complicated than it appears?

I model basically everyone I interact with as an agent. This is useful when trying to get help from people who don't want to help you, such as customer service or bureaucrats. By giving the agent agency, it's easy to identify the problem: the agent in question wants to get rid of you with the least amount of effort so they can go back to chatting with their coworkers/browsing the internet/listening to the radio. The solution is generally to make it seem like less effort to get rid of you by helping you with your problem (which is their job after all) than something else. This can be done by simply insisting on being helped, making a ruckus, or asking for a manager, depending on the situation.

One of my habits while driving is to attempt to model the minds of many of the drivers around me (in situations of light traffic). One result is that when someone does something unexpected, my first reaction is typically "what does he know that I don't?" rather than "what is that idiot doing?". From talking to other drivers, this part of my driving seem abnormal.

In this sense, I model my parents strongly as agents–I have close to 100% confidence that they will do whatever it takes to solve a problem for me.

One of the frequent complaints about the 'agent' concept space, and the "heroic responsibility" concept in particular, is that it rarely seems to take into account people's spheres of responsibility. Are your parents the sort of people who would be able to solve anyone's problem, or are they especially responsible for you? Are other people that seem to be NPCs to you just people that don't care enough about you to spend limited cognitive (and other) resources on you and your problems?

With people who I model as agents, I'm more likely to invoke phrases like "it was your fault that X happened" or "you said you would do Y, why didn't you?" The degree to which I feel blame or judgement towards people for not doing things they said they would do is almost directly proportional to how much I model them as agents. For people who I consider less agenty, whom I model more as complex systems, I'm more likely to skip the blaming step and jump right to "what are the things that made it hard for you to do Y? Can we fix them?"

Do you get more of what you want by blaming people or assigning fault?

The idea is, however we define P, how can we be that sure that there isn't some kind of uncomputable physics that would allow someone to build a device that can find the lexicographically least object x such that P(x) < 1/3^^^3 and present it us?

You just run the Prolog (or whatever logic system implements all this), and it either terminates with a failure or does not terminate within the time allowed by the competition. The time limit renders everything practically decidable.

Deem both of the agents to have not terminated?

Thinking about this further, given enough assertions, there's no need to have the programs at all. Let the agents be just the assertions that they make about themselves. Each agent would consist of a set of axioms, perhaps written in Prolog, about who they cooperate or defect against, and running a contest between two agents would just be a matter of taking the union of the two sets of axioms and attempting to deduce (by the standard Prolog proof-search) what choice each one makes.

What's wrong with "I will cooperate with anyone who verifiably asserts that they cooperate with me".

The word "me". By Rice's theorem, they can't tell that they're dealing with someone computationally equivalent to me, and there's no other sense of my "identity" that can be referred to.

Athough that could be added. Have all competitors choose a unique name and allow them to verifiably claim to be the owner of their name. Then "I will cooperate with anyone who verifiably asserts that they cooperate with me" can work if "me" is understood to mean "the entity with my name". Discovering a blob's name would have to be a second primitive operation allowed on blobs.

ETA: I think I missed your point a little. Yes, a cooperate-bot verifiably asserts that it cooperates with everyone, so it is an entity that "verifiably asserts (something that implies) that they will cooperate with me." And there will be other bots of which this is true. But I'm not sure that I can verifiably express that I will cooperate with the class of all such bots, because "the class of all such bots" looks undecidable.