In this post I make four arguments:

There's a certain type of knowledge which ML models probably don't learn. Even though they should be able to. This knowledge is arguably more fundamental than the idea of "causality", but simpler/easier to approach.
There's a certain type of knowledge which people rarely analyze. But probably should analyze more.
I describe different cognitive skills (including self-awareness) in terms of classification. Try to show many "dualities" between them.
I describe different recursive learning processes. Tie them to basic cognitive skills.

My arguments are informal and abstract. But I think they are meaningful, because they introduce new concepts and ideas. This post is not just a new way to describe already known ideas, I believe. You don't have to read the entire post, you can stop whenever you like and comment.

A short way to summarize the post: the post describes types of knowledge (K), then cognitive skills based on K (S), then recursive learning processes based on K/S (R)... then (recursive) mental objects based on K/S/R. Each idea builds on the previous one.

Part 1: Types of knowledge

Five types of statements

You can analyze knowledge about X by splitting it into types of statements you can make about X.

I want to discuss five types of statements about real-world objects: absolute, transformational, causal, situational and correlational. I'm going to give examples about visual perception, but similar examples can be made for language, argumentation or board games (e.g. chess, Go) learning and other skills.

"A dog is unlikely to look like a cat." This follows directly from the "definition" of a dog. In order to know it you don't need to know how cats and dogs are embedded in the real world. So, this is an absolute statement.

"A dog looks different from different angles." This statement is like an absolute statement, but it's harder to know. To know it you need to understand how dogs are embedded in some logical/mathematical space. So, this is a transformational statement.

"Dogs are unlikely to stand on top of one another." This doesn't follow from the "definition" of a dog. This follows from our (causal) model of the world, relates to a causal rule/mechanic: it's hard for a dog to stand on top of another dog, it's not a stable situation. To know this you need to understand how dogs are embedded in the local causal environment. This is a causal statement.

"A physical place is unlikely to be fully filled with dogs." This doesn't follow from the "definition" of a dog. And this doesn't relate to local causality. Rather, it describes the "shape" of the concept, a connection between the concept and many other concepts and basic properties of images. Dogs are not sheep or birds or insects, they don't make up hordes, they don't fill up your visual field. This is a situational statement. It's related to a global causal process which prevents the existence of dog hordes.

"If you see a dog, you're likely enough to see a human with a leash nearby." This statement is similar to the previous one. But it doesn't connect the "dog" concept to general properties of images or to many other concepts. It's much more specific. It's a correlational statement. But it's related to a global causal process too.

Now, consider this:

Situational statements are not equal to "human-like concepts". However, many of them should be more or less interpretable by humans. Because many of them are simple abstract facts.

It's likely that Machine Learning methods don't learn situational statements as well as they could, as sharply as they could. Otherwise we could be able to extract many of them. And Machine Learning methods don't have the incentive to learn such statements. They may opt for learning more low-hanging correlational statements and waste resources remembering tons of specific information (e.g. DALL-E which remembers tons of specific textures).

However, Machine Learning methods should have all the means to learn situational statements. Learning those statements shouldn't require specific data (like learning transformational statements does: for example, you may need pictures of objects from different angles to learn rotation transformations). And it's easier than learning causal or symbolic reasoning. At least it's closer to direct experience and statistics than pure causal/symbolic reasoning.

Learning/comparing situational statements is interesting enough on its own. It could be its own area of research. But we don't apply statistics/Machine Learning to try mining those statements. This may be a missed opportunity for humans.

There should exist a method to force a ML model to focus on learning situational statements.

Maybe there's a recursive method which forces a model to seek more and more situational statements in the data.

Learning situational statements may be key to learning causality and modeling qualia. More on that down below.

If this is true it means that there are important ideas missed in Machine Learning. I'm not a ML expert, but statements about learning don't have to be formalized in ML terms, ML doesn't have a monopoly on the topic of learning. However, I'm sorry if my statements about ML specifically are wrong.

Local causality

Learning situational statements may be key to learning causality

Some ingredients of thinking about causality:

Connecting knowledge about different objects.
Chunking reality into areas.
Ignoring overly specific information.
Imagining "abstract scenes" (situations stripped from most real-world features).

Situational statements give you all those ingredients. They are "a priori" building blocks you need for causality.

Global causality

Moreover, you can argue that situational statements already describe causality themselves. It's just global causality, not local causality. However, the argument for it is a bit subtle. Let's go back a bit:

"A physical place is unlikely to be fully filled with dogs." (...) This is a situational statement. It's related to a global causal process which prevent the existence of dog hordes.

At this point you might've been thinking "Wait, there can be a simple explanation of this dog fact. For example, we may be living in a world where only 5 dogs exist, so of course they can't create a horde. No global causal process required, it's just a simple logical fact."

In such case this very logical fact about the world will be a part of a "global causal process". Situational statements talk about global processes by definition. You see, the five types of statements are not English statements, they are interpretations of English statements. I.e. an English statement can change its type depending on how you expand it. "There can't be dog hordes because there are only 5 dogs" is not a situational statement anymore, it's a logical fact. "There can't be dog hordes but there can be swarms of insects because we ended up in a world where there are only 5 dogs and much more insects" is a situational statement. It may sound strange, but it's actually a simple and useful way to conceptualize situational statements. Situational statements are just a layer of cognition which is always there. By the way, this may be the reason why people believe in Gods, Fate, wishful thinking, karma and conspiracies: because those ideas don't have to be "made up", they're always there. A global causal process always exists in cognition, people just get confused about what it is.

Conclusions:

You may treat global causality as more fundamental than local causality and even more fundamental than logical inferences inside of a specific world. Logical Decision Theory does the same, it treats global causality as something more fundamental than local causality and even some logical inferences (because it considers logical counterfactuals).
Superstitious, motivated, emotional modes of thinking - all can teach you logic and causality. If you find a way to abstract those modes.

Recursive differentiation

Maybe there's a recursive method which forces a model to seek more and more situational statements in the data.

I can't give a formal example, but here's an informal example of such method:

Find a difference between two objects, some property.
Imagine that both objects have this property.
Seek the difference between objects again. Repeat.

For example:

Dogs and clouds have different features. Let's transform clouds into dogs through something like a DeepDream process.
Now dogs and clouds have the same features, but clouds have unrealistic chunks of dog-like features (e.g. multiple heads blended together). Let's smooth it.
Now dogs and clouds are even more similar. But still, "cloud dogs" have unrealistic shapes. Let's improve it even more.
Now dogs and clouds are identical. But dogs are usually on the ground and clouds are in the sky. Let's imagine that both are on the ground.
There are still differences: (1) clouds are usually surrounded by nothing, dogs are usually surrounded by something, (2) clouds usually make bigger "packs".

As you see, dreaming and image generation could be used to learn and explore situational statements. Even outside of this exact method.

This learning routine is beyond ML, a human could do the same routine. So, don't dismiss it completely. I already tried to give an example of it here. I know, the thing described in the linked post 90% likely doesn't work or doesn't specify enough. But it's just an illustration.

Qualia

Learning situational statements may be key to modeling qualia.

Simply because "situational statements" are the closest to modeling qualia compared to all other statement types (local causality, correlations, transformations).

And given the recursive idea above, you can argue that:

Experiencing qualia is like experiencing an unfolding and unstable difference between objects.
When recursive differentiation reaches some limit, you get qualia.

If you can trust my word: I have synesthetic and synethesia-like experiences. I'm extremely confident that the "recursive differentiation" method can create new qualia. It mixes your experiences and then splits them into flavors. Those flavors get contrasted and saturated. Then mixed again. This mixing and splitting repeats until it reaches something stable, the "pure flavors". But even then it continues to go on forever. I know because I experience it constantly. And if we don't take into account my experience, it's still an original conceptual idea ("qualia as the limit of a recursive process") that should be acknowledged.

Meta networks

I think it's important to consider that some analogs of processes which happen at the level of neurons may happen at the meta level (level of concepts). For example:

Backpropagation. Recursive differentiation of objects is somewhat similar to backpropagation. But instead of neurons you update objects and differences.
Diffusion. Recursive differentiation is somewhat similar to denoising. But instead of noise you add/delete features.

Part 2: Cognitive skills

Situational values/concepts

Situational statements can also be used to analyze definitions of human values and concepts. We analyzed situational statements as global causal processes that create configurations of objects. But we can also analyze them as global causal processes which create objects themselves.

For example, you can imagine such statement:

I consider Marcel Duchamp's Fountain to be real art IFF at least 80% of art is not like this. E.g. if 50% of art were like this I would stop considering it art.

We may determine if something is or isn't a member of the "Art set" by defining properties of the Art set itself. Not properties of all members; properties of the set itself. This is important for analyzing sets where members don't have enough common properties: see Family resemblance by Ludwig Wittgenstein. Those "properties of the set itself" can be treated as a global causal process which creates art objects.

I think it's a new way to combine uncertainty and logic. (We already have probability and fuzzy logic. This is the third way.)

Here's another possible situational statement:

We should be able to restrict 70% of the freedom of a single individual to increase the overall freedom of society, but not more. Because the freedom of society should be a proxy for the freedom of the individuals. If individuals are not safe from losing 100% of their freedom, the concept starts to contradict itself and eat itself.

This is a combination between deontological and consequentialist ethics. I think it's important to investigate for Alignment and whatnot.

Statements and probability

There's a way to look at the five types of statements in the context of conditional probability. Each type of statements introduces it's own type of probability/type of probability-creating process.

Assume this is a list of rare qualities of dogs:

A. Being blue.
B. Having very high speed.
C. Being very angry.
D. Being made of stone.
E. Being a toy robot.

Different combinations of those have different properties, knowing which requires understanding different types of statements:

(A + D) may be more likely than (A) alone. Because there can be an unexpected connection between A and D, e.g. there can be a lot of blue stone. To know this you need correlational statements.
(B + D) is "causally unlikely", stones usually don't have any way to move. To know this you need causal statements.
(D + E) is "unlikely by definition", robots are made of metal. To know this you need absolute statements.
(C + E) is very unlikely, but not because of local causality. Nobody makes angry toys (it would be strange, ridiculous and dangerous) and robot dogs are rare too. There shouldn't be any unexpected connection between C and E, because the outcome (C + E) feels especially wrong. You assume there's some global causality that prevents such outcome from occurring. In the role of global causality you can have "principles of human cooperation" or "human common sense" or "toy police" or ~~"God/karmic retribution"~~ which would prevent the existence of angry (robot) toys. To know this you need situational statements.

Combinations can have mixes of qualities, of course. For example, "robots made of stone" is simultaneously unlikely by definition, causally unlikely (can you make stone robots?) and "situationally unlikely" (stone robots are ridiculous and dangerous... and silly).

Reasoning types

I'm going to use the following terminology:

Object. By "object" I mean a class of objects.
Fundamental vs. specific. Fundamental means "abstracted", contained in the data many times. For example, circle is a fundamental shape for us (humans), because circles are everywhere.
Object space vs. situation space. "Object space" can be a space of clusters, where each cluster represents an object. "Situation space" is a space of all known situations in the real world.
Situational vs. meta-situational statements. Situational statements talk about objects in the real world, meta-situational statements talk about "shapes" of objects' clusters.
Mathematical and symbolic statements. I haven't introduced them. Basically, I just mean mathematical and symbolic reasoning.

With this terminology you can split statement types into components, which effectively gives you a classification of reasoning types:

Absolute statements describe specific configurations (in "object space") of specific objects. Unpacking: clusters in the object space may be arbitrary.
Causal statements describe fundamental configurations (in "situation space") of fundamental objects. Unpacking: casual rules should be abstract and should apply to most objects.
Situational statements describe fundamental configurations (in "situation space") of specific objects.
Meta-situational statements describe fundamental configurations (in "object space") of specific objects.
Symbolic statements describe fundamental configurations (in "object space") of fundamental objects. Unpacking: "symbols" are fundamental objects, their combinations (e.g. "A = A", "A = B", "A =/= C") are fundamental configurations.
Correlational statements describe specific configurations (in "situation space") of specific objects.
Mathematical statements describe specific configurations (in "situation space") of fundamental objects. Unpacking: math is largely about building specific structures out of abstract building blocks.
Self-aware statements describe specific configurations (in "object space") of fundamental objects. E.g. you take a fundamental object and embed it into a different object space, where it is arbitrary.

Let's unpack the last type of statements: they take a fundamental category and view it as if it was arbitrary. For example, "I" (ego) is a fundamental category for a person, it's the center and a necessary component of their universe. But the person may start asking "Why am I me?", "Why am I conscious?", "Why do I exist?", "What does the world look like from someone else's perspective?" or "What happens after I die?". All this self-awareness allows to view a fundamental category as an arbitrary one. By putting the ego in an ontology where it's not fundamental. Of course, not any self-aware statement is going to give you human-level self-awareness or any self-awareness at all... self-aware statements can be used for rethinking other categories and whatnot. Actual self-awareness is just one specific usage.

If I'm right, a hypothetical being could be incentivized to optimize for developing different kinds of reasoning, as if basic cognition is an RPG. The features of this model of cognition:

You can treat different reasoning types as vectors of development (or limits of development), not something binary.
You can find unexpected dualities between very different reasoning tasks. E.g. "develop pre-mathematical intuitions" and "become self-aware".

Part 3: Recursive learning

You can use the same terminology to explore different learning processes, of which "recursive differentiation" is only one example. My descriptions are going to be slightly confusing and very clumsy, but consider this: those are very real learning processes, a human can do them and they can (i.e. this is a possibility) bring insight - and they do bring insight if you trust my word. Moreover, they can describe any learning, e.g. value learning/value forming. And they're simple enough. So, however silly they might sound, they are worth considering: as learning mechanisms that can or could happen. They can or could work in our world. An AGI can or could deconstruct our concepts using those learning strategies. An AGI can or could get aligned or misaligned following those learning strategies.

And do we have a better theory of high-level learning? Any theory at all? Any idea of what high-level abstractions a learner may possibly learn? The classification down below at least gives us a glimpse into what's possible, shows us in what ways our concepts can be deconstructed.

And I know that one can't "code AI with English sentences". My "statements" describe what types of information a process produces and uses, they don't suppose to tell you how to create the process. They can help only indirectly.

Recursive learning processes

Processes 1-3

Differentiation. Choose objects. Mix their specific features. Update their specific configuration in the "object space". (Repeat until you get a fundamental configuration.)

Simplified example: "Mix features of horror movies and comedy movies. Update the difference between horrors and comedies. Repeat." This is the "recursive differentiation" method discussed above.

Filter. Choose objects. Mix their specific features. Separate their specific features using meta-situational statements. Update their specific configuration in the "object space". (Repeat until you get a fundamental configuration.)

Simplified example: "Imagine that every movie is a horror movie. Reconsider what movies are horrors, given that 70% of them can't be horrors. Repeat."

A more interesting way to describe "Filter":

Features are not contained in specific objects. Instead, there's a common pool of features. Different objects "fight" for common features. (A different way to put it: different objects share identities with each other.) It's like probability - all possible features correspond to 100%. The splitting of features also can't be arbitrary, e.g. 99% for one object and 1% for another. Because such splitting wouldn't count as a fundamental configuration.

Simplification. Choose objects. Turn all their specific features into fundamental ones using self-aware statements. Delete redundant features with meta-situational statements. Update the specific configuration in the "object space". (Repeat until you get a fundamental configuration.)

Simplified example: "Imagine that every feature of horror movies is fundamental to horrors, follows from a single source. Reconsider what are the features of horrors. Leave only 3 main features. Repeat."

Processes 4-5

Puzzle. Model objects with configurations of specific features. Evaluate the results. Try finding better configurations and better features. (Repeat until you get fundamental features and a bunch of fundamental configurations.)

People did this a lot in the history of mathematics and/or physics. People sought interesting, useful and universal "puzzle pieces" for modeling reality. And on the meta level this is what I'm doing here. Good puzzle pieces have to be abstract, but they also have to reveal some simple structures in the world, otherwise they're useless.

Monism. Choose a specific feature. Try finding this feature in all objects. What configurations in the "objects space" does it reveal? Update the universal feature. Repeat until you find fundamental configurations in the "object space". You will get a (meta-)situational statement which describes the structure of the "objects space".

Simplified example: "Watch every movie as if it's very scary. Now you're likely to classify a lot of movies as horrors. What configurations in the 'movie space' does it reveal? Update the universal feature until you find fundamental configurations in the 'movie space'."

A more interesting example: motivated cognition as "Bayesianism inside out".

Bayesianism assumes you can describe reality in terms of an infinity of micro atomic events with one parameter (probability). I think you can describe reality in terms of a single macro fuzzy event with two parameters (probability and complexity).

Processes 6-7

Percolation. Choose specific features. Generalize them until you reach a contradiction in the "object space". Split the "object space" into different objects to resolve the contradiction. Repeat until you find fundamental objects. (This method is heavily inspired by conversations with ZT5.)

Simplified example: "Imagine that every movie has most features of horrors and comedies at the same time. That's a contradiction. Split the 'movie space' into different objects, e.g. not 'movies' but 'sets of episodes'. See that some episodes of a movie can be horror-like and other can be comedy-like. Repeat."

Nesting. Choose two objects with two different features. Split the objects into different parts, so that you can find one of the features already contained in both objects. Like yin and yang. (Repeat until you get a fundamental feature.)

Simplified example (not very realistic): "Horror movies have bad endings and comedy movies have good endings. But if you split 'movies' into 'episodes', you can see that horror movies have episodes with 'good endings'. You can repeat the process." In this example "nesting" is very similar to "percolation", but they don't have to be similar.

Process space

Those processes exist in a vast process-space. You can use different statement types, multiple "object spaces" and whatnot to create hundreds of versions of each process.

Part 4: Mental objects

Instead of learning processes we can think about learned objects/configurations while still using the same terminology. Some of those objects may "modify themselves" in a recursive fashion.

My descriptions below may sound like a baseless and sloppy fantasy, but look at it this way:

Those "objects" are real objects which can be learned.
Some properties of those objects can be described from the first principles.
Those objects describe information. They don't have to actually and explicitly exist in anyone's neural net to be real.

We really should ask ourselves: why haven't we thought about this topic for at least 5 minutes?

Learned objects

Objects 1-5

Reification object. An object which can change from specific to fundamental. Example: random set of lines which can be connected by perception into a shape.

Texture. A fundamental (i.e. abstract) configuration which can be turned into a fundamental object. Like the uniform pattern of ocean waves which can be turned into a "wavy texture", a single object.

Glider. A specific object (cluster). When you split it, you find that it's better to "move" the boundary of the cluster a little bit. This can repeat, creating a "moving" cluster.

Fractal. A specific object. When you split it, you get fundamental objects (i.e. very frequent) and self-aware statements (which allow to see that the same object exists on different layers of reality).

Lantern. An object which turns specific configurations of objects around itself into fundamental configurations. Example: "Imagine a bird's eye view of a city. If you don't have any concept of the 'ground', the distribution of buildings looks pretty random/weird. However, when you know what 'ground' is and keep in mind an imaginary plane, the distribution of buildings looks very natural."

Objects 6-8

Aleph. "Aleph" is a fundamental feature which helps you to find an interesting feature of another object. This "interesting feature" can become a part of new Aleph. So, Aleph absorbs all (interesting) features.

Bubble/lump. An object which prevents a configuration from being fundamental. It's a "weak link" in a configuration. Example: "Imagine a person with a ball on their head. This is a specific configuration (a rare one). But it's close to a fundamental configuration - if the ball falls from the person's head. The ball here is a 'bubble'." The apple in The Son of Man by Rene Magritte is a bubble too.

Star is a unique object which counts as fundamental because it is contained in many parts of the data. Model is a unique object which counts as fundamental because of some fundamental feature. Example: "The Sun is a fundamental object to humans. It plays a role in a lot of situations. And it's a uniquely bright object (and uniquely sky-hanging object during the day). It's both a 'star' and a 'model'."

Learned configurations

It also makes sense to focus on learned configurations specifically. I know, this is getting tedious: another chunk of obscure examples. But it's needed to build understanding.

Configurations

Symmetric config. A fundamental configuration. Which you can split into smaller fundamental configurations. Example: Imagine a circle made out of dots. Objects in a circle is a fundamental configuration. A line of objects is a fundamental configuration too.

Loose config. You have a bunch of fundamental objects. However, when you form a configuration out of them, some of them become "less fundamental". Example: Imagine a bunch of circles of different sizes. "Circle" is a fundamental shape (frequent, abstract). However, in the configuration smaller circles are more fundamental because they're easier to notice/fully observe.

Balance. "Balance" is a specific configuration held together by a fundamental configuration. Example: Imagine any situation of delicate balance, such as an elephant standing on a ball or The Leaning Tower of Lire. It's not a fundamental configuration on some level (because it's not stable). But it is a fundamental configuration on another level (because it expresses a fundamental pattern: a symmetry or proportionality that creates the balance).

Tree config. "Tree" is an object which can have different amount of parts ("branches"). A tree config may work like this: "Imagine two 'trees'. When one tree loses a part, another grows a part. The amount of parts is conserved."

Joints/magnets. A specific object which is shaped by a fundamental object. (Like a shape warped by another shape.) Or a specific object which is "fixed" by a fundamental object. (Like a random object placed exactly at the center of something.)

Glow. A specific object surrounded by its more fundamental versions (versions stripped off features).

Palette. Fundamental objects ("colors") which capture most features of a set of specific objects.

You can combine all those objects in hundreds of ways.

The analysis method used in this post, the "method of statements", is explained and explored here.