[Metadata: crossposted from https://tsvibt.blogspot.com/2023/01/a-strong-mind-continues-its-trajectory.html. First completed January 29, 2023.]

A very strong mind is produced by a trajectory of creativity. A trajectory of creativity that produces a very strong mind is hard to separate from the mind's operation. So a strong mind continues on its trajectory of creativity as long as it is active.

A strong mind comes from a trajectory of creativity

If a mind is highly capable, it got to that point by gaining understanding in a voyage of novelty. If the mind gains understanding that is novel for all the surrounding minds (e.g., preexisting humans), it does so through creativity: generating novelty, rather than merely copying it. If the mind becomes much more capable than the surrounding minds, it does so by being on a trajectory of creativity: something about the mind implies that it generates understanding that is novel to the mind and its environment. If the mind is on a trajectory of creativity that brought it to the point of being highly capable, its trajectory of creativity probably carries the mind much further, making the mind much more capable than it already is.

The ex quo of a mind's creativity is the element (collection of elements) out of which comes novel structure. The ex quo of a modern AI system is almost entirely dependent on the search (i.e. training) apparatus, which is clearly separated out from running the found system. (The ex quo isn't entirely dependent on the search apparatus. Some non-zero creativity happens in the collision of elements that happens in, say, a single run of a stable diffusion image model or a large transformer language model. But it's not much creativity, and the found structure is about as temporarily grasped as possible.)

The proximal ex quo is that out of which novel structure comes directly. The distal ex quo is that out of which novel structure comes indirectly. So the mental context that's set up when a particular idea comes to you, and the other dark matter that goes into that abduction, is the proximal ex quo; human evolution is the distal ex quo; and the history of the development of your brain is an intermediate ex quo.

Trajectory and operation are hard to separate

An AI can simply be shut down, until it's able to and wants to stop you from shutting it down. But can an AI's improvement be shut down, without shutting down the AI? This can be done for all current AI systems in the framework of finding a fairly limited system by a series of tweaks. Just stop tweaking the system, and it will now behave as a fixed (perhaps stochastic) function that doesn't provide earth-shaking capabilities.

I suspect that the ex quo that puts a mind on a trajectory to being very strong, is hard to separate from the operation of the mind. Some gestures at why:

Making doesn't imply understanding

Just because you can make something, doesn't mean you understand how it works.

• Evolution is a distal ex quo of human understanding. But there's clearly an ex quo more proximal than evolution for, say, scientific understanding: human thought and investigation. Setting up an evolution that can produce humans doesn't imply that you understand how humans do science.

• The way we make neural networks today is by setting up a distal ex quo (the search process). A more proximal ex quo for a neural net comes from the accumulated hidden features: they set up the context in which the next little tweak is beneficial. We can know how to make neural nets that work well without knowing much about how the series of tweaks in context build up the computations that end up performing well at the given task.

• We can nevertheless turn off the ex quo of current AI systems because the ex quo is almost entirely dependent on the distal ex quo. We can't, however, turn off the proximal ex quo of human scientific understanding, just by turning off the distal ex quo: scientific creativity doesn't require that genetic evolution is continuing. We can, in a haphazard way, turn off some human creativity while retaining some operation, e.g. by taking sedatives or by punishing creativity.

The ex quo is self-created

Our understanding of creativity isn't deep enough for us to make explicit the ex quo of the creativity of a strong mind. So if we make a strong mind, it's because we made something that finds its own ex quo of creativity.

• The learning is itself being learned. So from our perspective, "what's learned" (as distinct from [the learning process that we explicitly set up]) as an undifferentiated blob includes [the learning process that the mind sets up for itself].

• Learning beyond a certain point has to be online learning. Online learning produces mental elements that combine synchronic and diachronic functions: the elements both participate in currently crystallized skills, and also in the production of new skills. (Analogy: a healthy living codebase is made of small components that both perform their function well and also make themselves and their context readily available to be effectively understood, modified, and extended.) Note: here "learning" is used as a metonymy for creativity; to me "learning" softly excludes, for example, problem solving and imagining, which creativity includes if they involve grasping new ideas. See here.

• So, the operation of the mind includes learning processes, even if our explicitly-set-up search processes have been shut off. More generally, the operation of the mind to perform even familiar tasks is set up to continue being creative.

Mental operation includes creativity

The change that accumulates to constitute creativity is part of and continuous with the everyday operation of the mind. Physiology is continuous with ontogeny.

• The mind's operation--what happens when the mind is applied, when the mind's capabilities are exercised--involves meeting the demands of new contexts. In a new context, concepts adapt by unfolding into concrete consequences; by widening their scope, in the form of guesses or analogies; and by interfacing with each other in the combinations suitable to the context.
  • For example, the idea of "good development" in chess has to interact with the specifics of a given position: maybe this move, even though it blocks your bishop, is best in this position because it prevents an important plan of your opponent. Unfolding the general idea of speedy development into the specifics of which pieces are blocked in which ways, makes the general idea available to interact with the other forces in the position. This work of unfolding seems necessary for the operation of the mind, and seems like it's a substantial portion of the work involved in creativity. It's imaginable to do this work but not remember any of it, i.e. avoid having that work leave traces that can accumulate, but that seems like a delicate, probably unnatural carving.
• The kind of activity that's well-described as "reusing old solutions as building blocks for new solutions" includes activity of a mind that's "obviously merely operational" (e.g., writing a computer program by writing functions that sometimes call previously-written functions), and includes activity of a mind that's much more ambiguous between being operational and being creative. Reusing old solutions as building blocks can add up to immense creativity. Desire paths form gradually.
  • For example, being able to use a solution at all, involves to some extent integrating the solution with other mental elements. A simple example is indexing: recording that some idea is relevant in certain contexts, so that the idea is brought up in those contexts. A less simple example is co-adaptation: other mental elements adjust to the presence of the novel elements, e.g. by learning to interpret signals from the novel elements.
  • For example, take the origin of phonetic writing in the rebus principle. Originary scripts (Mesopotamian cuneiform, Egyptian hieroglyphs, Old Chinese oracle bone script, Mayan script) are not made of "ideograms". Rather, they're made of a mix of symbols for words (lexigrams, which represent a unit of speech, not an idea); sometimes purely phonetic symbols; and sometimes phonetic-semantic symbol groups. That is, sometimes two symbols are combined to record one spoken word: one symbol for a word indicating the semantic category (e.g. the symbol for "mouth"), and one symbol giving the first sound of the specific intended word related to the semantic determiner (e.g. a symbol for a word beginning with the sound that "speaking" begins with). This, one speculates, is the result of accumulating small solutions to small problems of the form: record such-and-such speech, using the orthographic material at hand. I don't see a clear distinction between [just using the ideas (symbols) that one already has] and [inventing new symbols]. The little, operational solutions found in recording speech--the progressive defining, standardizing, recursively building, streamlining, combining, disambiguating, and other tendencies of the mental operation of writing down speech so that others can recover the speech--add up to new scripts. In an extreme form, it's believed that Canaanite slaves used the acrophonic principle (a pure application of the rebus principle) to record their speech in the form of a consonantal alphabet.
• When faced with a problem, the mind's motion [rearrange oneself to efficiently solve this problem] may be just one sort of motion, which gives rise both to what we'd recognize as [rearrange preexisting elements in preexisting ways] and to what we'd recognize as [call forth new elements, including new arrangements].
• Provisional elements tend to be set up to be open to revision in the course of operation.
  • For example, if you're manipulating some strange objects that you've so far only identified as [those floppy things sticking out of my shoe], it's natural to add the understanding [pulling outward on one floppy thing pulls inward on the other floppy thing].
  • For another example, the concept of a mathematical function was continually revised (expanded, abstracted), often in ways deriving from the force of straightforward intuitions about functions, but always in a way that kept a lot of the character of previous ideas of functions (e.g., composability). (See "The concept of function up to the middle of the 19th century", by A. P. Youschkevitch; sci-hub.ru pdf.)
• The availability of the criteria provided by an interesting context is already a lot of the work of creativity.
• This isn't to say that crystallized and fluid intelligence are the same thing, just that pulling them apart would require understanding more.

A strong mind continues its trajectory of creativity

If a mind's operation and creativity can't be separated, then the mind can't operate without also exercising its creativity. A very strong mind became strong by being on a strong trajectory of accumulating novelty through creativity. So as long as a very strong mind continues to operate, it continues forward on its trajectory of creativity, gaining more and more understanding.