In pure math, mathematicians seek "morality", which sounds similar to Ron's string theory conversion stories above. Eugenia Cheng's Mathematics, morally argues: 

I claim that although proof is what supposedly establishes the undeniable truth of a piece of mathematics, proof doesn’t actually convince mathematicians of that truth. And something else does. 

... formal mathematical proofs may be wonderfully watertight, but they are impossible to understand. Which is why we don’t write whole formal mathematical proofs. ... Actually, when we write proofs what we have to do is convince the community that it could be turned into a formal proof. It is a highly sociological process, like appearing before a jury of twelve good men-and-true. The court, ultimately, cannot actually know if the accused actually ‘did it’ but that’s not the point; the point is to convince the jury. Like verdicts in court, our ‘sociological proofs’ can turn out to be wrong—errors are regularly found in published proofs that have been generally accepted as true. So much for mathematical proof being the source of our certainty. Mathematical proof in practice is certainly fallible. 

But this isn’t the only reason that proof is unconvincing. We can read even a correct proof, and be completely convinced of the logical steps of the proof, but still not have any understanding of the whole. Like being led, step by step, through a dark forest, but having no idea of the overall route. We’ve all had the experience of reading a proof and thinking “Well, I see how each step follows from the previous one, but I don’t have a clue what’s going on!” 

And yet... The mathematical community is very good at agreeing what’s true. And even if something
is accepted as true and then turns out to be untrue, people agree about that as well. Why? ...

Mathematical theories rarely compete at the level of truth. We don’t sit around arguing about which theory is right and which is wrong. Theories compete at some other level, with questions about what the theory “ought” to look like, what the “right” way of doing it is. It’s this other level of ‘ought’ that we call morality. ... Mathematical morality is about how mathematics should behave, not just that this is right, this is wrong. Here are some examples of the sorts of sentences that involve the word “morally”, not actual
examples of moral things.

“So, what’s actually going on here, morally?”
“Well, morally, this proof says...”
“Morally, this is true because...”
“Morally, there’s no reason for this axiom.”
“Morally, this question doesn’t make any sense.”
“What ought to happen here, morally?”
“This notation does work, but morally, it’s absurd!”
“Morally, this limit shouldn’t exist at all”
“Morally, there’s something higher-dimensional going on here.”

Beauty/elegance is often the opposite of morality. An elegant proof is often a clever trick, a piece of magic as in Example 6 above, the sort of proof that drives you mad when you’re trying to understand something precisely because it’s so clever that it doesn’t explain anything at all.

Constructiveness is often the opposite of morality as well. If you’re proving the existence of something and you just construct it, you haven’t necessarily explained why the thing exists.

Morality doesn't mean 'explanatory' either. There are so many levels of explaining something. Explanatory to whom? To someone who’s interested in moral reasons. So we haven’t really got anywhere. The same goes for intuitive, obvious, useful, natural and clear, and as Thurston says: “one person’s clear mental image is another person’s intimidation”.

Minimality/efficiency is sometimes the opposite of morality too. Sometimes the most efficient way of proving something is actually the moral way backwards. eg quadratics. And the most minimal way of presenting a theory is not necessarily the morally right way. For example, it is possible to show that a group is a set X equipped with one binary operation / satisfying the single axiom for all x, y, z ∈ X, (x/((((x/x)/y)/z)/(((x/x)/x)/z))) = y. The fact that something works is not good enough to be a moral reason.

Polya’s notion of ‘plausible reasoning’ at first sight might seem to fit the bill because it appears to be about how mathematicians decide that something is ‘plausible’ before sitting down to try and prove it. But in fact it’s somewhat probabilistic. This is not the same as a moral reason. It’s more like gathering a lot of evidence and deciding that all the evidence points to one conclusion, without there actually being a reason necessarily. Like in court, having evidence but no motive.

Abstraction perhaps gets closer to morality, along with ‘general’, ‘deep’, ‘conceptual’. But I would say that it’s the search for morality that motivates abstraction, the search for the moral reason motivates the search for greater generalities, depth and conceptual understanding. ... 

Proof has a sociological role; morality has a personal role. Proof is what convinces society; morality is what convinces us. Brouwer believed that a construction can never be perfectly communicated by verbal or symbolic language; rather it’s a process within the mind of an individual mathematician. What we write down is merely a language for communicating something to other mathematicians, in the hope that they will be able to reconstruct the process within their own mind. When I’m doing maths I often feel like I have to do it twice—once, morally in my head. And then once to translate it into communicable form. The translation is not a trivial process; I am going to encapsulate it as the process of moving from one form of truth to another. 

Transmitting beliefs directly is unfeasible, but the question that does leap out of this is: what about the reason? Why don’t I just send the reason directly to X, thus eliminating the two probably hardest parts of this process? The answer is that a moral reason is harder to communicate than a proof. The key characteristic about proof is not its infallibility, not its ability to convince but its transferability. Proof is the best medium for communicating my argument to X in a way which will not be in danger of ambiguity, misunderstanding, or defeat. Proof is the pivot for getting from one person to another, but some translation is needed on both sides. So when I read an article, I always hope that the author will have included a reason and not just a proof, in case I can convince myself of the result without having to go to all the trouble of reading the fiddly proof. 

That last part is quite reminiscent of what the late Bill Thurston argued in his classic On proof and progress in mathematics:

Mathematicians have developed habits of communication that are often dysfunctional. Organizers of colloquium talks everywhere exhort speakers to explain things in elementary terms. Nonetheless, most of the audience at an average colloquium talk gets little of value from it. Perhaps they are lost within the first 5 minutes, yet sit silently through the remaining 55 minutes. Or perhaps they quickly lose interest because the speaker plunges into technical details without presenting any reason to investigate them. At the end of the talk, the few mathematicians who are close to the field of the speaker ask a question or two to avoid embarrassment. 

This pattern is similar to what often holds in classrooms, where we go through the motions of saying for the record what we think the students “ought” to learn, while the students are trying to grapple with the more fundamental issues of learning our language and guessing at our mental models. Books compensate by giving samples of how to solve every type of homework problem. Professors compensate by giving homework and tests that are much easier than the material “covered” in the course, and then grading the homework and tests on a scale that requires little understanding. We assume that the problem is with the students rather than with communication: that the students either just don’t have what it takes, or else just don’t care. 

Outsiders are amazed at this phenomenon, but within the mathematical community, we dismiss it with shrugs. 

Much of the difficulty has to do with the language and culture of mathematics, which is divided into subfields. Basic concepts used every day within one subfield are often foreign to another subfield. Mathematicians give up on trying to understand the basic concepts even from neighboring subfields, unless they were clued in as graduate students. 

In contrast, communication works very well within the subfields of mathematics. Within a subfield, people develop a body of common knowledge and known techniques. By informal contact, people learn to understand and copy each other’s ways of thinking, so that ideas can be explained clearly and easily. 

Mathematical knowledge can be transmitted amazingly fast within a subfield. When a significant theorem is proved, it often (but not always) happens that the solution can be communicated in a matter of minutes from one person to another within the subfield. The same proof would be communicated and generally understood in an hour talk to members of the subfield. It would be the subject of a 15- or 20-page paper, which could be read and understood in a few hours or perhaps days by members of the subfield. 

Why is there such a big expansion from the informal discussion to the talk to the paper? One-on-one, people use wide channels of communication that go far beyond formal mathematical language. They use gestures, they draw pictures and diagrams, they make sound effects and use body language. Communication is more likely to be two-way, so that people can concentrate on what needs the most attention. With these channels of communication, they are in a much better position to convey what’s going on, not just in their logical and linguistic facilities, but in their other mental facilities as well. 

In talks, people are more inhibited and more formal. Mathematical audiences are often not very good at asking the questions that are on most people’s minds, and speakers often have an unrealistic preset outline that inhibits them from addressing questions even when they are asked. 

In papers, people are still more formal. Writers translate their ideas into symbols and logic, and readers try to translate back. 

Why is there such a discrepancy between communication within a subfield and communication outside of subfields, not to mention communication outside mathematics? Mathematics in some sense has a common language: a language of symbols, technical definitions, computations, and logic. This language efficiently conveys some, but not all, modes of mathematical thinking. Mathematicians learn to translate certain things almost unconsciously from one mental mode to the other, so that some statements quickly become clear. Different mathematicians study papers in different ways, but when I read a mathematical paper in a field in which I’m conversant, I concentrate on the thoughts that are between the lines. I might look over several paragraphs or strings of equations and think to myself “Oh yeah, they’re putting in enough rigamarole to carry such-and-such idea.” When the idea is clear, the formal setup is usually unnecessary and redundant—I often feel that I could write it out myself more easily than figuring out what the authors actually wrote. It’s like a new toaster that comes with a 16-page manual. If you already understand toasters and if the toaster looks like previous toasters you’ve encountered, you might just plug it in and see if it works, rather than first reading all the details in the manual. 

People familiar with ways of doing things in a subfield recognize various patterns of statements or formulas as idioms or circumlocution for certain concepts or mental images. But to people not already familiar with what’s going on the same patterns are not very illuminating; they are often even misleading. The language is not alive except to those who use it.

Thurston's personal reflections below on the sociology of proof exemplify the search for mathematical morality instead of fully formally rigorous correctness. I remember being disquieted upon first reading "There were published theorems that were generally known to be false" a long time ago: 

When I started as a graduate student at Berkeley, I had trouble imagining how I could “prove” a new and interesting mathematical theorem. I didn’t really understand what a “proof” was. 

By going to seminars, reading papers, and talking to other graduate students, I gradually began to catch on. Within any field, there are certain theorems and certain techniques that are generally known and generally accepted. When you write a paper, you refer to these without proof. You look at other papers in the field, and you see what facts they quote without proof, and what they cite in their bibliography. You learn from other people some idea of the proofs. Then you’re free to quote the same theorem and cite the same citations. You don’t necessarily have to read the full papers or books that are in your bibliography. Many of the things that are generally known are things for which there may be no known written source. As long as people in the field are comfortable that the idea works, it doesn’t need to have a formal written source. 

At first I was highly suspicious of this process. I would doubt whether a certain idea was really established. But I found that I could ask people, and they could produce explanations and proofs, or else refer me to other people or to written sources that would give explanations and proofs. There were published theorems that were generally known to be false, or where the proofs were generally known to be incomplete. Mathematical knowledge and understanding were embedded in the minds and in the social fabric of the community of people thinking about a particular topic. This knowledge was supported by written documents, but the written documents were not really primary. 

I think this pattern varies quite a bit from field to field. I was interested in geometric areas of mathematics, where it is often pretty hard to have a document that reflects well the way people actually think. In more algebraic or symbolic fields, this is not necessarily so, and I have the impression that in some areas documents are much closer to carrying the life of the field. But in any field, there is a strong social standard of validity and truth. Andrew Wiles’s proof of Fermat’s Last Theorem is a good illustration of this, in a field which is very algebraic. The experts quickly came to believe that his proof was basically correct on the basis of high-level ideas, long before details could be checked. This proof will receive a great deal of scrutiny and checking compared to most mathematical proofs; but no matter how the process of verification plays out, it helps illustrate how mathematics evolves by rather organic psychological and social processes.

I would also comment that, if the environment was so chaotic that roughly everything important to life could not be modeled—if general-purpose modeling ability was basically useless—then life would not have evolved that ability, and "intelligent life" probably wouldn't exist.

Matt Leifer, who works in quantum foundations, espouses a view that's probably more extreme than Eric Raymond's above to argue why the effectiveness of math in the natural sciences isn't just reasonable but expected-by-construction. In his 2015 FQXi essay Mathematics is Physics Matt argued that 

... mathematics is a natural science—just like physics, chemistry, or biology—and that this can explain the alleged “unreasonable” effectiveness of mathematics in the physical sciences. 

The main challenge for this view is to explain how mathematical theories can become increasingly abstract and develop their own internal structure, whilst still maintaining an appropriate empirical tether that can explain their later use in physics. In order to address this, I offer a theory of mathematical theory-building based on the idea that human knowledge has the structure of a scale-free network and that abstract mathematical theories arise from a repeated process of replacing strong analogies with new hubs in this network. 

This allows mathematics to be seen as the study of regularities, within regularities, within . . . , within regularities of the natural world. Since mathematical theories are derived from the natural world, albeit at a much higher level of abstraction than most other scientific theories, it should come as no surprise that they so often show up in physics. 

... mathematical objects do not refer directly to things that exist in the physical universe. As the formalists suggest, mathematical theories are just abstract formal systems, but not all formal systems are mathematics. Instead, mathematical theories are those formal systems that maintain a tether to empirical reality through a process of abstraction and generalization from more empirically grounded theories, aimed at achieving a pragmatically useful representation of regularities that exist in nature. 

(Matt notes as an aside that he's arguing for precisely the opposite of Tegmark's MUH.) 

Why "scale-free network"?

It is common to view the structure of human knowledge as hierarchical... The various attempts to reduce all of mathematics to logic or arithmetic reflect a desire view mathematical knowledge as hanging hierarchically from a common foundation. However, the fact that mathematics now has multiple competing foundations, in terms of logic, set theory or category theory, indicates that something is wrong with this view. 

Instead of a hierarchy, we are going to attempt to characterize the structure of human knowledge in terms of a network consisting of nodes with links between them... Roughly speaking, the nodes are
supposed to represent different fields of study. This could be done at various levels of detail. ... Next, a link should be drawn between two nodes if there is a strong connection between the things they represent. Again, I do not want to be too precise about what this connection should be, but examples would include an idea being part of a wider theory, that one thing can be derived from the other, or that there exists a strong direct analogy between the two nodes. Essentially, if it has occurred to a human being that the two things are strongly related, e.g. if it has been thought interesting enough to do something like publish an academic paper on the connection, and the connection has not yet been explained in terms of some intermediary theory, then there should be a link between the corresponding nodes in the network.

If we imagine drawing this network for all of human knowledge then it is plausible that it would have
the structure of a scale-free network. Without going into technical details, scale-free networks have a small number of hubs, which are nodes that are linked to a much larger number of nodes than the average. This is a bit like the 1% of billionaires who are much richer than the rest of the human population. If the knowledge network is scale-free then this would explain why it seems so plausible that knowledge is hierarchical. In a university degree one typically learns a great deal about one of the hubs, e.g. the hub representing fundamental physics, and a little about some of the more specialized subjects that hang from it. As we get ever more specialized, we typically move away from our starting hub towards more obscure nodes, which are nonetheless still much closer to the starting hub than to any other hub. The local part of the network that we know about looks much like a hierarchy, and so it is not surprising that physicists end up thinking that everything boils down to physics whereas sociologists end up thinking that everything is a social construct. In reality, neither of these views is right because the global structure of the network is not a hierarchy.

As a naturalist, I should provide empirical evidence that human knowledge is indeed structured as a scale-free network. The best evidence that I can offer is that the structure of pages and links on the Word Wide Web and the network of citations to academic papers are both scale free [13]. These are, at best, approximations of the true knowledge network. ... However, I think that these examples provide evidence that the information structures generated by a social network of finite beings are typically scale-free networks, and the knowledge network is an example of such a structure.

As an aside, Matt's theory of theory-building explains (so he claims) what mathematical intuition is about: "intuition for efficient knowledge structure, rather than intuition about an abstract mathematical world". 

So what? How does this view pay rent?

Firstly, in network language, the concept of a “theory of everything” corresponds to a network with one enormous hub, from which all other human knowledge hangs via links that mean “can be derived from”. This represents a hierarchical view of knowledge, which seems unlikely to be true if the structure of human knowledge is generated by a social process. It is not impossible for a scale-free network to have a hierarchical structure like a branching tree, but it seems unlikely that the process of knowledge growth would lead uniquely to such a structure. It seems more likely that we will always have several competing large hubs and that some aspects of human experience, such as consciousness and why we experience a unique present moment of time, will be forever outside the scope of physics. 

Nonetheless, my theory suggests that the project of finding higher level connections that encompass more of human knowledge is still a fruitful one. It prevents our network from having an unwieldy number of direct links, allows us to share more common vocabulary between fields, and allows an individual to understand more of the world with fewer theories. Thus, the search for a theory of everything is not fruitless; I just do not expect it to ever terminate.

Secondly, my theory predicts that the mathematical representation of fundamental physical theories will continue to become increasingly abstract. The more phenomena we try to encompass in our fundamental theories, the further the resulting hubs will be from the nodes representing our direct sensory experience. Thus, we should not expect future theories of physics to become less mathematical, as they are generated by the same process of generalization and abstraction as mathematics itself. 

Matt further develops the argument that the structure of human knowledge being networked-not-hierarchical implies that the idea that there is a most fundamental discipline, or level of reality, is mistaken in Against Fundamentalism, another FQXi essay published in 2018. 

Minimizing friction is surprisingly difficult. I keep plain-text notes in a hierarchical editor (cherrytree), but even that feels too complicated sometimes. This is not just about the tool... what you actually need is a combination of the tool and the right way to use it.

(Every tool can be used in different ways. For example, suppose you write a diary in MS Word. There are still options such as "one document per day" or "one very long document for all", and things in between like "one document per month", which all give different kinds of friction. The one megadocument takes too much time to load. It is more difficult to search in many small documents. Or maybe you should keep your current day in a small document, but once in a while merge the previous days into the megadocument? Or maybe switch to some application that starts faster than MS Word?)

Forgetting is an important part. Even if you want to remember forever, you need some form of deprioritizing. Something like "pages you haven't used for months will get smaller, and if you search for keywords, they will be at the bottom of the result list". But if one of them suddenly becomes relevant again, maybe the connected ones become relevant, too? Something like associations in brain. The idea is that remembering the facts is only a part of the problem; making the relevant ones more accessible is another. Because searching in too much data is ultimately just another kind of friction.

It feels like a smaller version of the internet. Years ago, the problem used to be "too little information", now the problem is "too much information, can't find the thing I actually want".

Perhaps a wiki, where the pages could get flagged as "important now" and "unimportant"? Or maybe, important for a specific context? And by default, when you choose a context, you would only see the important pages, and the rest of that only if you search for a specific keyword or follow a grey link. (Which again would require some work creating and maintaining the contexts. And that work should also be as frictionless as possible.)

@dkl9 wrote a very eloquent and concise piece arguing in favor of ditching "second brain" systems in favor of SRSs (Spaced Repetition Systems, such as Anki).

Try as you might to shrink the margin with better technology, recalling knowledge from within is necessarily faster and more intuitive than accessing a tool. When spaced repetition fails (as it should, up to 10% of the time), you can gracefully degrade by searching your SRS' deck of facts.

If you lose your second brain (your files get corrupted, a cloud service shuts down, etc), you forget its content, except for the bits you accidentally remember by seeing many times. If you lose your SRS, you still remember over 90% of your material, as guaranteed by the algorithm, and the obsolete parts gradually decay. A second brain is more robust to physical or chemical damage to your first brain. But if your first brain is damaged as such, you probably have higher priorities than any particular topic of global knowledge you explicitly studied.

I write for only these reasons:

• to help me think
• to communicate and teach (as here)
• to distill knowledge to put in my SRS
• to record local facts for possible future reference

Linear, isolated documents suffice for all those purposes. Once you can memorise well, a second brain becomes redundant tedium.

I like to think of learning and all of these things as self-contained smaller self-contained knowledge trees. Building knowledge trees that are cached, almost like creatin zip files and systems where I store a bunch of zip files similar to what Elizier talks about in The Sequences. 

Like when you mention the thing about Nielsen on linear algebra it opens up the entire though tree there. I might just get the association to something like PCA and then I think huh, how to ptimise this and then it goes to QR-algorithms and things like a householder matrix and some specific symmetric properties of linear spaces...

If I have enough of these in an area then I might go back to my anki for that specific area. Like if you think from the perspective of schedulling and storage algorithms similar to what is explored in algorithms to live by you quickly understand that the magic is in information compression and working at different meta-levels. Zipped zip files with algorithms to expand them if need be. Dunno if that makes sense, agree with the exobrain creep that exists though.

It's very funny that Rorschach linguistic ability is totally unremarkable comparing to modern LLMs.

How interesting, I was curious about copyright etc but this is annotated by the author himself!

The parts of the science I understand were all quite plausible (mind duplication/fractioning and motivations for doing so).

Beyond the accuracy of the science, this was one of the most staggeringly imaginative and beautifully written scifi books I've ever read. It's for a very particular audience, but if you're here you might be that audience. If you are, this might be the best book you've read.

Attention conservation notice: 3,000+ words of longform quotes by various folks on the nature of personal identity in a posthuman future, and hiveminds / clans

As an aside, one of the key themes running throughout the Quantum Thief trilogy is the question of how you might maintain personal identity (in the pragmatic security sense, not the philosophical one) in a future so posthuman that minds can be copied and forked indefinitely over time. To spoil Hannu's answer: 

... Jean & the Sobornost Founders & the zoku elders are all defined by what, at their core, they want. Anyone who wants the same thing is, for all (their) intents and purposes, the same person as them; because they want the same unchanging things, they can be trusted as the original. The ‘Founder codes’, and Jean’s final password to unlock his sealed memories, are all memories of what defines their wants: the Founder Sumanguru wants blood & fire & electricity & screaming children, and enemies to destroy; the Founder Chen recall the trauma of livestreaming their father’s assassination, remaining eternally resolved that the last enemy that shall be defeated is death; while seared into the minds of the Founder Joséphine Pellegrinis is the final thought of their founder, her desperate dying wish that her lover Jean le Flambeur someday return to her… (And the zoku elders want to empower their zoku clans.)

But even personal identity frays under the power of time: given freedom to change, sooner or later, like the Ship of Theseus, the mind which sets out is not the mind which arrives. So the price of immortality must be that one cannot change: one is condemned to want the same things, forever.^7⁠ (“There is no prison, except in your mind.”) Joséphine Pellegrini cannot stop seeking after her lost Jean—nor can Jean stop his thieving nor trying to escape her, because le Flambeur, what does Jean le Flambeur remember?

I take Anders Sandberg's answer to be on the other end of this spectrum; he doesn't mind changing over time such that he might end up wanting different things: 

Anders Sandberg: I think one underappreciated thing is that if we can survive for a very long time individually, we need to reorganise our minds and memories in interesting ways. There is a kind of standard argument you sometimes hear if you’re a transhumanist — like I am — that talks about life extension, where somebody cleverly points out that you would change across your lifetime. If it’s long enough, you will change into a different person. So actually you don’t get an indefinitely extended life; you just get a very long life thread. I think this is actually an interesting objection, but I’m fine with turning into a different future person. Anders Prime might have developed from Anders in an appropriate way — we all endorse every step along the way — and the fact that Anders Prime now is a very different person is fine. And then Anders Prime turns into Anders Biss and so on — a long sequence along a long thread.

(I have mixed feelings about Anders' take: I have myself changed so profoundly since youth that that my younger self would not just disendorse but be horrified by the person I am now, yet I did endorse every step along the way, and current-me still does upon reflection (but of course I do). Would current-me also endorse a similar degree of change going forward, even subject to every step being endorsed by the me right before change? Most likely not, perhaps excepting changes towards some sort of reflective equilibrium.) 

I interpret Holden Karnofsky's take to be somewhere in between, perhaps closer to Hannu's answer. Holden remarked that he doesn't find most paradoxical thought experiments about personal identity (e.g. "Would a duplicate of you be "you?"" or "If you got physically destroyed and replaced with an exact duplicate of yourself, did you die?") all that confounding because his personal philosophy on "what counts as death" dissolves them, and that his philosophy is simple, comprising just 2 aspects: constant replacement ("in an important sense, I stop existing and am replaced by a new person each moment") and kinship with future selves. Elaborating on the latter:

My future self is a different person from me, but he has an awful lot in common with me: personality, relationships, ongoing projects, and more. Things like my relationships and projects are most of what give my current moment meaning, so it's very important to me whether my future selves are around to continue them.

So although my future self is a different person, I care about him a lot, for the same sorts of reasons I care about friends and loved ones (and their future selves).³

If I were to "die" in the common-usage (e.g., medical) sense, that would be bad for all those future selves that I care about a lot.⁴

... 

[One of the pros of this view]

It seems good that when I think about questions like "Would situation __ count as dying?", I don't have to give answers that are dependent on stuff like how fast the atoms in my body turn over - stuff I have basically never thought about and that doesn't feel deeply relevant to what I care about. Instead, when I think about whether I'd be comfortable with something like teleportation, I find myself thinking about things I actually do care about, like my life projects and relationships, and the future interactions between me and the world.

Richard Ngo goes in a different direction with the "personal identity in a posthuman future" question:

Rob Wiblin: ... one of the non-AI blog posts you’ve written, which I really enjoyed reading this week when I was prepping for the conversation, is called Characterising utopia. ... Some of the shifts that you envisaged wouldn’t be super surprising. Like we could reduce the amount that people experience physical pain, and we could make people be a lot more energetic and a lot more cheerful. But you had a section called “Contentious changes.” What are some of the contentious changes, or possible changes, that you envisage in a utopia?

Richard Ngo: One of the contentious changes here is to do with individualism, and how much more of it or less of it we have in the future than we have today. Because we’ve been on this trend towards much more individualistic societies, where there are fewer constraints on what people do that are externally imposed by society.

I could see this trend continuing, but I could also see it going in the opposite direction. Maybe, for example, in a digital future, we’ll be able to make many copies of ourselves, and so this whole concept of my “personal identity” starts to shift a little bit and maybe I start to think of myself as not just one individual, but a whole group of individuals or this larger entity. And in general, it feels like being part of a larger entity is really meaningful to people and really shapes a lot of people’s lives, whether that’s religion, whether that’s communities, families, things like that.

The problem historically has just been that you don’t get to choose it — you just have to get pushed into this entity that maybe isn’t looking out for your best interests. So it feels interesting to me to wonder if we can in fact design these larger entities or larger superorganisms that are really actually good for the individuals inside, as well as providing this more cohesive structure for them. Is that actually something we want? Would I be willing to lose my individuality if I were part of this group of people who were, for example, reading each other’s minds or just having much less privacy than we have today, if that was set up in such a way that I found it really fulfilling and satisfying?

I really don’t know at all, but it seems like the type of question that is really intriguing and provides a lot of scope for thinking about how technology could just change the ways in which we want to interact with each other.

Rob Wiblin: I’m so inculcated into the individualist culture that the idea slightly makes my skin crawl thinking about any of this stuff. But I think if you tried to look objectively at what has caused human wellbeing throughout history, then it does seem like a somewhat less individualistic culture, where people have deeper ties and commitments to one another, maybe that is totally fine — and I’ve just drunk the Kool-Aid thinking that being an atomised individual is so great.

Richard Ngo: If you know the book, The WEIRDest People in the World, which describes the trend towards individualism and weaker societal ties, I think the people in our circles are the WEIRDest people of the WEIRDest people in the world — where “WEIRD” here is an acronym meaning “Western, educated, industrialised, rich, and democratic,” not just “weird.” So we are the WEIRDest people of the WEIRDest countries. And then you’re not a bad candidate for the WEIRDest person in the WEIRDest community in the WEIRDest countries that we currently have, Rob. So I’m not really too surprised by that.

(I thought it was both interesting and predictable that Rob would find the idea discomfiting; coming from a non-WEIRD culture, I found Richard's idea immediately attractive and aesthetically "right".)

Richard gives an fictional example of what this might look like from a first-person perspective in his recent short story The Gentle Romance --- if you're reading this Richard, do let me know if you want this removed:

As ze reconnects more deeply with zir community, that oceanic sense of oneness arises more often. Some of zir friends submerge themselves into a constant group flow state, rarely coming out. Each of them retains their individual identity, but the flows of information between them increase massively, allowing them to think as a single hivemind. Ze remains hesitant, though. The parts of zir that always wanted to be exceptional see the hivemind as a surrender to conformity. But what did ze want to be exceptional for? Reflecting, ze realizes that zir underlying goal all along was to be special enough to find somewhere ze could belong. The hivemind allows zir to experience that directly, and so ze spends more and more time within it, enveloped in the warm blanket of a community as close-knit as zir own mind.

Outside zir hivemind, billions of people choose to stay in their physical bodies, or to upload while remaining individuals. But over time, more and more decide to join hiveminds of various kinds, which continue to expand and multiply. By the time humanity decides to colonize the stars, the solar system is dotted with millions of hiveminds. A call goes out for those willing to fork themselves and join the colonization wave. This will be very different from anything they’ve experienced before — the new society will be designed from the ground up to accommodate virtual humans. There will be so many channels for information to flow so fluidly between them that each colony will essentially be a single organism composed of a billion minds.

Ze remembers loving the idea of conquering the stars — and though ze is a very different person now, ze still feels nostalgic for that old dream. So ze argues in favor when the hivemind debates whether to prioritize the excitement of exploration over the peacefulness of stability. It’s a more difficult decision than any the hivemind has ever faced, and no single satisfactory resolution emerges. So for the first time in its history, the hivemind temporarily fractures itself, giving each of its original members a chance to decide on an individual basis whether they’ll go or stay.

I think Richard's notion of 'hivemind' is cousin to Robin Hanson's 'clan' from Age of Em (although unlike Richard's lovely story, Hanson's depiction of an em-filled future has never stopped seeming dystopian to me, Hanson's protestation to the contrary that "[readers repelled by aspects of the em era should] try hard to see this world from its residents’ point of view, before disowning these their plausible descendants", albeit far more granular, comprehensive and first-principles-based): 

The set of all em copies of the same original human constitutes a “clan.” Most wages go to the 1000 most productive clans, who are each known by one name, like “John,” who know each other very well, and who discriminate against less common clans. Compared with people today, ems are about as elite as billionaires, heads of state, and Olympic gold medalists. The em world is more competitive than ours in more quickly eliminating less productive entities and practices. This encourages more job punishment, less product variety and identity enhancement, and more simple functionality. Because they are more productive, ems tend to be married, religious, smart, gritty, mindful, extraverted, conscientiousness, agreeable, non-neurotic, and morning larks.

Many myths circulate about factors that increase economic growth rates. For example, the fact that ems can run faster than humans should not much increase growth. Even so, the em economy grows faster than does ours because of stronger competition, computers mattering more, and especially because factories can make labor as fast as non-labor capital. An em economy doubling time estimate of a few weeks comes from the time for factories to duplicate their mass today, and from the historical trend in growth rates. In response, capital becomes less durable, and one-time-use products become more attractive. Clans become a unit of finance, private firms and hostile takeovers get more support, and asset prices more closely approximate the predictions derived from strong financial competition.

Ems trust their clans more than we trust families or identical twins. So clans are units of finance, liability, politics, labor negotiations, and consumer purchasing. To promote unity, clans avoid members arguing or competing. Em firms are larger, better managed, put more effort into coordination, have more specific job roles, focus more on costs relative to novelty, and have higher market shares and lower markups. Clan reputations and clans buying into firms promotes clan-firm trust, which supports locating employees at firms, using distinctive work styles, and focusing more on being useful instead of gaming firm evaluation systems. Em work teams tend to have similar social-category features like age but a diversity of information sources and thinking styles. In mass-labor markets, ems are created together, end or retire together, almost never break up, and mostly socialize internally. In niche-labor markets, associates coordinate less regarding when they are created or retire.

Faster ems have many features that mark them as higher status, and the clumping of speeds creates a class system of distinct status levels. Strong central rulers are more feasible for ems, as leaders can run faster, put spurs in high-trust roles, and use safes to reassure wary citizens. Decision markets can help advise key government decisions, while combinatorial auctions can help to make complex interdependent allocations. The em world selects for personalities good at governing that same personality. Competitive clans and cities may commit to governing via decision markets that promote profit or long-term influence. One em one vote works badly, but speed-weighted voting seems feasible, although it requires intrusive monitoring. Shifting coalitions of em clans may dominate the politics of em firms and cities, inducing high costs of lobbying and change. Ems may try many policies to limit such clan coalition politics.

As ems don’t need sex to reproduce, sex is left more to individual choice, and may be suppressed as in eunuchs. But demand for sex and romantic pair-bonding likely persists, as do many familiar gendered behavioral patterns. A modestly unequal demand for male versus female workers can be accommodated via pairs whose partners run at different speeds, or who use different ratios of spurs to other workers. Ems have spectacularly good looks in virtual reality, and are very accomplished. Open-source em lovers give all ems an attractive lower bound on relation quality. Clan experience helps ems guess who are good receptive matches. Having only one em from each clan in each social setting avoids complicating relations.

Ems show off their abilities and loyalties, although less than we do because ems are poorer and better-known to each other. Because speed is easy to pay for, ems show off more via clever than fast speech. Celebrities matter less to ems, and it is easy to meet with a celebrity, but hard to get them to remember you. Clans coordinate to jointly signal shared features like intelligence, drive, and fame. Clans fund young ems to do impressive things, about which many older copies can brag. Innovation may matter less for em intellectuals. Mind-theft inspires great moral outrage and charity efforts. Secure in identifying with their clan, most ems focus personal energy more on identifying with their particular job, team, and associates. It isn’t clear if em identity degrades continuously or discretely as copies get more different. Copy-events are identity-defining, and newly copied teams quickly create distinct team cultures.

Ems are likely to reverse our recent trend away from religion and overt rituals, perhaps via more group singing. Traditional religions can continue, but need doctrinal clarifications on death and sins of copies. Like high stress workers today, em work groups pushed to their limits swear, insult, and tease. Ems deal with a wider range of mind opacity and transparency, allowing mind reading within teams, but manipulating expressions to hide from outsiders. Clans can offer members life-coaching via voices in their heads, using statistics from similar copies, but teams may create unique cultures which limit the usefulness of that. Avoiding direct meetings helps clans bond better. Em relations are often in the context of similar relations between copies. At work, ems try more to make relations similar, to gain from learning and scale economics. But friends keep relations more different, to emphasize loyalty and natural feelings.

Em culture emphasizes industriousness, work and long-term orientations, and low context attitudes toward rules and communication. Being poorer, ems tend towards farmer/conservative values, relative to forager/liberal values. So ems more value honor, order, hierarchy, religion, work, and less value sharing, consensus, travel, leisure, and variety. Sex attitudes stay more forager-like, however. Ems are divided like we are by geographic region, young versus old, male versus female, rich versus poor, and city center versus periphery. Ems also divide by varying speeds, physical versus virtual work, remembering the human era versus not, and large versus small clans. Ems travel to visit or swap with other copies of themselves. An exotic travel destination is other speed cultures. Like us, ems tell stories of conflict and norm violations, set in ancestral situations. Stories serve as marketing, with many characters coming from well-known clans. Em stories have less death and fast-action.

The short story The Epiphany of Gliese 581 by Fernando Borretti has something of the same vibe as Rajaniemi's QT trilogy; Borretti describes it as inspired by Orion's Arm and the works of David Zindell. Here's a passage describing a flourishing star system already transformed by weakly posthuman tech:

The world outside Susa was a lenticular cloud of millions of lights, a galaxy in miniature, each a world unto itself. There were clusters of green lights that were comets overgrown with vacuum trees, and plant and animal and human life no Linnaeus would recognize. There were points of dull red light, the reversible computers where bodyless people lived. And there were arcs of blue that were ring habitats: ribbons tied end-to-end, holding concave ocean, and the oceans held continents, islands, mountain ranges, rivers, forests and buried ruins, endless forms of life, cities made of glass, paradise regained. All this had been inanimate dust and cratered wasteland, which human hands had made into an oasis in the sky, where quadrillions live who will never die.

The posthumans who live there called it Ctesiphon. And at times they call it paradise, after the Persian word for garden.

And at the center of the oasis there was a star that travelled backwards across the H-R diagram: already one one-hundredth of it had been whittled away; made into a necklace of artificial gas giants in preparation for the end of time; or sent through reactors where disembodied chemists made protons into carbon, oxygen, lithium and sodium, the vital construction material. And in time nothing would be left but a dim red ember encircled by cryojovian fuel depots. And the habitats would be illuminated by electric diodes.

Another star system, this time still being transformed: 

Wepwawet was a dull red star, ringed by water droplets the size of mountains, where some two hundred billion people lived who breathed water. There was a planet made of stone shrouded in steam, and a train of comets, aimed by human hands from beyond the frostline, delivered constant injections of water. When the vapour condensed there would be ocean, and the shapers would get to work on the continents. Other Earths like this had been cast, like seeds, across the entire breadth of the cosmos.

The system was underpopulated: resources were abundant and people were few, and they could bask in the sun and, for a time, ignore the prophecies of Malthus, whose successors know in time there won’t be suns. 

This was the first any of them had seen of nature. Not the landscaped, continent-sized gardens of Ctesiphon, where every stone had been set purposefully and after an aesthetic standard, but nature before human hands had redeemed it: an endless, sterile wasteland. The sight of scalding, airless rocks disturbed them.

Linking to a previous comment: 3,000+ words of longform quotes by various folks on the nature of personal identity in a posthuman future, and hiveminds / clans, using Hannu Rajaniemi's Quantum Thief trilogy as a jumping-off point.

Hal Finney's reflections on the comprehensibility of posthumans, from the Vinge singularity discussion which took place on the Extropians email list back in the day:

Date: Mon, 7 Sep 1998 18:02:39 -0700
From: Hal Finney 
Message-Id: <199809080102.SAA02658@hal.sb.rain.org>
To: extropians@extropy.com
Subject: Singularity: Are posthumans understandable?

[This is a repost of an article I sent to the list July 21.]

It's an attractive analogy that a posthuman will be to a human as a human is to an insect.  This suggests that any attempt to analyze or understand the behavior of post-singularity intelligence is as hopeless as it would be for an insect to understand human society. Since insects clearly have essentially no understanding of humans, it would follow by analogy that we can have no understanding of posthumans.

On reflection, though, it seems that it may be an oversimplification to say that insects have no understanding of humans.  The issue is complicated by the fact that insects probably have no "understanding" at all, as we use the term.  They may not even be conscious, and may be better thought of as nature's robots, of a similar level of complexity as our own industrial machines.  Since insects do not have understanding, the analogy to humans does not work very well.  If we want to say that our facility for understanding will not carry over into the posthuman era, we need to be able to say that insect's facility for would not work when applied to humans.

What we need to do is to translate the notion of "understanding" into something that insects can do.  That makes the analogy more precise and improves the quality of the conclusions it suggests.

It seems to me that while insects do not have "understanding" as we do, they do nevertheless have a relatively detailed model of the world which they interact with.  Even if they are robots, programmed by evolution and driven by unthinking instinct, still their programming embodies a model of the world.  A butterfly makes its way to flowers, avoides predators, knows when it is hungry or needs to rest.  These decisions may be made unconsciously like a robot, but they do
represent a true model of itself and of the world.

What we should ask, then, is whether insect's model of the world can be successfully used to predict the behavior of humans, in the terms captured by the model itself.  Humans are part of the world that insects must deal with.  Are they able to successfully model human behavior at the level they are able to model other aspects of the world, so that they can thrive alongside humanity?

Obviously insects do not predict many aspects of human behavior. Still, in terms of the level of detail that they attempt to capture, I'd say they are reasonably effective.  Butterflies avoid large animals, including humans.  Some percentage of human-butterfly interactions would involve attempts by the humans to capture the butterflies, and so the butterflies' avoidance instinct represents a success of their model.  Similarly for many other insects for whom the extent of their model of humans is as "possible threat, to be avoided".

Other insects have historically thrived in close association with humans, such as lice, fleas, ants, roaches, etc.  Again, without attempting to predict the full richness of human behavior, their models are successful in expressing those aspects which they care about, so that they have been able to survive, often to the detriment of the human race.

If we look at the analogy in this way, it suggests that we may expect to be able to understand some aspects of posthuman behavior, without coming anywhere close to truly understanding and appreciating the full power of their thoughts.  Their mental life may be far beyond anything we can imagine, but we could still expect to draw some simple conclusions about how they will behave, things which are at the level which we can understand.  Perhaps Robin's reasoning based on fundamental principles of selection and evolution would fall into this category.

We may be as ants to the post singularity intelligences, but even so, we may be able to successfully predict some aspects of their behavior, just as ants are able to do with humans.