I started collecting a list of once “secret” documents that were, in one way or another, broadly released (whether it be through legal process, government disclosures, leaks, or the ideas themselves eventually turned into books):
The analysis done by Deutsche Bank investors to predict the 2008 financial crisis (dramatized as the Jenga scene from the movie “The Big Short”)
A consulting project completed by BCG for the city of Dallas on their loose dog problem
Deeply insightful tech emails surfaced through court cases written by people like Bill Gates, Jeff Bezos, Mark Zuckerberg, etc capturing some of their strategies, views, or ideas
The book, Bulletproof Problem Solving started as a professional development document within McKinsey and was eventually refined and published
With all of these examples, we can see that significant human capital is funneled into fields like tech, finance, and consulting (or law, startups, VC, government/statecraft, etc), and it is therefore unsurprising that much of the work produced there is made to be valuable. In fact, it is very valuable, but it is largely inaccessible to external audiences.
This also happens more broadly; in any human endeavor involving any intellectual effort at all.
The takeaway is that there is an incomprehensible amount of amazing work being done that is nearly, or even completely, impossible for others to find.
This is information dark matter.
It’s out there, and now we know it’s out there, but it is not something we can go and meaningfully interact with.
And while information created for internal company use is just one example, I began to consider where else we might find information dark matter and how we can make it accessible.
Why is this important?
It’s simple: with more information accessible to us, the more we can expand our intellectual arsenals for tackling bigger problems.
To solve this problem of information dark matter, the questions I hope to answer are:
What are the main types of information dark matter?
Where can they be found?
How can we more effectively make the information accessible?
To start answering the first question, I wanted to build a taxonomy starting with first principles. That meant starting my exploration from where all information and ideas emerge, the brain.
Unproduced information
The first type of information dark matter is the kind stuck in someone’s head.
A beautiful song, a lovely poem, a novel product, an innovative process, and so many other things all amount to precisely...
Nothing. When they are unproduced.
Ideas in your head might be incredibly valuable, but due to the very fact that they are not stored in a stable form (a form that will last, not change, and can be shared), they are a form of dark matter.
Identifying where it can be found is easy: the brain. Yet you also need to ask when it can be found.
These ideas are inherently ephemeral.
If I don’t write down a poem immediately when it presents itself to me, I will lose it. Even if I can later recall its rough outline, the intricacies of its verses will be lost forever.
So, for this reason, unproduced information exists not just within a mind, but also within a moment.
How can we more effectively make the information accessible? We need to instantiate the idea into physical reality.
It is common to tell others to “create more”, so my goal will be not to chastise you for not creating, but instead to improve the onramp which leads to creation. The critical step is to focus on increasing our propensity to produce.
In order to increase the likelihood of producing ideas you can either make it easier to produce or systematize the creation process.
Making it easier to produce an idea is simple enough: For writers, it may be leaving pens and pads of paper around the house; for singers, it may be putting a recording studio in the basement, for painters, it may be leaving an easel set up in the living room, or for engineers, they may want a workshop in the garage.
With each of these setups, you reduce the latency between inspiration and perspiration.
As that lag shortens, not only are you more likely to act (there is less friction to do so), but you also more accurately produce the idea that inspired you.
Systematizing the process of producing an idea is a more complex option.
“So: lower your bar for what’s worth writing about! My personal standard is anything that I’ve said more than once in a conversation.”
Me, a non-writer at the time, was intrigued by this.
This was not just an interesting point, but rather a solution to a hard problem I had. I always felt like my ideas had already been written about and I could not add anything of value. After reading this essay I realized that there were many ideas I had worth bringing up, evidenced by the fact that I had brought them up in conversation many more times than once!”
This is a good example of a system that increases your propensity to produce. Importantly, a good system does not over-optimize. For example, a good system would not be to write an essay each time I bring up something in a conversation. Similarly, a painter should not paint everything they see. Our goal should be to create an algorithm that promotes creation but does not inundate you with creative chores. Another system may be setting a fixed routine to work on a task each day (get into the studio at 8 am).
Despite its trite reputation, Atomic Habits by James Clear serves as a strong launch point for anyone hoping to create these types of systems. Afterwards, there is good domain-specific reading that explores this further such as The Creative Act by Rick Rubin.
Adding this up, we want to make it easy to quickly instantiate an idea in physical reality, and we can do that by removing friction between idea and execution, and introducing systems that prompt creation in the first place.
Dark matter of the brain now exists in the real world, so what’s next?
Unpublished information
Once that information is produced, it falls into a new limbo of information dark matter, unpublished information.
Unpublished information is information that has not been shared. This might exist in notebooks, laptops, digital drives, on whiteboards, canvases hidden in workshops, and more.
It can be found anywhere, but it’s often hard to seek out. Importantly, even if you did try to find more unpublished information (by hacking into online files or snooping elsewhere), you couldn’t know ahead of time whether it’s valuable or not.
This question of value is fairly unique to this level of information dark matter. For unproduced information, a lot of the value can simply come from the process of production. A sculptor who creates a new sculpture may not have made their masterpiece, but they are likely to have refined their skills at the very least.
Now, it is not just the act of production that improves the sculptors’ work, but also the act of publication.
Anecdotally, every time I release a work of mine, be it essays, a book, tech products, or whatever else, I discover “unknown unknowns” and learn much more than I thought I would.
The process you follow is valuable regardless.
Yet even if you follow the process all the way to completion, why take the leap to publish? Is what you have valuable enough to publish?
Whether or not your work is valuable or helpful shouldn’t dictate your decision. Many branches of mathematics were seen as aesthetic, useless, and wasteful, and now they drive some of our most exciting branches of innovation (imaginary numbers, prime numbers, and knot theory, led to advancements in quantum theory, cryptography, and genetics respectively).
The question of value is more easily answered when famous artists, mathematicians, writers, or other prolific people pass away. We often see their creations released posthumously because we already “know” that what they have created merits publication (at least if you judge from past works), and even if it requires some editing, refining, mixing, and mastering, it is worth it.
Yet even for those of us who are not famous, it is still in our interest to publish as much of the unpublished as we can: it improves our works and value can be created whether we expect it to or not.
Similarly to the last type of dark matter, where people are encouraged to create more, many people are pushed to share and publish. I’m not going to chastise you for not publishing or sharing, so once again my goal is to increase our propensity to publish.
So, what makes it easier to publish?
Some of the answers to this question are more psychological and rather personal, but there are some common woes that often impede people from publishing. Let’s see if we can find the necessary remedies:
If fear of judgment prevents you from publishing work, post anonymously.
Guillaume Verdon founded a technology optimism movement entirely from an anonymous Twitter account parodying Jeff Bezos. While this is a bad example of anonymity working in the long term (he eventually had his identity leaked by journalists), it is a good example of anonymity enabling someone to post more and share thoughts they may not have otherwise felt comfortable sharing when encumbered by their own identity.
Speaking on the Lex Fridman podcast, Guillaume even discusses how the anonymity contributed to a novel way of thought, allowing him to let his thoughts wander further than what Guillaume himself was constrained to. So not only does anonymity contribute to the likelihood of posting, but it may in fact contribute to entirely new areas of inquiry (better production).
2. If thinking no one will see your work is stopping you, solve for distribution.
Frankly, if you aren’t worried about this already you probably should be. Very few people create, yet the amount of content published around the world is immense. On YouTube alone, 30,000 hours of content are uploaded every hour. In practice, this means that very few people may see your work if you just throw it out into the void. We will expand on this concept later.
For now, solve for distribution by sharing it amongst friends, publishing your work into appropriate communities (which I explore more later), or by sending it directly to key persons of interest.
The challenge with distribution is that any new channel or strategy to reach people eventually becomes saturated, and this effect tends to create epistemic monopolies. For example, Wendy’s sassy marketing, which appeals to current trends and pop culture, was an original idea upon its release, but if others followed suit it would be seen as tacky or lame. Idea space gets “exhausted”. Find new ways to reach others, to get in front of people, and if it’s something someone else already did, move on and think of the next thing. If you find yourself waiting to act because no one has shown you a novel way to do so, you will find yourself never acting. Stop looking for advice; if someone is sharing some new path it means it is likely already overdone.
That being said, personally sending your work to someone is probably a fairly timeless way to get more direct exposure. When relevant, I have attached an essay of mine when discussing these concepts with friends or people who I am meeting. While this isn’t scalable, I know that at a minimum, these people have read it. In fact, when it comes to virality, it is not scalability that matters, but instead accuracy. You need to light the right spark just once, not thousands of the wrong sparks many times.
3. If finding the motivation is hard, find a peer group that pushes you.
Certain things are only difficult in context.
Amongst peers who do not publish anything, publishing something is impressive. Amongst peers who are prolific, publishing nothing is the disappointing exception.
Sometimes just being around people who wear down the snow on the ski hill can make following those tracks easier.
A final thought about publishing:
People often ask me about how successful my essays have been. I reply with a question:
“What do you mean by ‘successful’?”
They normally propose a metric like viewership or engagement. Maybe one day that will be my goal, but I’m currently betting on the long game of essay publishing.
For me, it does something important regardless of viewership: it is a panopticon in my thinking.
The Panopticon is a theoretical prison design where just one warden can watch any prisoner at any time (but not all). However, prisoners can never see the warden. Therefore, there is an information asymmetry, and prisoners have to act as if they are always being watched despite that being truly impossible.
I write far more than I publish.
But every time I write, I know that there is a chance I will publish. I won’t publish all of my thoughts, just as a guard cannot watch all of the inmates, but I write (and therefore think) as if everything will be published. This pushes me to be a better writer; if I take shortcuts, poorly explain things, or fail to truly understand an idea, that will be conveyed clearly to my audience.
You are my warden in this panopticon.
As long as I intend to publish my work, even if it never comes to fruition, my writing will always be of publishable quality. To make this work, I must regularly publish my essays in order to maintain the possibility that any given piece may be published.
So, even if eyes never flit across these very words, I act as if they had. For that, I am a better creator.
So, don’t just produce more, but publish more. The action of publishing improves the act of production.
Restricted information
Now that our information is produced and published, we must question the mode and location of publication. Depending on where the information has been published, we may still be dealing with some form of dark matter.
This is restricted information.
Restricted information is produced and published but only for some in-groups, usually to create some type of specific value.
This happens one of two ways:
Incidentally: When I was going through school, I found a helpful network within my alumni, so now when students of my Alma Mater reach out to me I am more likely to respond and share useful information (I am not intentionally gatekeeping, it’s more a function of social pressure and systems). I am “accidentally” restricting information and I don’t particularly capture any value from this.
Intentionally: I start writing paid essays that are super valuable but only to those who spend the money to read them. I am intentionally restricting information so that I can capture value.
My goal is to find ways for us to increase the propensity we have to reveal this information. But first, let’s explore why information becomes restricted.
One incidental way is tacit knowledge. Through your experiences, you learn things that are then difficult to communicate or teach (sometimes, you might not even know what you know). The closest way to pass tacit knowledge is through an apprenticeship-like system. Trying to produce some explanatory work may be quite difficult, so you might just have someone watch what you do and how you do it. Suggesting that people take on more apprentices may be impossible (both on the supply side of the mentor and on the demand side), but as new tools emerge we can create a similar effect. Tacit Knowledge Videos are a way to share this information and catalog the “best” videos across various disciplines. The curated videos range from software to therapy, from music to construction, and more. These disciplines are also taught elsewhere through traditional courses, but in videos like these, we can find the practical process better captured. This is where tacit knowledge occurs.
This tacit knowledge is information dark matter by accident. Even if a master is intentional about which apprentice they choose, it would be unfair to describe this as intentionally excluding everyone else. There may not be enough room in the workshop or time in the day. Circumstance bottlenecks tacit knowledge transfer, so one way this can be fixed is by welcoming a video camera into your process.
Another incidental way information is restricted is through inside knowledge. Without desiring, someone who produces a work may accidentally restrict the enjoyment of the work to those who understand the inside concepts or who are willing to do the intellectual labor to learn it (which may be entirely unfeasible for most).
Why does this occur? To create something is also to put part of you into it. This makes inside knowledge a very broad category since it’s anything that you have an understanding of and others may not:
Producing something highly technical (mathematical papers, scientific research, etc) keeps those not privy to the domain “outside.”
Producing something in your language (this is written in English) keeps those who don’t read English “outside.” Accessing it through translation does not capture my original thoughts.
Producing something with jokes may render the information inaccessible without the shared context within which those jokes are based.
Producing something with references to pop culture makes the information potentially temporally inaccessible, where an “outside” reader in the future would not understand certain call-outs.
The list goes on...
Beyond incidentally restricted tacit or inside knowledge, a form of intentionally restricted information are secrets.
Businesses charge a fee and provide a service or product (or information). The thinking here is slightly circular:
Create value because people are paying, and people are paying because value was created.
Yes, you could create that same value without paying, or it could be offered for free, but compensatory incentive systems seem to work fairly well.
Secrets are an intentional way of restricting information. You use this to give your business an advantage or to “monopolize” on some opportunity. Having the ability to solely benefit from what you have created is what underpins intellectual property law and property rights as concepts. It is a motivating force. Despite pharmaceutical companies charging prices some may deem unfair, this is just an unfortunate byproduct of a system that does exactly what we want it to (generate novel drugs that help humans live their lives).
Reconciling a belief in property rights with a desire to prevent information dark matter brings me to a more unorthodox idea:
“At Tesla, however, we felt compelled to create patents out of concern that the big car companies would copy our technology and then use their massive manufacturing, sales and marketing power to overwhelm Tesla. We couldn’t have been more wrong. The unfortunate reality is the opposite: electric car programs (or programs for any vehicle that doesn’t burn hydrocarbons) at the major manufacturers are small to non-existent, constituting an average of far less than 1% of their total vehicle sales.
At best, the large automakers are producing electric cars with limited range in limited volume. Some produce no zero emission cars at all.
Given that annual new vehicle production is approaching 100 million per year and the global fleet is approximately 2 billion cars, it is impossible for Tesla to build electric cars fast enough to address the carbon crisis. By the same token, it means the market is enormous. Our true competition is not the small trickle of non-Tesla electric cars being produced, but rather the enormous flood of gasoline cars pouring out of the world’s factories every day.
We believe that Tesla, other companies making electric cars, and the world would all benefit from a common, rapidly-evolving technology platform.
Technology leadership is not defined by patents, which history has repeatedly shown to be small protection indeed against a determined competitor, but rather by the ability of a company to attract and motivate the world’s most talented engineers. We believe that applying the open source philosophy to our patents will strengthen rather than diminish Tesla’s position in this regard.”
(This quote was previously on the Tesla website and during the writing of this essay it was changed to the currently linked patent pledge. I have included this version because it better explains the strategy behind why Tesla chooses to keep their patents open to good faith use.)
This strategy will not and does not work for everyone, but Tesla has chosen to give up its IP secrets in exchange for improving its brand/recruiting prestige. There are likely other strategic benefits, such as Tesla having one of the more advanced charging networks (which benefits from having more cars on the grid using it). Whatever the reasoning, it has undeniably moved the entire field forward.
And while this makes Tesla a strong force against information dark matter, it appears to be more exception than rule.
Nat Friedman, ex-CEO of GitHub (and a lot more), spoke about hedge fund secrets on the Dwarkesh Patel podcast:
“I was trying to understand the role secrets play in the success of a hedge fund.
So I asked some traders at a very successful hedge fund, “If you had your smartest trader get on Twitch for 10 minutes once a month, and on that Twitch stream describe their 30-day-old trading strategies. Not your current ones, but the ones that are a month old. What would that... How would that affect your business after 12 months of doing that?”
So 12 months, 10 minutes a month, 30-day look back. That’s two hours in a year. And to my shock, they told me about an 80% reduction in their profits. It would have a huge impact.
And then I asked – So how long would the lookback window have to be before it would have a relatively small effect on your business? And they said 10 years. So that I think is quite strong evidence that the world's not perfectly efficient because these folks make billions of dollars using secrets that could be relayed in an hour or something like that. And yet others don't have them, or their secrets wouldn't work. So I think there are different levels of efficiency in the world, but on the whole, our default estimate of how efficient the world is is far too charitable.”
This excerpt from the podcast helps clarify a few important ideas:
Sharing secrets could add value (the world is not perfectly efficient)
Not only is there no good incentive to share, but there is also a true disincentive (losing 80% of your profit) to not share
I do not think that secrets should be shared immediately, of course. The creator of the work should benefit from the secrets that provide them with value. There are lessons to be learned from other types of IP however:
In other areas of IP (patents, trademarks, copyrights, etc.) we have limits to the time one can enforce their claim on a piece of information. The goal of these systems is to privilege the creator of value with the benefit of that value until some point in the future. At that point, we agree it is societally beneficial for everyone to access that technology, use those images, and more.
Secrets share this “beneficial” property. They are veiled, and it is hard to attribute from the outside which secrets contribute to what output. Importantly, there is no defined time when secrets must be revealed.
Secrets stay secret.
This could be limiting our potential.
So, the conclusion that follows with this logic is to bring secrets into parity with other forms of IP. Yet, that feels like a reach. It now directly conflicts with certain privacy concerns. Regardless of whether or not your company benefits from a report created decades ago does not change the intrusiveness of knowing that it must be revealed at one point.
The panopticon appears once again.
But, I can also highlight the potentially adverse effects of the panopticon.
Knowing that you will face scrutiny may indeed promote good behavior (I imagine the McKinsey consultants would think twice about their advice to “Turbocharge opioid sales”). In this regard, the panopticon increases accountability, just as it does when I write.
Yet, business is a fundamentally non-perfect practice: moving fast and breaking things, having a bias for action, iterating quickly, and many more aphorisms all seem to understand this reality. And while these philosophies come under fire, they are effective at creating disruptive firms (because these philosophies are inherently pro-disruption).
Revealing internal documents after some set amount of time may do some good because people will be more likely to “price-in” the reputational damage they would experience in two decades; however, it may also limit the risks we are willing to take and limit the ability to argue the contrary or play devil's advocate.
Does this turn knowledge creators into overly stressed-out perfectionists?
Does this lobotomize nuanced thinking?
How does the fear of the Overton window shifting impact our decisions?
Requiring the release of these secrets may just subdue all actions of all companies so that executives don’t risk causing problems that are yet to even be identified.
We will pore over those past actions with the present lens, and actions found to be incongruent with our current beliefs will be judged not as they were, but as they are.
“Every record has been destroyed or falsified, every book has been rewritten, every picture has been repainted, every statue and street and building has been renamed, every date has been altered. And that process is continuing day by day and minute by minute. History has stopped.
Nothing exists except an endless present in which the Party is always right.”
— George Orwell, 1984
Or perhaps, having no way out may ensure people act more erratically because they have nothing to lose.
Oil and gas companies were early to identify the potential downsides of emissions. Unlike secrets, this was information that anyone could stumble upon with the appropriate effort. To combat this, what did these firms do?
According to Greenpeace, Exxon spent more than $30 million on think tanks that promote climate denial.
A half-century of tobacco industry deception has had tragic consequences: Since the “Frank Statement,” approximately 16 million Americans have died from smoking, and millions more have suffered from debilitating diseases ranging from emphysema to heart disease. Had the industry come clean in 1954—matching deeds with promises—many of these deaths would almost certainly have been prevented. No one knows how many. Perhaps 3 million. Maybe 5 million. Maybe 7 million—just in the United States. An honest approach by industry might have saved more lives than any public health measure taken during the past fifty years. Furthermore, if industry had made good faith efforts globally, rather than exploit and addict the developing world, the benefits could have been stunning.
Both oil and gas and tobacco companies hired the same researchers in their attempt to hold onto their power. If you suspect that your time is running out, the actions you take can become amplified. For these reasons, we cannot simply require all firms to unload their dirty laundry after some period of time: It would cause some firms to stagnate, and it would cause others to explode.
So, how do we promote the release of restricted information? I believe a system can exist in the private sector to do just that.
The solution must:
Incentivize individual actors to want to reveal secrets (as demonstrated by Tesla)
Not damage present performance as a result of revealing secrets (as demonstrated by hedge funds)
Reduce risk possibly by anonymizing or aggregating the secrets (as demonstrated by O&G/tobacco firms)
One way to accomplish this is by creating an opt-in digital commons. A startup could buy up old information/secrets that are sanitized and then train AI models around this formerly secret information. Companies could then purchase or use these models that are incredibly domain-specific and benefit from the broadened horizons of thinking as contributed by the other participating firms.
In this way, each firm is incentivized to release restricted information, while not risking damage to present performance, and also not creating individual liability or scrutiny past what exists in current knowledge work.
How the specific business model would work, how profitable this could be, and problems with risk would have to be solved; but, as a thought experiment, it could work. In practice, it could also be completed by a coalition/joint venture style collaboration across similar firms (but if membership is restricted it could be oligopolistic and potentially increase barriers to entry).
The question that must be raised once again is that of value. Knowing what to look for and finding the needle in the haystack is part of the secret. What does the dumping of corporate files accomplish? If this was being manually explored I would likely have a different opinion, but if we can take good thought and analyze the body of work quickly, maybe needles will be more common than originally imagined.
Adding this up, we want to make tacit knowledge easier to share, and firm secrets more valuable to share.
This would un-restrict information and eliminate another level of information dark matter.
Unfindable information
We have produced the ideas, published our creations, and unrestricted our work.
What dark matter is left?
“If the Library of Alexandria was the emblem of our ambition of omniscience, the Web is the emblem of our ambition of omnipresence; the library that contained everything has become the library that contains anything.”
— Alberto Manguel
In the library that contains anything, dark matter fills the shelves. On an infinitely long bookshelf, knowing what to pick up and read is the battle.
I’ve explored the question of value throughout this essay, but here is where it faces its most challenging opponent. The final form of dark matter is unfindable information. Information that exists out in the universe, but it’s so distant, so far, so untethered to anything else, that it is effectively not there.
To solve this dark matter problem, it’s not about making things more findable, since this leads to ”rule beating” or Chesterston’s Fence.
So, my goal here is not to recommend that everyone crank up their signal boosting and rule beating, but instead, my goal is to explore how both sides of the market can improve.
Consider the internet as not just a library, but instead a large brain. The internet doesn’t just store readable information, it is an ecosystem of people posting, sharing, and engaging with each other and the content. In this brain, we don’t want all information to be more findable, we want it to be easier for better information to be more findable. If you signal-boost by rule-beating and SEO-hacking, you “enshittify” the commons, and while it is in the best interest of each party to abuse this system(the tragedy), it also does damage your brand. Signal boosting all things just makes the brain noisier and messier, it doesn’t make it more efficient.
We have produced, published, and unrestricted more information. Now we want those ideas, and all ideas, to be able to effectively compete.
Information (or memes) competes in the same way genes do.
My whole aim of information dark matter until this point has been to bring more ideas and information out from the woods and onto the field.
Now, my goal will be to ensure that the field is competitive, stays competitive, and brings the right good ideas to the right people.
This will mean optimizing the “brain” and the way we transmit information. Publishers of information dark matter can think of themselves as effective neurons, and these neurons operate as both a demander and supplier of information.
An “effective neuron” in the human-internet brain should:
Originate the best ideas it can
Curate the signals it receives
Share at the appropriate proportions
Let’s explore how each of these principles works.
An effective neuron should originate the best ideas it can.
As we have explored so far, ideas should be produced, creations should be published, and information should be unrestricted. Part of the reason we aim to unrestrict is because we do not know what is, could be, or will be valuable. The value of information is subjective. So, your goal should be to both create and share the best ideas you can, and your self-selection process will ensure these ideas are certainly worthwhile to someone because they’re at least worthwhile to you. Nowhere do I advocate for all ideas to be shared; however, I do believe that more people and institutions should have systems to create, publish, and reveal more.
It is not your job to decide the value of what you create. It is your responsibility, however, to ensure that you originate the best you can within the limitations you have. The system works because neurons receive and curate what other neurons produce.
An effective neuron should then curate the signals it receives.
As a receiver, it is your responsibility as an effective neuron to “curate” the signal. That means passing along good things (amplifying) or shutting down bad things (dampening).
Curation gets a bad rep with statements like “It’s easier to be a critic than a creator,” but curation is also important and potentially more impactful:
“Let me ask a pointed question—who contributes more to the human enterprise:
Someone who writes a great book no one ever reads?
Or someone who leads millions of others to an unknown, life-changing book?
It’s not my intention to downplay the pleasure of doing original work (even done privately!), but only to emphasize the importance of the curators, who amplify the great works of others. They may not have their names on the canvases, but if a piece of work has a valuable effect on its viewers, certainly those who curated / promoted /preserved deserve a hefty cut of the glory.
... great works are worthless unless discovered, so curation is just as critical as traditional creation.”
This post perfectly captures the essence of why curation is not something that can be ignored, even if we acknowledge that it may require less skill or effort than creation.
Further, curation happens automatically.
Humans judge in the blink of an eye, so I’d rather emphasize that curation should be done intentionally and thoughtfully as opposed to pretending it plays no part in effective neuron behavior.
As you live your life, you are curating. The products you buy, the places you choose to go, and the media you consume... you are curating your own life. Through social interactions, you convey your curated preferences, whether you want to or not. An effective neuron cannot be detached. An effective neuron therefore curates things for others, whether or not they personally find it valuable to do so.
An effective neuron should share at the appropriate proportions.
Originating and curating the signals at the appropriate proportions can be argued to be more important than the actual act of origination and curation in the first place. You need to be accurate to be valuable.
Part of this is being directionally correct: Good things should be amplified and bad things should be dampened.
But it goes past this, being specifically accurate is also important. Boys who cry wolf stop getting listened to. It is in your best interest to be accurate in your reviews if you want people to trust your recommendations.
An example of this in action comes from my high school conversations. In my senior year, when my friends and I were in the process of applying for university, there was a discussion about the “adjustment factor” applied to applicant grades by the University of Waterloo, an engineering school in Canada.
The adjustment factor was created by their faculty of engineering to discount the grades coming from certain schools that potentially inflated their students’ performance. Due to the Freedom of Information Act you can find those adjustments.
This example demonstrates what happens when schools (senders) don’t evaluate at a fair objective level: universities (receivers) will create their own filter. Similarly, in the shared human brain, pessimists and optimists get their judgments calibrated, becoming comparable.
For someone (or some school) who is purely self-interested, this process keeps you in check. A school wants you to take their students, so they ought to give everyone perfect grades; however, you as a receiver will conduct your own evaluation of their signals.
In a perfect world (continuing with the school example) where all neurons are accurate, this is what would happen:
Receive the grades of the student
Accept/reject the student
In our world:
Receive the grades of the student
Adjust the grades of the student
Accept/reject the student
Track the performance of the student
Analyze how grades translate to real performance
Revise adjustment factors
For the self-interested school, if you keep inflating students' grades who then end up poorly performing in post-secondary, the adjustment factor will keep inflating to make it increasingly difficult for your students to be selected. Eventually, it becomes an impossible hurdle to leap over; even students with otherwise perfect grades would be discounted below the admission thresholds. At this point, the connection between neurons is severed. So, it is in your own “information self-interest” to be accurate and precise.
In summary, if you act as an effective neuron continuously, you become a more central node through which signals are passed: more people listen to you, people listen more attentively to what you have to say, and people adjust what you say less.
Or, in other words:
Want to stay findable?
Be an effective neuron
Add time
Being an effective neuron sets a foundation for, but doesn’t precisely answer, the question I seek to answer: How can we effectively make the information more accessible?
So, if signal boosting and rule-beating are off limits but we still want to surface “unfindable information”, we need to discuss the systems that can accomplish this across all domains. To explore these thoughts, I will first turn to people who have had great success at virality.
“I know a lot of people who make things, who don't stand proudly by their stuff...
they'll put a song out, they'll put it in their [Instagram] story and that's it.
That's it.
You went through something,
you wrote words down.
You figured it out in a structural format.
Found music to go along with it.
It's a whole thing. And you mean to tell me that you're going to be passive with your own work and just put it on your [Instagram] story? Once? Are you crazy, bro?
I'm still promoting my album that came out last year.
I put time, love, and energy into this finished project just to put it on Instagram and forget about it? Like no, promote.
You let people know. Be proud of what you made.”
(Transcript edited for clarity and brevity)
Promoting your own work and sharing it with others is not rule-beating. You should be evangelizing your work and “be proud of what you made.”
Let’s return to this question: “Who contributes more to the human enterprise: Someone who writes a great book no one ever reads? Or someone who leads millions of others to an unknown, life-changing book?” It’s important to realize that the answer is not to be either-or, they don’t have to be mutually exclusive choices. Write the great book, and then give people the opportunity to see it for themselves by appropriately sharing the work. Don’t be “passive with your own work.”
Now, Virgil Abloh, the late founder of Off-White and artistic director at LVMH, on context and design:
"If I put this candle in an all-white gallery, it looks like art, if I put it in a garage it looks like a piece of trash.
I could design the candle and spend a lot of time, like, telling you about the candle, or I can just design the room it sits in."
A lot of incredibly creative people are laser-focused.
They create their art, craft their machines, and write their ideas.
Once the process that they enjoy is done, the rest of the sequence is left unfinished.
The painter is rarely ever the auctioneer for their own works.
However, crafting the context around the creation has the ability to change a simple candle from trash to art. How do you craft the context around your work?
Don’t just focus on the product.
It’s common knowledge that J.K. Rowling’s Harry Potter & The Philosopher’s Stone was rejected by twelve publishers.
This seems like a lot, but the fact is that many other books have faced far more rejections:
Lord of the Flies received 21 rejections
Dune received 23 rejections
Dr. Seuss received 27 rejections
Zen and the Art of Motorcycle Maintenance received 121 rejections
This is not a book publishing problem alone:
Airbnb was turned down by 20 investors before Y Combinator accepted them (after themselves having rejected them earlier)
The Beatles, Lady Gaga, and Ed Sheeran all faced rejections from labels
Claude Monet’s and Vincent Van Gogh’s works were derided by critics on release
Great things, across all domains, are not always given the chance they deserve.
Are you destined to face rejection? Is there a better way to go about sharing your work with the right people? How many times should you try?
An important start to finding these answers is acknowledging that the liberty to attempt endlessly is not afforded in all circumstances. Eventually, there are no more publishers, record labels, investors, etc. left to go to.
To optimally share your work, you must consider evaluation cost.
Each of your attempts is not “free,” but instead costs time, energy, money, or some other resource.
Every time you submit a manuscript to a publishing house, the evaluation cost of your attempt is high: it takes time to read a book and their time is valuable (given many people want them to evaluate their books). What this implies is that it will first be difficult to even get an audience with the publisher, but then it will be more difficult still to get them to read your second, third, or fourth attempts if they have rejected you for having unsatisfactory work.
This evaluation costliness is important to be aware of, especially in domains where a limited number of people have decision power or “taste.”
This environment (high evaluation cost, limited decision makers) has significant downsides:
It’s difficult to get a first attempt noticed
It’s hard to bounce back from a bad attempt
It reinforces and entrenches people who have made good attempts in the past
If your work is novel and different, you may find it difficult to get sufficient buy-in. You are going against the grain.
Thankfully, things are mostly different now, a great deal in part due to the internet. The channels of distribution and who owns them have radically changed. Now, if you want to be a writer, you don’t have to hope a publisher sees the diamond in the rough, you instead have many ways to self-publish. Models don’t have to hope to be “discovered” walking down the street, they can post on social media, amass a following, become an influencer, and get discovered. And then even still, there is no need to sign with a modeling agency if you can manage your brand deals and source work.
In this open market environment, there are uneven evaluation costs and unlimited decision-makers.
So what is the approach to success in this kind of world?
Justin Bieber was discovered on YouTube, and the underlying dynamics are most fascinating.
As a way to share his singing with family, Bieber and his mom began posting clips of him in 2007 performing covers of Stevie Wonder, Michael Jackson, and Ne-Yo on YouTube.
Within months, he was an Internet sensation, with a large following of fans. In 2008, music manager Scooter Braun discovered Bieber’s YouTube videos and arranged for him to fly to Atlanta to meet Usher, who promptly signed the 14-year-old to his record label.
Stories like this are happening more and more, where someone who isn’t chasing the record deal, big publishing house, or acting gig somehow ends up “stumbling” into success.
The reason this works to identify high-quality talent is simple: effective neurons working together efficiently is powerful.
When you post on the right platforms or into the appropriate communities, you start with people who have no, low, or potentially negative evaluation cost.
For Justin Bieber, rather than going straight to studios, he posted on YouTube. The first watchers were not being “compensated” nor were they charitably giving their time; they were watching because they enjoyed watching. Evaluation cost was negative, they were actively looking for this type of content. These people viewed, liked, commented, and otherwise evangelized this to others. Now, people who were not actively seeking this type of content had it recommended enough to be deemed sufficiently worth their evaluation cost. You see this phenomenon in practice when a movie gets recommended by enough people that you go see it for yourself, even if you were originally unenthused. This grassroots, bottom-up audience ends up creating a sort of neuronal pressure upwards, triggering more and more people to engage. As the content becomes increasingly curated, those with higher evaluation costs will begin to take notice, and if this propagates far enough up through the collective brain, you get the right people (agents and tastemakers) engaged.
Justin Bieber did not get lucky randomly, he leveraged the power of good human-neurons to transmit his works.
So, for you, what is the place where you can share and make your work findable? What place doesn’t incur an evaluation cost that hampers the sharing itself?
Platforms, communities, or systems that prioritize or offer discovery are critical. Understanding where engagement with your creation is based in joy, not duty, is the key to making your work initially shareable.
Share your work where and with whom “evaluation” is less costly.
Changes in tools and technology also materially change the way we find information.
This essay is full of examples I was able to access with quick search queries.
I typed “Books that received the most rejections” into Google and this provided me with articles and pages that brought me exactly what I was looking for. This is powerful. In the past you would have to wait for a library to open to do your research, the book you wanted may have been taken out, and multi-book searching was barely doable (asking the librarian); the entire process was overwhelmingly manual.
I also recognize that in the future, the way we search will continue to change. More semantic or “understand the meaning” searching with tools like Perplexity or Exa.ai. Both of these tools have become critical steps in researching my essay topics; often my ideas don’t have a clear Google-able term and so using Exa.ai has exposed me to other people on a similar train of thought, even if the way they describe it is quite different. It searches for similar meanings, not for similar words. Through these tools I have found things that would have otherwise been dark matter. New tools also change the way you may construct or design the information you seek to produce.
Use the best tools you can to find information
We’ve now explored how you can appropriately make information more findable as both a supplier and searcher of this content. Inevitably, some of this information is doomed to be exactly what we have been discussing, dark matter. It will line the shelves of this endless library and amount to nothing. We are better for the information to have been produced, published, revealed, and found, even if that has led to the information being evaluated as not useful.
Ideas should be allowed and incentivized to enter the collective human-internet brain and fight. It is this process that selects the best and most meaningful works.
In The Myth of Sisyphus, Albert Camus discusses a failed attempt at marketing by one author in the footnotes: “This post-war writer who, after having finished his first book, committed suicide to attract attention to his work. Attention was in fact attracted, but the book was judged no good.”
At least the idea was evaluated, even if the cost paid for that was far too high.
Information matter
Throughout this essay, I have explored how we can begin to solve this information dark matter problem.
Starting at the very inception of an idea until its triumphant release into the world.
We have identified the main types of information dark matter (unproduced, unpublished, restricted, unfindable), located them, and discussed how to make the information more accessible.
Now, it is up to us:
Produce more ideas.
Publish more creations.
Unrestrict secrets.
Make hidden works more findable.
Following these steps improves our abilities, strengthens our knowledge bases, makes us more effective neurons, and benefits society by revealing more valuable information dark matter.
That’s what this essay was all about for me; it’s my treatise on producing, publishing, revealing, and broadcasting with intentionality.
I am surrounded by dark matter. One post at a time, I hope to fill our shared brain with more information matter. Now be an effective neuron and share this.
I started collecting a list of once “secret” documents that were, in one way or another, broadly released (whether it be through legal process, government disclosures, leaks, or the ideas themselves eventually turned into books):
With all of these examples, we can see that significant human capital is funneled into fields like tech, finance, and consulting (or law, startups, VC, government/statecraft, etc), and it is therefore unsurprising that much of the work produced there is made to be valuable. In fact, it is very valuable, but it is largely inaccessible to external audiences.
This also happens more broadly; in any human endeavor involving any intellectual effort at all.
The takeaway is that there is an incomprehensible amount of amazing work being done that is nearly, or even completely, impossible for others to find.
This is information dark matter.
It’s out there, and now we know it’s out there, but it is not something we can go and meaningfully interact with.
And while information created for internal company use is just one example, I began to consider where else we might find information dark matter and how we can make it accessible.
Why is this important?
It’s simple: with more information accessible to us, the more we can expand our intellectual arsenals for tackling bigger problems.
To solve this problem of information dark matter, the questions I hope to answer are:
To start answering the first question, I wanted to build a taxonomy starting with first principles. That meant starting my exploration from where all information and ideas emerge, the brain.
Unproduced information
The first type of information dark matter is the kind stuck in someone’s head.
A beautiful song, a lovely poem, a novel product, an innovative process, and so many other things all amount to precisely...
Nothing. When they are unproduced.
Ideas in your head might be incredibly valuable, but due to the very fact that they are not stored in a stable form (a form that will last, not change, and can be shared), they are a form of dark matter.
Identifying where it can be found is easy: the brain. Yet you also need to ask when it can be found.
These ideas are inherently ephemeral.
If I don’t write down a poem immediately when it presents itself to me, I will lose it. Even if I can later recall its rough outline, the intricacies of its verses will be lost forever.
So, for this reason, unproduced information exists not just within a mind, but also within a moment.
How can we more effectively make the information accessible? We need to instantiate the idea into physical reality.
It is common to tell others to “create more”, so my goal will be not to chastise you for not creating, but instead to improve the onramp which leads to creation. The critical step is to focus on increasing our propensity to produce.
In order to increase the likelihood of producing ideas you can either make it easier to produce or systematize the creation process.
Making it easier to produce an idea is simple enough: For writers, it may be leaving pens and pads of paper around the house; for singers, it may be putting a recording studio in the basement, for painters, it may be leaving an easel set up in the living room, or for engineers, they may want a workshop in the garage.
With each of these setups, you reduce the latency between inspiration and perspiration.
As that lag shortens, not only are you more likely to act (there is less friction to do so), but you also more accurately produce the idea that inspired you.
Systematizing the process of producing an idea is a more complex option.
I’ve written about this in the past:
“I’ve been writing my thoughts down as essays for a whole year now.
In my early introduction to essays before I started writing, I came across an interesting essay about publishing essays:
“So: lower your bar for what’s worth writing about! My personal standard is anything that I’ve said more than once in a conversation.”
Me, a non-writer at the time, was intrigued by this.
This was not just an interesting point, but rather a solution to a hard problem I had. I always felt like my ideas had already been written about and I could not add anything of value. After reading this essay I realized that there were many ideas I had worth bringing up, evidenced by the fact that I had brought them up in conversation many more times than once!”
This is a good example of a system that increases your propensity to produce. Importantly, a good system does not over-optimize. For example, a good system would not be to write an essay each time I bring up something in a conversation. Similarly, a painter should not paint everything they see. Our goal should be to create an algorithm that promotes creation but does not inundate you with creative chores. Another system may be setting a fixed routine to work on a task each day (get into the studio at 8 am).
Despite its trite reputation, Atomic Habits by James Clear serves as a strong launch point for anyone hoping to create these types of systems. Afterwards, there is good domain-specific reading that explores this further such as The Creative Act by Rick Rubin.
Adding this up, we want to make it easy to quickly instantiate an idea in physical reality, and we can do that by removing friction between idea and execution, and introducing systems that prompt creation in the first place.
Dark matter of the brain now exists in the real world, so what’s next?
Unpublished information
Once that information is produced, it falls into a new limbo of information dark matter, unpublished information.
Unpublished information is information that has not been shared. This might exist in notebooks, laptops, digital drives, on whiteboards, canvases hidden in workshops, and more.
It can be found anywhere, but it’s often hard to seek out. Importantly, even if you did try to find more unpublished information (by hacking into online files or snooping elsewhere), you couldn’t know ahead of time whether it’s valuable or not.
This question of value is fairly unique to this level of information dark matter. For unproduced information, a lot of the value can simply come from the process of production. A sculptor who creates a new sculpture may not have made their masterpiece, but they are likely to have refined their skills at the very least.
Now, it is not just the act of production that improves the sculptors’ work, but also the act of publication.
Anecdotally, every time I release a work of mine, be it essays, a book, tech products, or whatever else, I discover “unknown unknowns” and learn much more than I thought I would.
The process you follow is valuable regardless.
Yet even if you follow the process all the way to completion, why take the leap to publish? Is what you have valuable enough to publish?
Whether or not your work is valuable or helpful shouldn’t dictate your decision. Many branches of mathematics were seen as aesthetic, useless, and wasteful, and now they drive some of our most exciting branches of innovation (imaginary numbers, prime numbers, and knot theory, led to advancements in quantum theory, cryptography, and genetics respectively).
The question of value is more easily answered when famous artists, mathematicians, writers, or other prolific people pass away. We often see their creations released posthumously because we already “know” that what they have created merits publication (at least if you judge from past works), and even if it requires some editing, refining, mixing, and mastering, it is worth it.
Yet even for those of us who are not famous, it is still in our interest to publish as much of the unpublished as we can: it improves our works and value can be created whether we expect it to or not.
Similarly to the last type of dark matter, where people are encouraged to create more, many people are pushed to share and publish. I’m not going to chastise you for not publishing or sharing, so once again my goal is to increase our propensity to publish.
So, what makes it easier to publish?
Some of the answers to this question are more psychological and rather personal, but there are some common woes that often impede people from publishing. Let’s see if we can find the necessary remedies:
Guillaume Verdon founded a technology optimism movement entirely from an anonymous Twitter account parodying Jeff Bezos. While this is a bad example of anonymity working in the long term (he eventually had his identity leaked by journalists), it is a good example of anonymity enabling someone to post more and share thoughts they may not have otherwise felt comfortable sharing when encumbered by their own identity.
Speaking on the Lex Fridman podcast, Guillaume even discusses how the anonymity contributed to a novel way of thought, allowing him to let his thoughts wander further than what Guillaume himself was constrained to. So not only does anonymity contribute to the likelihood of posting, but it may in fact contribute to entirely new areas of inquiry (better production).
2. If thinking no one will see your work is stopping you, solve for distribution.
Frankly, if you aren’t worried about this already you probably should be. Very few people create, yet the amount of content published around the world is immense. On YouTube alone, 30,000 hours of content are uploaded every hour. In practice, this means that very few people may see your work if you just throw it out into the void. We will expand on this concept later.
For now, solve for distribution by sharing it amongst friends, publishing your work into appropriate communities (which I explore more later), or by sending it directly to key persons of interest.
The challenge with distribution is that any new channel or strategy to reach people eventually becomes saturated, and this effect tends to create epistemic monopolies. For example, Wendy’s sassy marketing, which appeals to current trends and pop culture, was an original idea upon its release, but if others followed suit it would be seen as tacky or lame. Idea space gets “exhausted”. Find new ways to reach others, to get in front of people, and if it’s something someone else already did, move on and think of the next thing. If you find yourself waiting to act because no one has shown you a novel way to do so, you will find yourself never acting. Stop looking for advice; if someone is sharing some new path it means it is likely already overdone.
That being said, personally sending your work to someone is probably a fairly timeless way to get more direct exposure. When relevant, I have attached an essay of mine when discussing these concepts with friends or people who I am meeting. While this isn’t scalable, I know that at a minimum, these people have read it. In fact, when it comes to virality, it is not scalability that matters, but instead accuracy. You need to light the right spark just once, not thousands of the wrong sparks many times.
3. If finding the motivation is hard, find a peer group that pushes you.
Certain things are only difficult in context.
Amongst peers who do not publish anything, publishing something is impressive. Amongst peers who are prolific, publishing nothing is the disappointing exception.
Sometimes just being around people who wear down the snow on the ski hill can make following those tracks easier.
A final thought about publishing:
People often ask me about how successful my essays have been. I reply with a question:
“What do you mean by ‘successful’?”
They normally propose a metric like viewership or engagement. Maybe one day that will be my goal, but I’m currently betting on the long game of essay publishing.
For me, it does something important regardless of viewership: it is a panopticon in my thinking.
The Panopticon is a theoretical prison design where just one warden can watch any prisoner at any time (but not all). However, prisoners can never see the warden. Therefore, there is an information asymmetry, and prisoners have to act as if they are always being watched despite that being truly impossible.
I write far more than I publish.
But every time I write, I know that there is a chance I will publish. I won’t publish all of my thoughts, just as a guard cannot watch all of the inmates, but I write (and therefore think) as if everything will be published. This pushes me to be a better writer; if I take shortcuts, poorly explain things, or fail to truly understand an idea, that will be conveyed clearly to my audience.
You are my warden in this panopticon.
As long as I intend to publish my work, even if it never comes to fruition, my writing will always be of publishable quality. To make this work, I must regularly publish my essays in order to maintain the possibility that any given piece may be published.
So, even if eyes never flit across these very words, I act as if they had. For that, I am a better creator.
So, don’t just produce more, but publish more. The action of publishing improves the act of production.
Restricted information
Now that our information is produced and published, we must question the mode and location of publication. Depending on where the information has been published, we may still be dealing with some form of dark matter.
This is restricted information.
Restricted information is produced and published but only for some in-groups, usually to create some type of specific value.
This happens one of two ways:
My goal is to find ways for us to increase the propensity we have to reveal this information. But first, let’s explore why information becomes restricted.
One incidental way is tacit knowledge. Through your experiences, you learn things that are then difficult to communicate or teach (sometimes, you might not even know what you know). The closest way to pass tacit knowledge is through an apprenticeship-like system. Trying to produce some explanatory work may be quite difficult, so you might just have someone watch what you do and how you do it. Suggesting that people take on more apprentices may be impossible (both on the supply side of the mentor and on the demand side), but as new tools emerge we can create a similar effect. Tacit Knowledge Videos are a way to share this information and catalog the “best” videos across various disciplines. The curated videos range from software to therapy, from music to construction, and more. These disciplines are also taught elsewhere through traditional courses, but in videos like these, we can find the practical process better captured. This is where tacit knowledge occurs.
This tacit knowledge is information dark matter by accident. Even if a master is intentional about which apprentice they choose, it would be unfair to describe this as intentionally excluding everyone else. There may not be enough room in the workshop or time in the day. Circumstance bottlenecks tacit knowledge transfer, so one way this can be fixed is by welcoming a video camera into your process.
Another incidental way information is restricted is through inside knowledge. Without desiring, someone who produces a work may accidentally restrict the enjoyment of the work to those who understand the inside concepts or who are willing to do the intellectual labor to learn it (which may be entirely unfeasible for most).
Why does this occur? To create something is also to put part of you into it. This makes inside knowledge a very broad category since it’s anything that you have an understanding of and others may not:
Beyond incidentally restricted tacit or inside knowledge, a form of intentionally restricted information are secrets.
Businesses charge a fee and provide a service or product (or information). The thinking here is slightly circular:
Create value because people are paying, and people are paying because value was created.
Yes, you could create that same value without paying, or it could be offered for free, but compensatory incentive systems seem to work fairly well.
Secrets are an intentional way of restricting information. You use this to give your business an advantage or to “monopolize” on some opportunity. Having the ability to solely benefit from what you have created is what underpins intellectual property law and property rights as concepts. It is a motivating force. Despite pharmaceutical companies charging prices some may deem unfair, this is just an unfortunate byproduct of a system that does exactly what we want it to (generate novel drugs that help humans live their lives).
Reconciling a belief in property rights with a desire to prevent information dark matter brings me to a more unorthodox idea:
Make sharing secrets valuable for your company.
Tesla has contributed massively to the EV space, and yet effectively refuses to enforce their patents:
“At Tesla, however, we felt compelled to create patents out of concern that the big car companies would copy our technology and then use their massive manufacturing, sales and marketing power to overwhelm Tesla. We couldn’t have been more wrong. The unfortunate reality is the opposite: electric car programs (or programs for any vehicle that doesn’t burn hydrocarbons) at the major manufacturers are small to non-existent, constituting an average of far less than 1% of their total vehicle sales.
At best, the large automakers are producing electric cars with limited range in limited volume. Some produce no zero emission cars at all.
Given that annual new vehicle production is approaching 100 million per year and the global fleet is approximately 2 billion cars, it is impossible for Tesla to build electric cars fast enough to address the carbon crisis. By the same token, it means the market is enormous. Our true competition is not the small trickle of non-Tesla electric cars being produced, but rather the enormous flood of gasoline cars pouring out of the world’s factories every day.
We believe that Tesla, other companies making electric cars, and the world would all benefit from a common, rapidly-evolving technology platform.
Technology leadership is not defined by patents, which history has repeatedly shown to be small protection indeed against a determined competitor, but rather by the ability of a company to attract and motivate the world’s most talented engineers. We believe that applying the open source philosophy to our patents will strengthen rather than diminish Tesla’s position in this regard.”
(This quote was previously on the Tesla website and during the writing of this essay it was changed to the currently linked patent pledge. I have included this version because it better explains the strategy behind why Tesla chooses to keep their patents open to good faith use.)
This strategy will not and does not work for everyone, but Tesla has chosen to give up its IP secrets in exchange for improving its brand/recruiting prestige. There are likely other strategic benefits, such as Tesla having one of the more advanced charging networks (which benefits from having more cars on the grid using it). Whatever the reasoning, it has undeniably moved the entire field forward.
And while this makes Tesla a strong force against information dark matter, it appears to be more exception than rule.
Nat Friedman, ex-CEO of GitHub (and a lot more), spoke about hedge fund secrets on the Dwarkesh Patel podcast:
“I was trying to understand the role secrets play in the success of a hedge fund.
So I asked some traders at a very successful hedge fund, “If you had your smartest trader get on Twitch for 10 minutes once a month, and on that Twitch stream describe their 30-day-old trading strategies. Not your current ones, but the ones that are a month old. What would that... How would that affect your business after 12 months of doing that?”
So 12 months, 10 minutes a month, 30-day look back. That’s two hours in a year. And to my shock, they told me about an 80% reduction in their profits. It would have a huge impact.
And then I asked – So how long would the lookback window have to be before it would have a relatively small effect on your business? And they said 10 years. So that I think is quite strong evidence that the world's not perfectly efficient because these folks make billions of dollars using secrets that could be relayed in an hour or something like that. And yet others don't have them, or their secrets wouldn't work. So I think there are different levels of efficiency in the world, but on the whole, our default estimate of how efficient the world is is far too charitable.”
This excerpt from the podcast helps clarify a few important ideas:
I do not think that secrets should be shared immediately, of course. The creator of the work should benefit from the secrets that provide them with value. There are lessons to be learned from other types of IP however:
In other areas of IP (patents, trademarks, copyrights, etc.) we have limits to the time one can enforce their claim on a piece of information. The goal of these systems is to privilege the creator of value with the benefit of that value until some point in the future. At that point, we agree it is societally beneficial for everyone to access that technology, use those images, and more.
Secrets share this “beneficial” property. They are veiled, and it is hard to attribute from the outside which secrets contribute to what output. Importantly, there is no defined time when secrets must be revealed.
Secrets stay secret.
This could be limiting our potential.
So, the conclusion that follows with this logic is to bring secrets into parity with other forms of IP. Yet, that feels like a reach. It now directly conflicts with certain privacy concerns. Regardless of whether or not your company benefits from a report created decades ago does not change the intrusiveness of knowing that it must be revealed at one point.
The panopticon appears once again.
But, I can also highlight the potentially adverse effects of the panopticon.
Knowing that you will face scrutiny may indeed promote good behavior (I imagine the McKinsey consultants would think twice about their advice to “Turbocharge opioid sales”). In this regard, the panopticon increases accountability, just as it does when I write.
Yet, business is a fundamentally non-perfect practice: moving fast and breaking things, having a bias for action, iterating quickly, and many more aphorisms all seem to understand this reality. And while these philosophies come under fire, they are effective at creating disruptive firms (because these philosophies are inherently pro-disruption).
Revealing internal documents after some set amount of time may do some good because people will be more likely to “price-in” the reputational damage they would experience in two decades; however, it may also limit the risks we are willing to take and limit the ability to argue the contrary or play devil's advocate.
Does this turn knowledge creators into overly stressed-out perfectionists?
Does this lobotomize nuanced thinking?
How does the fear of the Overton window shifting impact our decisions?
Requiring the release of these secrets may just subdue all actions of all companies so that executives don’t risk causing problems that are yet to even be identified.
We will pore over those past actions with the present lens, and actions found to be incongruent with our current beliefs will be judged not as they were, but as they are.
“Every record has been destroyed or falsified, every book has been rewritten, every picture has been repainted, every statue and street and building has been renamed, every date has been altered. And that process is continuing day by day and minute by minute. History has stopped.
Nothing exists except an endless present in which the Party is always right.”
— George Orwell, 1984
Or perhaps, having no way out may ensure people act more erratically because they have nothing to lose.
Oil and gas companies were early to identify the potential downsides of emissions. Unlike secrets, this was information that anyone could stumble upon with the appropriate effort. To combat this, what did these firms do?
According to Greenpeace, Exxon spent more than $30 million on think tanks that promote climate denial.
Tobacco companies similarly knew of the danger their product caused. They funded denial and misinformation campaigns:
A half-century of tobacco industry deception has had tragic consequences: Since the “Frank Statement,” approximately 16 million Americans have died from smoking, and millions more have suffered from debilitating diseases ranging from emphysema to heart disease. Had the industry come clean in 1954—matching deeds with promises—many of these deaths would almost certainly have been prevented. No one knows how many. Perhaps 3 million. Maybe 5 million. Maybe 7 million—just in the United States. An honest approach by industry might have saved more lives than any public health measure taken during the past fifty years. Furthermore, if industry had made good faith efforts globally, rather than exploit and addict the developing world, the benefits could have been stunning.
Both oil and gas and tobacco companies hired the same researchers in their attempt to hold onto their power. If you suspect that your time is running out, the actions you take can become amplified. For these reasons, we cannot simply require all firms to unload their dirty laundry after some period of time: It would cause some firms to stagnate, and it would cause others to explode.
So, how do we promote the release of restricted information? I believe a system can exist in the private sector to do just that.
The solution must:
One way to accomplish this is by creating an opt-in digital commons. A startup could buy up old information/secrets that are sanitized and then train AI models around this formerly secret information. Companies could then purchase or use these models that are incredibly domain-specific and benefit from the broadened horizons of thinking as contributed by the other participating firms.
In this way, each firm is incentivized to release restricted information, while not risking damage to present performance, and also not creating individual liability or scrutiny past what exists in current knowledge work.
How the specific business model would work, how profitable this could be, and problems with risk would have to be solved; but, as a thought experiment, it could work. In practice, it could also be completed by a coalition/joint venture style collaboration across similar firms (but if membership is restricted it could be oligopolistic and potentially increase barriers to entry).
The question that must be raised once again is that of value. Knowing what to look for and finding the needle in the haystack is part of the secret. What does the dumping of corporate files accomplish? If this was being manually explored I would likely have a different opinion, but if we can take good thought and analyze the body of work quickly, maybe needles will be more common than originally imagined.
Adding this up, we want to make tacit knowledge easier to share, and firm secrets more valuable to share.
This would un-restrict information and eliminate another level of information dark matter.
Unfindable information
We have produced the ideas, published our creations, and unrestricted our work.
What dark matter is left?
“If the Library of Alexandria was the emblem of our ambition of omniscience, the Web is the emblem of our ambition of omnipresence; the library that contained everything has become the library that contains anything.”
— Alberto Manguel
In the library that contains anything, dark matter fills the shelves. On an infinitely long bookshelf, knowing what to pick up and read is the battle.
I’ve explored the question of value throughout this essay, but here is where it faces its most challenging opponent. The final form of dark matter is unfindable information. Information that exists out in the universe, but it’s so distant, so far, so untethered to anything else, that it is effectively not there.
To solve this dark matter problem, it’s not about making things more findable, since this leads to ”rule beating” or Chesterston’s Fence.
Consider Google search engine optimization. While sensible (a set of ways to improve your discoverability), it leads to people gaming the system and creating worse websites and user experiences. Rather than creating better content, they solely aim to “beat the rules.”
So, my goal here is not to recommend that everyone crank up their signal boosting and rule beating, but instead, my goal is to explore how both sides of the market can improve.
Consider the internet as not just a library, but instead a large brain. The internet doesn’t just store readable information, it is an ecosystem of people posting, sharing, and engaging with each other and the content. In this brain, we don’t want all information to be more findable, we want it to be easier for better information to be more findable. If you signal-boost by rule-beating and SEO-hacking, you “enshittify” the commons, and while it is in the best interest of each party to abuse this system(the tragedy), it also does damage your brand. Signal boosting all things just makes the brain noisier and messier, it doesn’t make it more efficient.
We have produced, published, and unrestricted more information. Now we want those ideas, and all ideas, to be able to effectively compete.
Information (or memes) competes in the same way genes do.
My whole aim of information dark matter until this point has been to bring more ideas and information out from the woods and onto the field.
Now, my goal will be to ensure that the field is competitive, stays competitive, and brings the right good ideas to the right people.
This will mean optimizing the “brain” and the way we transmit information. Publishers of information dark matter can think of themselves as effective neurons, and these neurons operate as both a demander and supplier of information.
An “effective neuron” in the human-internet brain should:
Let’s explore how each of these principles works.
An effective neuron should originate the best ideas it can.
As we have explored so far, ideas should be produced, creations should be published, and information should be unrestricted. Part of the reason we aim to unrestrict is because we do not know what is, could be, or will be valuable. The value of information is subjective. So, your goal should be to both create and share the best ideas you can, and your self-selection process will ensure these ideas are certainly worthwhile to someone because they’re at least worthwhile to you. Nowhere do I advocate for all ideas to be shared; however, I do believe that more people and institutions should have systems to create, publish, and reveal more.
It is not your job to decide the value of what you create. It is your responsibility, however, to ensure that you originate the best you can within the limitations you have. The system works because neurons receive and curate what other neurons produce.
An effective neuron should then curate the signals it receives.
As a receiver, it is your responsibility as an effective neuron to “curate” the signal. That means passing along good things (amplifying) or shutting down bad things (dampening).
Curation gets a bad rep with statements like “It’s easier to be a critic than a creator,” but curation is also important and potentially more impactful:
“Let me ask a pointed question—who contributes more to the human enterprise:
It’s not my intention to downplay the pleasure of doing original work (even done privately!), but only to emphasize the importance of the curators, who amplify the great works of others. They may not have their names on the canvases, but if a piece of work has a valuable effect on its viewers, certainly those who curated / promoted /preserved deserve a hefty cut of the glory.
... great works are worthless unless discovered, so curation is just as critical as traditional creation.”
This post perfectly captures the essence of why curation is not something that can be ignored, even if we acknowledge that it may require less skill or effort than creation.
Further, curation happens automatically.
Humans judge in the blink of an eye, so I’d rather emphasize that curation should be done intentionally and thoughtfully as opposed to pretending it plays no part in effective neuron behavior.
As you live your life, you are curating. The products you buy, the places you choose to go, and the media you consume... you are curating your own life. Through social interactions, you convey your curated preferences, whether you want to or not. An effective neuron cannot be detached. An effective neuron therefore curates things for others, whether or not they personally find it valuable to do so.
An effective neuron should share at the appropriate proportions.
Originating and curating the signals at the appropriate proportions can be argued to be more important than the actual act of origination and curation in the first place. You need to be accurate to be valuable.
Part of this is being directionally correct: Good things should be amplified and bad things should be dampened.
But it goes past this, being specifically accurate is also important. Boys who cry wolf stop getting listened to. It is in your best interest to be accurate in your reviews if you want people to trust your recommendations.
An example of this in action comes from my high school conversations. In my senior year, when my friends and I were in the process of applying for university, there was a discussion about the “adjustment factor” applied to applicant grades by the University of Waterloo, an engineering school in Canada.
The adjustment factor was created by their faculty of engineering to discount the grades coming from certain schools that potentially inflated their students’ performance. Due to the Freedom of Information Act you can find those adjustments.
This example demonstrates what happens when schools (senders) don’t evaluate at a fair objective level: universities (receivers) will create their own filter. Similarly, in the shared human brain, pessimists and optimists get their judgments calibrated, becoming comparable.
For someone (or some school) who is purely self-interested, this process keeps you in check. A school wants you to take their students, so they ought to give everyone perfect grades; however, you as a receiver will conduct your own evaluation of their signals.
In a perfect world (continuing with the school example) where all neurons are accurate, this is what would happen:
In our world:
For the self-interested school, if you keep inflating students' grades who then end up poorly performing in post-secondary, the adjustment factor will keep inflating to make it increasingly difficult for your students to be selected. Eventually, it becomes an impossible hurdle to leap over; even students with otherwise perfect grades would be discounted below the admission thresholds. At this point, the connection between neurons is severed. So, it is in your own “information self-interest” to be accurate and precise.
In summary, if you act as an effective neuron continuously, you become a more central node through which signals are passed: more people listen to you, people listen more attentively to what you have to say, and people adjust what you say less.
Or, in other words:
Want to stay findable?
Being an effective neuron sets a foundation for, but doesn’t precisely answer, the question I seek to answer: How can we effectively make the information more accessible?
So, if signal boosting and rule-beating are off limits but we still want to surface “unfindable information”, we need to discuss the systems that can accomplish this across all domains. To explore these thoughts, I will first turn to people who have had great success at virality.
Two-time Grammy winner Tyler, The Creator discusses promoting your own work:
“I know a lot of people who make things, who don't stand proudly by their stuff...
they'll put a song out, they'll put it in their [Instagram] story and that's it.
That's it.
You went through something,
you wrote words down.
You figured it out in a structural format.
Found music to go along with it.
It's a whole thing. And you mean to tell me that you're going to be passive with your own work and just put it on your [Instagram] story? Once? Are you crazy, bro?
I'm still promoting my album that came out last year.
I put time, love, and energy into this finished project just to put it on Instagram and forget about it? Like no, promote.
You let people know. Be proud of what you made.”
(Transcript edited for clarity and brevity)
Promoting your own work and sharing it with others is not rule-beating. You should be evangelizing your work and “be proud of what you made.”
Let’s return to this question: “Who contributes more to the human enterprise: Someone who writes a great book no one ever reads? Or someone who leads millions of others to an unknown, life-changing book?” It’s important to realize that the answer is not to be either-or, they don’t have to be mutually exclusive choices. Write the great book, and then give people the opportunity to see it for themselves by appropriately sharing the work. Don’t be “passive with your own work.”
Now, Virgil Abloh, the late founder of Off-White and artistic director at LVMH, on context and design:
"If I put this candle in an all-white gallery, it looks like art, if I put it in a garage it looks like a piece of trash.
I could design the candle and spend a lot of time, like, telling you about the candle, or I can just design the room it sits in."
A lot of incredibly creative people are laser-focused.
They create their art, craft their machines, and write their ideas.
Once the process that they enjoy is done, the rest of the sequence is left unfinished.
The painter is rarely ever the auctioneer for their own works.
However, crafting the context around the creation has the ability to change a simple candle from trash to art. How do you craft the context around your work?
Don’t just focus on the product.
It’s common knowledge that J.K. Rowling’s Harry Potter & The Philosopher’s Stone was rejected by twelve publishers.
This seems like a lot, but the fact is that many other books have faced far more rejections:
This is not a book publishing problem alone:
Great things, across all domains, are not always given the chance they deserve.
Sometimes it takes 121 rejections before you release one of the best-selling philosophy books of all time.
Are you destined to face rejection? Is there a better way to go about sharing your work with the right people? How many times should you try?
An important start to finding these answers is acknowledging that the liberty to attempt endlessly is not afforded in all circumstances. Eventually, there are no more publishers, record labels, investors, etc. left to go to.
To optimally share your work, you must consider evaluation cost.
Each of your attempts is not “free,” but instead costs time, energy, money, or some other resource.
Every time you submit a manuscript to a publishing house, the evaluation cost of your attempt is high: it takes time to read a book and their time is valuable (given many people want them to evaluate their books). What this implies is that it will first be difficult to even get an audience with the publisher, but then it will be more difficult still to get them to read your second, third, or fourth attempts if they have rejected you for having unsatisfactory work.
This evaluation costliness is important to be aware of, especially in domains where a limited number of people have decision power or “taste.”
This environment (high evaluation cost, limited decision makers) has significant downsides:
If your work is novel and different, you may find it difficult to get sufficient buy-in. You are going against the grain.
Thankfully, things are mostly different now, a great deal in part due to the internet. The channels of distribution and who owns them have radically changed. Now, if you want to be a writer, you don’t have to hope a publisher sees the diamond in the rough, you instead have many ways to self-publish. Models don’t have to hope to be “discovered” walking down the street, they can post on social media, amass a following, become an influencer, and get discovered. And then even still, there is no need to sign with a modeling agency if you can manage your brand deals and source work.
In this open market environment, there are uneven evaluation costs and unlimited decision-makers.
So what is the approach to success in this kind of world?
Justin Bieber was discovered on YouTube, and the underlying dynamics are most fascinating.
The story:
As a way to share his singing with family, Bieber and his mom began posting clips of him in 2007 performing covers of Stevie Wonder, Michael Jackson, and Ne-Yo on YouTube.
Within months, he was an Internet sensation, with a large following of fans. In 2008, music manager Scooter Braun discovered Bieber’s YouTube videos and arranged for him to fly to Atlanta to meet Usher, who promptly signed the 14-year-old to his record label.
Stories like this are happening more and more, where someone who isn’t chasing the record deal, big publishing house, or acting gig somehow ends up “stumbling” into success.
The reason this works to identify high-quality talent is simple: effective neurons working together efficiently is powerful.
When you post on the right platforms or into the appropriate communities, you start with people who have no, low, or potentially negative evaluation cost.
For Justin Bieber, rather than going straight to studios, he posted on YouTube. The first watchers were not being “compensated” nor were they charitably giving their time; they were watching because they enjoyed watching. Evaluation cost was negative, they were actively looking for this type of content. These people viewed, liked, commented, and otherwise evangelized this to others. Now, people who were not actively seeking this type of content had it recommended enough to be deemed sufficiently worth their evaluation cost. You see this phenomenon in practice when a movie gets recommended by enough people that you go see it for yourself, even if you were originally unenthused. This grassroots, bottom-up audience ends up creating a sort of neuronal pressure upwards, triggering more and more people to engage. As the content becomes increasingly curated, those with higher evaluation costs will begin to take notice, and if this propagates far enough up through the collective brain, you get the right people (agents and tastemakers) engaged.
Justin Bieber did not get lucky randomly, he leveraged the power of good human-neurons to transmit his works.
So, for you, what is the place where you can share and make your work findable? What place doesn’t incur an evaluation cost that hampers the sharing itself?
Platforms, communities, or systems that prioritize or offer discovery are critical. Understanding where engagement with your creation is based in joy, not duty, is the key to making your work initially shareable.
Share your work where and with whom “evaluation” is less costly.
Changes in tools and technology also materially change the way we find information.
This essay is full of examples I was able to access with quick search queries.
I typed “Books that received the most rejections” into Google and this provided me with articles and pages that brought me exactly what I was looking for. This is powerful. In the past you would have to wait for a library to open to do your research, the book you wanted may have been taken out, and multi-book searching was barely doable (asking the librarian); the entire process was overwhelmingly manual.
I also recognize that in the future, the way we search will continue to change. More semantic or “understand the meaning” searching with tools like Perplexity or Exa.ai. Both of these tools have become critical steps in researching my essay topics; often my ideas don’t have a clear Google-able term and so using Exa.ai has exposed me to other people on a similar train of thought, even if the way they describe it is quite different. It searches for similar meanings, not for similar words. Through these tools I have found things that would have otherwise been dark matter. New tools also change the way you may construct or design the information you seek to produce.
Use the best tools you can to find information
We’ve now explored how you can appropriately make information more findable as both a supplier and searcher of this content. Inevitably, some of this information is doomed to be exactly what we have been discussing, dark matter. It will line the shelves of this endless library and amount to nothing. We are better for the information to have been produced, published, revealed, and found, even if that has led to the information being evaluated as not useful.
Ideas should be allowed and incentivized to enter the collective human-internet brain and fight. It is this process that selects the best and most meaningful works.
In The Myth of Sisyphus, Albert Camus discusses a failed attempt at marketing by one author in the footnotes: “This post-war writer who, after having finished his first book, committed suicide to attract attention to his work. Attention was in fact attracted, but the book was judged no good.”
At least the idea was evaluated, even if the cost paid for that was far too high.
Information matter
Throughout this essay, I have explored how we can begin to solve this information dark matter problem.
Starting at the very inception of an idea until its triumphant release into the world.
We have identified the main types of information dark matter (unproduced, unpublished, restricted, unfindable), located them, and discussed how to make the information more accessible.
Now, it is up to us:
Produce more ideas.
Publish more creations.
Unrestrict secrets.
Make hidden works more findable.
Following these steps improves our abilities, strengthens our knowledge bases, makes us more effective neurons, and benefits society by revealing more valuable information dark matter.
That’s what this essay was all about for me; it’s my treatise on producing, publishing, revealing, and broadcasting with intentionality.
I am surrounded by dark matter. One post at a time, I hope to fill our shared brain with more information matter. Now be an effective neuron and share this.