In risk modeling, there is a well-known distinction between aleatory and epistemic uncertainty, which is sometimes referred to, or thought of, as irreducible versus reducible uncertainty. Epistemic uncertainty exists in our map; as Eliezer put it, “The Bayesian says, ‘Uncertainty exists in the map, not in the territory.’” Aleatory uncertainty, however, exists in the territory. (Well, at least according to our map that uses quantum mechanics, according to Bells Theorem – like, say, the time at which a radioactive atom decays.) This is what people call quantum uncertainty, indeterminism, true randomness, or recently (and somewhat confusingly to myself) ontological randomness – referring to the fact that our ontology allows randomness, not that the ontology itself is in any way random. It may be better, in Lesswrong terms, to think of uncertainty versus randomness – while being aware that the wider world refers to both as uncertainty. But does the distinction matter?

To clarify a key point, many facts are treated as random, such as dice rolls, are actually mostly uncertain – in that with enough physics modeling and inputs, we could predict them. On the other hand, in chaotic systems, there is the possibility that the “true” quantum randomness can propagate upwards into macro-level uncertainty. For example, a sphere of highly refined and shaped uranium that is *exactly* at the critical mass will set off a nuclear chain reaction, or not, based on the quantum physics of whether the neutrons from one of the first set of decays sets off a chain reaction – after enough of them decay, it will be reduced beyond the critical mass, and become increasingly unlikely to set off a nuclear chain reaction. Of course, the question of whether the nuclear sphere is above or below the critical mass (given its geometry, etc.) can be a difficult to measure uncertainty, but it’s not aleatory – though some part of the question of whether it kills the guy trying to measure whether it’s just above or just below the critical mass will be random – so maybe it’s not worth finding out. And that brings me to the key point.

In a large class of risk problems, there are factors treated as aleatory – but they may be epistemic, just at a level where finding the “true” factors and outcomes is prohibitively expensive. Potentially, the timing of an earthquake that would happen at some point in the future could be determined exactly via a simulation of the relevant data. Why is it considered aleatory by most risk analysts? Well, doing it might require a destructive, currently technologically impossible deconstruction of the entire earth – making the earthquake irrelevant. We would start with measurement of the position, density, and stress of each relatively macroscopic structure, and the perform a very large physics simulation of the earth as it had existed beforehand. (We have lots of silicon from deconstructing the earth, so I’ll just assume we can now build a big enough computer to simulate this.) Of course, this is not worthwhile – but doing so would potentially show that the actual aleatory uncertainty involved is negligible. Or it could show that we need to model the macroscopically chaotic system to such a high fidelity that microscopic, fundamentally indeterminate factors actually matter – and it was truly aleatory uncertainty. (So we have epistemic uncertainty about whether it’s aleatory; if our map was of high enough fidelity, and was computable, we would know.)

It turns out that most of the time, for the types of problems being discussed, this distinction is irrelevant. If we know that the value of information to determine whether something is aleatory or epistemic is negative, we can treat the uncertainty as randomness. (And usually, we can figure this out via a quick order of magnitude calculation; Value of Perfect information is estimated to be worth $100 to figure out which side the dice lands on in this game, and building and testing / validating any model for predicting it would take me at least 10 hours, my time is worth at least $25/hour, it’s negative.) But sometimes, slightly improved models, and slightly better data, are feasible – and then worth checking whether there is some epistemic uncertainty that we can pay to reduce. In fact, for earthquakes, we’re doing that – we have monitoring systems that can give several minutes of warning, and geological models that can predict to some degree of accuracy the relative likelihood of different sized quakes.

So, in conclusion; most uncertainty is lack of resolution in our map, which we can call epistemic uncertainty. This is true even if lots of people call it “truly random” or irreducibly uncertain – or if they are fancy, aleatory uncertainty. Some of what we assume is uncertainty is really randomness. But lots of the epistemic uncertainty can be safely treated as aleatory randomness, and value of information is what actually makes a difference. And knowing the terminology used elsewhere can be helpful.

If you like this orange you must like that orange.  Well, maybe.  Tastes change, and maybe I already had an orange a little while ago, and maybe I'm not in the mood while someone else would be glad to have it, so it doesn't follow that because I liked this orange I must like that orange.

Comparing oranges and oranges seems like a set of two objects, but it's really four.  There's you, there's the orange, there's the other orange, and there's the perceived relation between you and the two oranges.  When it's just you and the oranges, things usually find a simple way work themselves out.

But when someone else comes into the room it's seldom oranges and oranges.   Other people are ever ready to tell you what you like.  If you like this orange you must like that apple, because they're both fruit.  Nah, can't stand apples unless they are baked.  It doesn't matter that they are both fruit, I don't care for apples.  Then the helping helpers will infer the inverse.  If you like this orange you can't like that apple.  Watch me - I'll like an apple just to spite you, or choke it down because there aren't any oranges to be had.

The nonsense comparisons just get more nonsensical.  If you like this orange you must like that color orange, you must!  That's the way it's always gone!  Well, I say if you like this orange you must like that porcupine.  See how silly it sounds?  As long as someone sees that fourth object in the set, a connection between the two things and you, they will hard-sell you that the orange and the very-not-orange are fully fungible.

That fourth object in the set, the perceived relation between the other three, gets its power from being invisible and assumed.  The assumption of relations in the set overpowers all the other objects in the set.  If you like this orange you are an orange-ist, because there's (a) you (b) the orange (c) your liking of the orange and (d) anybody that likes that orange is an orange-ist, that's the relation between you and the orange caused by your liking it.  The invisible fourth object in the set, the assumption of a relation, is now a stand-in for you.  You are no longer a person who in one place, in one time, in one way, liked an orange.  You are are an orange-ist.

If you are friends with that guy / read that book, and that guy / book exposed that idea, and that whole other guy with that idea did that thing, then you did that thing!  The four step process of replacing the man with a mannequin is the start of superstition.  Religion is realized in the replacement of the representation for the real.  Hard to believe that belief is so beleaguered but right here on this very planet in this very year there are nations where if you draw the wrong cartoon, read the wrong poem, or question the wrong answer, you go to prison.  Or worse.

Here's how they make the rotten trolly run.  If you said this one thing this one time then you believe - no, you are - this other thing.  A clergyman is not only a clergyman, they are a Good Person.  Good People do Good Deeds, and if the clergyman doesn't do good deeds, or if he does bad deeds, well, he's still a Good Person.  All four stations of Goodnessity are there: the clergyman, the Good Deeds clergymen are associated with, Good Deeds associated with Good People, and halleluia! clergymen are Good People.  And oh my but the four stations of Badnessism are there as well.  If you tell that one joke then you're a Bad Person.  That joke has the Bad Word in it, Bad People use that Bad Word, Bad People do Bad Deeds, so you did a Bad Deed!

It's four things. You, that thing you like, another thing and the proposed connection between the things. That connection is presented as more important than you.  The evidence shows that nothing is more to me than myself.  I'd not be here to tell you if this was not the case.  What other people think and do about me has its influences, but I don't confuse that with right or wrong or especially not Rights and Sins.  Egoism is the school of thought closest to my own, and that association draws from my own luster.

The pressure to be packed in a package deal comes in many forms.  Don't like too many kinds of art or music, be part of a scene.  Don't hold political or philosophical views, be a member of a party or a school.  Don't be online, be in a social network.  And most of all don't have a yen for truth, beauty and strength - be spiritual.

When the crowd crowns you with a trait, you're trapped.  To be identified as a whole by one of your parts is cutting.  Oh you're a massage therapist?  I have this pinch in my back.  You're a car mechanic?  You know, my car is just outside.  You do stand-up?  Tell me a joke, funny guy.  I heard you're a porn star, is that right?  Let's see those tits.  So you're a professional wrestler, eh?  I like that other wrestler better, the nice guy.  In every variation we are made out to be not ourselves but the thing other people think you are.  Man, that dude's a racist.  Heil hitler, you cartoon-drawer!  Her over there, she has a suicidal level of self-hatred and is an active enemy of all women.  She quit her job to be a mom when she was in her 20s.  There's something just creepy about that family down the hall, they're always happy.  Yeah, they're Mormons.  Fake vegan meat supports the aesthetic of carnivore culture.  No one more intolerant than the loud champions of toleration, no one more ready to divide than the unifiers of diversity.

In the United States, a slave knew he had a place: that of a slave.  In India, an Untouchable knew he had a place: that of an Untouchable.  The modern moral minders, starting with Stalin onward, developed a different delineator.  If you are seen to stray too far from the approved set of beliefs, you have no place.  You are to be stripped of your job, your career, your credentials, your home and your money.  The Good Guys in the White Hats are ever vigilant for any infraction.  Call them the improperatzzi.  What a remarkable coincidence that the virtue they advocate is the same as the group they are a member of.

I can't say I judge all men in all moments anew.  I've also decided to not ask you to do so.  That sounds too much like work.  I don't have the time or energy, much less the inclination, to always cast aside generalities, stereotypes, and biases.  In this very essay I may lump a whole spectrum of people I disagree with into the base categories of liars and fools.  But you and I both know some people are just jerks, and some people are solid citizens.  I'm a member of some groups, a friend of others.  Everyone I don't like has me in common.  If it suits me I'll give you a chance, but maybe I'm busy or angry that day and you're just going be hidden behind what I think of you based on some other thing at some other time.  You'll live.  My opinion isn't even all that important to me.

The troubles come when people decide that those who are different aren't to live.  Except for liars and fools, everyone on the planet knows that the Religion of Peace currently holds the title belt for murdering those who think or act differently than they do.  I keep hearing that there's a majority of Muslims who aren't like that, but I also keep not hearing about what they are doing to enlighten their brothers and sisters who keep misunderstanding Islam in the same way, century after century.  Maybe the numbers are there for the majority to reform the minority, but let's see some action.  A sound public shaming is a good start, and in this regard I do my part.  But again - I limit myself to that most pathetic and un-magical of all activities, writing, when I disagree.  The beheaders, the child-rapers, the enslavers, the kidnappers, the hijackers, the perpetually grieved - the Muslims - not so much.

There's no controversy, only a nontroversy.  A man can like music by ADULT. and Mildred Bailey.  A man can know a great deal about far right politics without being of the far right.  A man can be interested in beliefs about UFOs without believing in UFOs.  The scolds and the bullies secretly know this but don't want you in on their game.  They know what is bad for other people because they've seen the evidence - but somehow, they saw the evidence and didn't suffer from the exposure.  They are good enough to tell you what's good for you, but you aren't.  No thank you, you pinch-faced busybodies, I'll decide for myself what I like and do and think and believe.  I'll even take my lumps for the luxury.

The heart wants what the heart wants.  So does the groin.  I've made up a name for those who think otherwise: quantisexual.  A quantisexual is deeply invested in quantifying sex.  Who can have sex with who, what the arrangement is named, who shares that name and who doesn't.  Who is doing it right, who is doing it right but for the wrong reasons, who is doing it all wrong.  Not satisfied with the real-life cooties you can get from sex, a quantisexual invents forms of ritual contamination and cleanliness.  If you have even one stray thought about your own sex, you're bisexual.  If you're bisexual then you're queer.  If you're queer then you have to support all the other queers in all their queeriosities.  Even if you don't have sex at all there's a whole slew of cooties you can accessorize yourself with like 'cis' and 'demisexual' and 'asexual.'  The name for a thing becomes more important than the thing itself, like sheets being more sexy than what goes on between them.  The alphabet soup of alt-sex has more rules and restrictions than the Roman Catholic Church.  Quantisexuality is a fetish.  Hip hip hooray if you were born that way or if, by pretending it's your thing, you get to join the right in-groups.  Sex will go on without your names for it.

Standing at the rich banquet of life, far too many go with a cuisine they've been gifted by someone not even alive to share the meal.  Only these foods go together, and only in this order, and in this amount.  Not because to do otherwise leads to sickness or death, but because, well, other people might... see...  See what?  Me getting a few of these and a few of those, concerned less than they, enjoying more than they.  You do go on if you must keep kosher, hold halal and avoid fish on Friday.  All the more for me, pal, or maybe I'll just have a bite and be done.  What we do and like isn't limited to one item from column A and two items from column B.  Life is not a family meal or a package deal.  Beliefs and interests are all a big mess and probably not very important, so pull them together in a way that makes sense to you.  Just don't insist I sign on to your supper club.

The thing you like is the thing you like.  You didn't used to like it, and maybe you won't like it later.  You don't have to explain or understand it.  You don't have to get my approval for it.  If it stops working for you, you stop working for it.  Move on, and I'll be doing the same.

- Trevor Blake is the author of Confessions of a Failed Egoist.

The following is a summary of the short article The Avenger Ignored by Charles Sanders, published in Military History Magazine.

French Intelligence bought what turned out to be partial German military plans for WWI around a decade before the outbreak of the war.  The plans detailed a German invasion of Belgium for the obvious purpose of then attacking France from the North.  French intelligence compared map to territory finding that the plans explained much German construction which until then had seemed “random and unthreatening”.   Many in the French high command came to correctly believe in the plans’ authenticity and by 1907 French military strategy reflected this.

In 1913 many in the French military wanted to take an offensive posture with respect to the German threat.  This posture would be more justified if Germany intended to directly attack France rather than go via Belgium.  Therefore, French military officer Lt. Col. Edmond Buat falsely claimed to have found a copy of a German military document “under his seat during a train trip in Germany” that showed this.  This imaginary document purportedly outlined a direct German attack on France that would largely ignore Belgium.

Buat described but never showed the document to anyone.  His hoax was still believed and France based its military deployment on the imaginary document, to disastrous effects.  When the French military command received reports of an actual gigantic German attack on Belgium (consistent with the real military plans French Intelligence bought a decade ago) an important French General telephoned French commanders to say “reports on German forces in Belgium are greatly exaggerated.  There is no cause for alarm.”  France went ahead and executed its existing military strategy “as if the massive, deadly threat now clearly sweeping down from the north did not exist.”

In 1915 Buat admitted his deception, but this didn't stop him from going on to hold “numerous important assignments in the postwar army.”

I found The Avenger Ignored article through a History According to Bob Podcast.

As soon as I got out of college I got a job at a restaurant. At the time I had never had a job at a restaurant, but my mom had known the owners and I felt obligated to avoid performing badly. Yet inevitably I did perform badly, and how this performance was evaluated would greatly affect my way of perceiving my mistakes.

If you're entrenched in an organization, there's a good chance you have an idea of what it is you're supposed to do and what mistakes you will or will not be making. But suppose you're in a position like this one: by way of your ignorance you know you're going to make a lot of mistakes, and it's just a question of when and how much. Further, you know that if you make too many mistakes, you make people you care about look bad. And finally, there are a lot of unknown unknowns: you don't know what possible mistakes and acts of ignorance exist to begin with, so many mistakes you've made you will be blind to.

The proactive thing to do, naturally, is to try to minimize how many mistakes you make.

There are two key ways to gauge the depth of being told you have made a mistake. The first way is to take mistakes literally, as if no other mistake exists, and any other mistake would be pointed out to you. So if you correct this mistake, everything else should be fine. This is how you'd expect to take mistakes if you were, say, under the supervision of an editor.

But the second kind is where the title of this writeup comes in. Not everyone is literal, or critical enough to notice every mistake. Much of the time, you'll only receive news of a mistake if many other mistakes are already afoot, and this mistake just happens to stand out from the set of mistakes you've already made. And since you don't know what mistakes you could be making, you don't know if there are many more mistakes under your level of awareness that you could be correcting for, but aren't.

In short, you're tasked with avoiding a wrongness iceberg: a mistake indicative of a nautical mile of mistakes below the surface and your level of awareness.

This is a debilitating position to be in, because your mental map of your performance prior to discovering the iceberg needs to be completely rewritten; in addition to accounting for all of the new areas you need to work on, you will likely account for the embarrassment of realizing that you have opened up a new frontier of mistakes to reflect on from your period of unaware incompetence.

While I don't think it's impossible that people exist who have never been in a situation like this, I think anyone who dives into a new field or skill is familiar, at least, with this feeling of brief yet total incompetence. And if you're in a field with enough depth and subjective calls to allow for a wrongness iceberg scenario, there might not be much you can do to prevent it. The most you can do is provide adequate resistance for the inevitable.

That's why I've created this mental model to think about it constructively. In every situation where I've faced a wrongness iceberg, the anxiety has been catastrophic. If you can at least deal with it, you can realize why it is you're anxious and what's going on with your assessment of your own mistakes. From experience, knowing that I'm worried about making this kind of iceberg-revealing mistake is helpful for mitigating my stress. And if you can somehow preempt an iceberg, that's even better.

side note: I've extended this concept to other domains, and it works well. A "dishonesty iceberg" is when one person's lie reveals a nautical mile of lies below the surface, and an "attraction iceberg" is when one person's expression of attraction toward you are indicative of a much greater level of internal attraction.

Here is a presentation on the map and territory I'm planning on giving to my game theory class.

It's based on Liron's You Are A Brain post.

Any suggestions for improvements?

Related to: Living Luminously

Well? Should you?

Linked is a treatise on exactly this concept. If the effects of recording and classifying every thought pan out like the author says they'll pan out... well, read a (limited) excerpt (from the Introduction), and I'll let you decide whether it's worth your time.

If you do the things described in this book, you will be IMMOBILIZED for the duration of your commitment.The immobilization will come on gradually, but steadily. In the end, you will be incapable of going somewhere without your cache of notes, and will always want a pen and paper w/ you. When you do not have pen and paper, you will rely on complex memory pegging devices, described in "The Memory Book''. You will NEVER BE WITHOUT RECORD, and you will ALWAYS RECORD.

YOU MAY ALSO ARTICULATE. Your thoughts will be clearer to you than they have ever been before. You will see things you have never seen before. When someone shows you one corner, you'll have the other 3 in mind. This is both good and bad. It means you will have the right information at the right time in the right place. It also means you may have trouble shutting up. Your mileage may vary.

You will not only be immobilized in the arena of action, but you will also be immobilized in the arena of thought. This appears to be contradictory, but it's not really. When you are writing down your thoughts, you are making them clear to yourself, but when you revise your thoughts, it requires a lot of work - you have to update old ideas to point to new ideas. This discourages a lot of new thinking. There is also a "structural integrity'' to your old thoughts that will resist change. You may actively not-think certain things, because it would demand a lot of note keeping work. (Thus the notion that notebooks are best applied to things that are not changing.)

The full text is written in a stream-of-consciousness style, which is why I hesitated to post this topic in the first place. But there are probably note-taking junkies, or luminosity junkies, or otherwise interested folk amongst LW. So why not?

(Incidentally I'm reminded of Buckminster Fuller's Dymaxion Chronofile. I wonder how he managed it, or what benefits/costs it wrought?)

Or "The problems inherent in making a goal maximiser with a changing world model."

No paper clips were created or destroyed in the making of this script.

*This is an experimental post to try and get this point across. I'll write something similar for the type of systems I would like to explore, if this goes down well.*

Goal maximisers are great when you have a fixed ontology and you only have limited ways of getting information about the world. These aren't the case in AGI. Remember the map is not the territory, and the map is all that the utility maximiser can look at when deciding the utility of the future of actions.

TL;DR You can't have a utility maximiser choose how to alter the world model or how the world model should progress, if the utility is derived from that world model. If you have something else derive the world model it will conflict with the utility maximiser over resources and what to do in the world. Some method of resolving the conflicts is necessary which means we must go beyond normal model based utility maximisers.