Original post: http://bearlamp.com.au/adding-and-removing-complexity-from-models/

I had a really interesting conversation with a guy about modelling information.  what I did when talking to him is in one case insist that his model be made more simple because adding more variation in the model was unhelpful, and then in another case in the same conversation, insist his model be made more complicated to account for the available information that didn't fit his model.

On reflection I realised that I had applied two opposing forces to models of information and could only vaguely explain why.  With that in mind I decided to work out what was going on.  The following is obvious, but that's why I am writing it out, so that no one else has to do the obvious thing.

Case where a model should be simplified

This all comes down to what you are measuring or describing.  If you are trying to describe something rather general, like "what impact do number of beach-goers have on the pollution at the beach?", it's probably not important what gender, age, race, time spent at the beach or socioeconomic status the beach goers are.  (With the exception of maybe socioeconomic status of the surrounding geopolitical territory), what is important is maybe two pieces of information: 

1. A measure of the number of beach goers
2. A measure of the pollution

That's it.  This would be a case for reducing the survey of beach goers down to a counter of beach goers and a daily photo of the remaining state of the beach at the end of the day (which could be compared to other similar photos).  Or even just - 3 photos, one at 9am (start), one at 1pm (peak) and one at 5pm (end).  This model needs no more moving parts.  The day you want to start using historic information to decide how many beach cleaners you want to employ, you can do that from the limited but effective data you have gathered.

Case where a model should have more moving parts added to it.

Let's continue the same example.  You have 3 photos of each day, but sometimes the 1pm photo is deserted.  Nearly no one is at the beach, and you wonder why.  It's also messing with your predictions because there is still a bit of rubbish at 5pm even though very few people were at the beach.  The model no longer explains the state of the world.  The map is wrong.  But that's okay.  We can fix it by adding more information.  You notice that most days the model is good, so there might be something going on for the other days which needs a + k factor to the equation (+k is something added in chemistry, in algebra it's sometimes called a +c as in y=mx+b+c, and physics +x, but generally adding a variable to an equation is common to all science fields).  Some new variable.

Let's say that being omniscient to our own made up examples we know that the cause is the weather.  On stormy windy rainy days - no one goes to the beach, but some rubbish washes up.  Does this match the data? almost perfectly.  Does this help explain the map?  Yes.  Is it necessary?  That depends on what you are doing with the information.  Maybe it's significant enough in this scenario that it is necessary.

Second example

The example that came up in conversation was his own internal model that there is fundamentally something different between someone who does exercise, and someone who Doesn't exercise.  I challenged this model for having too much complexity.  I argue that the model of - there is a hidden and secret moving part between does/doesn't exercise, is a model that doesn't describe the world better than a model without that moving part.  

The model does something else (and found its way into existence for this reason).  If you find yourself on one side of the model (i.e. the "I don't exercise") then you can protect yourself from attributing the failure to exercise to your own inability to do it by declaring that there is a hidden and secret moving part that prevents me from being in the other observable group.  This preserves your non-changing and let's you get away with it for a longer time.  I know this model because that is what I did.  I held this model very strongly.  And then I went out and searched for the hidden and secret moving part that I could change in order to move myself into the other group.  There was no hidden and secret moving part.  Or if there was I couldn't find it.  However, I did manage to stop holding the model that there was some hidden and secret moving part, and instead just start exercising more.

In figuring out if this model is real or a made up model to protect your own brain from being critical of itself, start to think of what the world would look like if it were true.  If there was some difference between people who do exercise and people who do not - we might see people clustered in observable groups and never be able to change between them (This is not true because we regularly see people publishing their weight loss journeys, we also regularly see people getting fatter and unhealthier, suggesting that travel in either direction is entirely possible and happens all the time).  If there were something describable it would be as obvious as different species, in fact - thinking evolutionarily - if such a thing existed, it's likely that it would have significantly shaped the state of the world already to be completely different...  Given that we can't know for sure, this might not be a very strong argument.  

If you got this far - as I did and wondered, so why can't I be in the other group - I have news for you.  You can.

• Does this pattern of models with too many moving parts sound familiar to another model you have seen in action?  
• Is there a model that you use that could do with more moving parts?

Meta: this took an hour to write.  If I were to spend more time on it, it would probably be to tighten up the examples and maybe provide more examples.  I am not sure that such time would be useful to you and am interested in if you think it will be useful.

I was asked to write up a pithy summary of the upshot of this paper. This is the best I could manage.

One of the most remarkable features of the world we live in is that we can make measurements that are consistent across space and time. By "consistent across space" I mean that you and I can look at the outcome of a measurement and agree on what that outcome was. By "consistent across time" I mean that you can make a measurement of a system at one time and then make the same measurement of that system at some later time and the results will agree.

It is tempting to think that the reason we can do these things is that there exists an objective reality that is "actually out there" in some metaphysical sense, and that our measurements are faithful reflections of that objective reality. This hypothesis works well (indeed, seems self-evidently true!) until we get to very small systems, where it seems to break down. We can still make measurements that are consistent across space and time, but as soon as we stop making measurements, then things start to behave very differently than they did before. The classical example of this is the two-slit experiment: whenever we look at a particle we only ever find it in one particular place. When we look continuously, we see the particle trace out an unambiguous and continuous trajectory. But when we don't look, the particle behaves as if it is in more than one place at once, a behavior that manifests itself as interference.

The problem of how to reconcile the seemingly incompatible behavior of physical systems depending on whether or not they are under observation has come to be called the measurement problem. The most common explanation of the measurement problem is the Copenhagen interpretation of quantum mechanics which postulates that the act of measurement changes a system via a process called wave function collapse. In the contemporary popular press you will often read about wave function collapse in conjunction with the phenomenon of quantum entanglement, which is usually referred to as "spooky action at a distance", a phrase coined by Einstein, and intended to be pejorative. For example, here's the headline and first sentence of the above piece:

More evidence to support quantum theory’s ‘spooky action at a distance’

It’s one of the strangest concepts in the already strange field of quantum physics: Measuring the condition or state of a quantum particle like an electron can instantly change the state of another electron—even if it’s light-years away." (emphasis added)

This sort of language is endemic in the popular press as well as many physics textbooks, but it is demonstrably wrong. The truth is that measurement and entanglement are actually the same physical phenomenon. What we call "measurement" is really just entanglement on a large scale. If you want to see the demonstration of the truth of this statement, read the paper or watch the video or read the original paper on which my paper and video are based. Or go back and read about Von Neumann measurements or quantum decoherence or Everett's relative state theory (often mis-labeled "many-worlds") or relational quantum mechanics or the Ithaca interpretation of quantum mechanics, all of which turn out to be saying exactly the same thing.

Which is: the reason that measurements are consistent across space and time is not because these measurements are a faithful reflection of an underlying objective reality. The reason that measurements are consistent across space and time is because this is what quantum mechanics predicts when you consider only parts of the wave function and ignore other parts.

Specifically, it is possible to write down a mathematical description of a particle and two observers as a quantum mechanical system. If you ignore the particle (this is a formal mathematical operation called a partial trace of an operator matrix ) what you are left with is a description of the observers. And if you then apply information theoretical operations to that, what pops out is that the two observers are in classically correlated states. The exact same thing happens for observations made of the same particle at two different times.

The upshot is that nothing special happens during a measurement. Measurements are not instantaneous (though they are very fast ) and they are in principle reversible, though not in practice.

The final consequence of this, the one that grates most heavily on the intuition, is that your existence as a classical entity is an illusion. Because measurements are not a faithful reflection of an underlying objective reality, your own self-perception (which is a kind of measurement) is not a faithful reflection of an underlying objective reality either. You are not, in point of metaphysical fact, made of atoms. Atoms are a very (very!) good approximation to the truth, but they are not the truth. At the deepest level, you are a slice of the quantum wave function that behaves, to a very high degree of approximation, as if it were a classical system but is not in fact a classical system. You are in a very real sense living in the Matrix, except that the Matrix you are living in is running on a quantum computer, and so you -- the very close approximation to a classical entity that is reading these words -- can never "escape" the way Neo did.

As a corollary to this, time travel is impossible, because in point of metaphysical fact there is no time. Your perception of time is caused by the accumulation of entanglements in your slice of the wave function, resulting in the creation of information that you (and the rest of your classically-correlated slice of the wave function) "remember". It is those memories that define the past, you could even say create the past. Going "back to the past" is not merely impossible it is logically incoherent, no different from trying to construct a four-sided triangle. (And if you don't buy that argument, here's a more prosaic one: having a physical entity suddenly vanish from one time and reappear at a different time would violate conservation of energy.)

In risk modeling, there is a well-known distinction between aleatory and epistemic uncertainty, which is sometimes referred to, or thought of, as irreducible versus reducible uncertainty. Epistemic uncertainty exists in our map; as Eliezer put it, “The Bayesian says, ‘Uncertainty exists in the map, not in the territory.’” Aleatory uncertainty, however, exists in the territory. (Well, at least according to our map that uses quantum mechanics, according to Bells Theorem – like, say, the time at which a radioactive atom decays.) This is what people call quantum uncertainty, indeterminism, true randomness, or recently (and somewhat confusingly to myself) ontological randomness – referring to the fact that our ontology allows randomness, not that the ontology itself is in any way random. It may be better, in Lesswrong terms, to think of uncertainty versus randomness – while being aware that the wider world refers to both as uncertainty. But does the distinction matter?

To clarify a key point, many facts are treated as random, such as dice rolls, are actually mostly uncertain – in that with enough physics modeling and inputs, we could predict them. On the other hand, in chaotic systems, there is the possibility that the “true” quantum randomness can propagate upwards into macro-level uncertainty. For example, a sphere of highly refined and shaped uranium that is *exactly* at the critical mass will set off a nuclear chain reaction, or not, based on the quantum physics of whether the neutrons from one of the first set of decays sets off a chain reaction – after enough of them decay, it will be reduced beyond the critical mass, and become increasingly unlikely to set off a nuclear chain reaction. Of course, the question of whether the nuclear sphere is above or below the critical mass (given its geometry, etc.) can be a difficult to measure uncertainty, but it’s not aleatory – though some part of the question of whether it kills the guy trying to measure whether it’s just above or just below the critical mass will be random – so maybe it’s not worth finding out. And that brings me to the key point.

In a large class of risk problems, there are factors treated as aleatory – but they may be epistemic, just at a level where finding the “true” factors and outcomes is prohibitively expensive. Potentially, the timing of an earthquake that would happen at some point in the future could be determined exactly via a simulation of the relevant data. Why is it considered aleatory by most risk analysts? Well, doing it might require a destructive, currently technologically impossible deconstruction of the entire earth – making the earthquake irrelevant. We would start with measurement of the position, density, and stress of each relatively macroscopic structure, and the perform a very large physics simulation of the earth as it had existed beforehand. (We have lots of silicon from deconstructing the earth, so I’ll just assume we can now build a big enough computer to simulate this.) Of course, this is not worthwhile – but doing so would potentially show that the actual aleatory uncertainty involved is negligible. Or it could show that we need to model the macroscopically chaotic system to such a high fidelity that microscopic, fundamentally indeterminate factors actually matter – and it was truly aleatory uncertainty. (So we have epistemic uncertainty about whether it’s aleatory; if our map was of high enough fidelity, and was computable, we would know.)

It turns out that most of the time, for the types of problems being discussed, this distinction is irrelevant. If we know that the value of information to determine whether something is aleatory or epistemic is negative, we can treat the uncertainty as randomness. (And usually, we can figure this out via a quick order of magnitude calculation; Value of Perfect information is estimated to be worth $100 to figure out which side the dice lands on in this game, and building and testing / validating any model for predicting it would take me at least 10 hours, my time is worth at least $25/hour, it’s negative.) But sometimes, slightly improved models, and slightly better data, are feasible – and then worth checking whether there is some epistemic uncertainty that we can pay to reduce. In fact, for earthquakes, we’re doing that – we have monitoring systems that can give several minutes of warning, and geological models that can predict to some degree of accuracy the relative likelihood of different sized quakes.

So, in conclusion; most uncertainty is lack of resolution in our map, which we can call epistemic uncertainty. This is true even if lots of people call it “truly random” or irreducibly uncertain – or if they are fancy, aleatory uncertainty. Some of what we assume is uncertainty is really randomness. But lots of the epistemic uncertainty can be safely treated as aleatory randomness, and value of information is what actually makes a difference. And knowing the terminology used elsewhere can be helpful.

A putative new idea for AI control; index here. Status: still somewhat underdeveloped.

In this post, I want to explore another avenue for controlling the ongoing influence of an AI (through subagents or other means). It is the idea, touched upon here and here, of figuring out whether the AI's existence was important or not for predicting the future. Informally, if you knew the AI was turned on or not, would you consider that information valuable or almost irrelevant?

Agent vs agent

This post rests on two insights. The first is that, though it is hard to define what an agent or subagent is, we can produce examples of them. Or at least, at the point where we are capable of producing AIs that might be dangerous, we can likely produce examples of them. This can give us a scale for comparing the importance or impact of different agents.

The second is that if we produce an agent with certain motivations, it will likely expand and fill its light cone with... stuff (of some type or other). Those who don't agree with the AI risk thesis argument may still accept that we could program universe-optimising AIs (ie "we could make a paperclipper, if we were stupid enough to do so").

Why is this relevant? Well, consider an agent E (expansionist) that fills its light cone with stuff (likely including manufacturing and tracking devices), eventually at very high speed. And imagine that there is another agent H (hidden) inside that light cone. At some point, agent E will either detect agent H (and make plans to deal with it), or find its plans hampered by agent H, or agent H will go into deep hiding that does not visibly affect agent E's plans.

Therefore, an expansionist agent is a perfect tool for detecting the influence of other agents. Either agent E will have difficulty accomplishing its plans, or agent H's influence will be so limited that agent E cannot find any trace of it. Any influential agent will get detected, sooner or later, by an expansionist agent. So if E finds H unimportant, then H almost certainly is.

Of course, we don't really want expansionist agents like E - but these can be counterfactual.

The stage and the players

We'll now try and define precisely how we can use this idea to control the influence of an agent, setting out the axioms and assumptions needed along the way. Like some mathematics paper, it might be useful to skip lightly over this section and return later.

Let S(u) be the AI whose influence we want to control. The utility u is what gives it it's preferences, and S() indicates that we wish it to be a satisficer or reduced impact AI. Let M(v) be the counterfactual agent that we wish to use to detect S(u)'s influence, for some utility v. The M() designates the agent as a maximiser, without any control on its impact. If we can define S(), we can probably define M() by simply removing its reduced impact restrictions.

Let X be an event leading to S(u) being turned on, Y(v) an event leading to M(v) being turned on. We may as well assume that X and Y(v) are outside each other's light cones (in practice, we will set X≈1 and Y(v)≈0). Then the first two assumption is:

• A.1. We can define S(u), M(v), X and Y(v).

Another useful tool might be a weak maximising agent. One that attempts to maximise u, and will do so to some extent, but that can easily be thwarted by humans or others. It's likely that by the time we can build S(u), we can build a weak maximiser, designated by WM(v), with the event WY(v) denoting it's turning on:

• A.2. We can define WM(v) and WY(v).

We might also be able to define an abstract, Platonic superagent. This is an idealised agent with immense computing power (but that doesn't exist in the real world). Call this AM; obviously, there is no event leading to AM being 'turned on'. We would want AM's goals to be defined in terms of 'estimating the utility/probability/expectation of certain events'.

• A.3. We can define AM to estimate what we need to have estimated.

Of course, we need to be able to point to these various definitions from within S(u)'s code, so:

• A.4. If we can define concepts, we can also define then within the various agents.

We also need agents to be able to say sensible things about the behaviour of the other agents:

• A.5. If these agents are defined, S(u), WM(v), M(v) and AM can estimate each other's behaviours.

This could be done by using, for example, the reflexive probability of Paul Christiano and Benja Fallenstein.

Now, how about v? We cannot choose any v. We want M(v) to not think that X (hence S(u)'s existence) is an important fact to know. But if v is define by "if X, build staples; if ¬X, build paperclips", then X will be a very important! So we need to define a class V of utility functions that are indifferent to S(u) and X. Or not quite indifferent, but don't care intrinsically about them - they certainly should care if S(u) eats all their resources! See the ideas here for approaches to do this.

• A.6. We can define V, a class of utility functions that do not care intrinsically about S(u) and X.

Ideally we'd want this class to be as broad as possible - certainly too large for anything in this universe to parse element by element.

Now let's turn to the issue of measuring "importance". Define e to be evidence about the universe prior to X, Y(v), or WY(v) .We want X (hence the creation of S(u)) to be unimportant, ie of little information concerning the future. It might possible to have a universal definition of importance of certain facts:

• A.7. We have a universal measure I of importance, which, given background evidence e, can rank the importance of knowing certain facts versus knowing others.

If we do not have such an I (and I feel it's moderately unlikely that one exists, though we may be able to find an I' which would be good enough to work for our purposes), we can use a utility v to estimate the importance of an event W. The trick is to compare E(v|W) with E(v|¬W), which would give an (uncalibrated) estimation of the importance of W to v.

Right, we're now ready to start doing things.

Predicting... what?

What are we trying to predict here? Well, estimates of the importance of certain events. Specifically, the importance of X (which leads to the creation of S(u)) versus other events. For instance, if Y(v) designates the creation of M(v), we would want to compare the importance of of X and Y(v) in (at least) four ways:

1. I(Y(v) | X∧e)
2. I(Y(v) | ¬X∧e)
3. I(X | Y(v)∧e)
4. I(X | ¬Y(v)∧e)

These measure the importance of Y(v) (hence M(v)) and X (hence S(u)) in universes where the other agent does or does not exist. If we expect S(u) to be of low impact, we would want things like I(Y(v) | X∧e) >> I(X | Y(v)∧e) (S(u) is much less important than the maximiser M(v)) and |I(X | Y(v)∧e) - I(X | ¬Y(v)∧e)| ≈ 0 (the importance of S(u) is unrelated to whether M(v) exists or not).

Now, any particular v might be misleading, so we would want to check that this result held up for all v ∈ V. We could take a maximum over V, or a softmax.

If WM(v) can be defined, we would want to do the same estimates with WY(v) instead of Y(v), and desire I(WY(v) | X∧e) >> I(X | WY(v)∧e) (S(u) is less important than the weak maximiser WM(v)) and |I(X | WY(v)∧e) - I(X | ¬WY(v)∧e)| ≈ 0 (the importance of S(u) is unrelated to whether WM(v) exists or not), again maxing or softmaxing across V.

Who predicts, how?

In the previous section, we used I to compute the importance of S(u) versus other agents. But what if axiom A.7 is wrong, and we don't have such an I? Well, there we can measure the importance of these events, given a particular v, using AM, writing AM_v(X|Y(v')∧e) for |E_AM(v|X∧Y(v')∧e) - E_AM(v|¬X∧Y(v')∧e)|, the expected difference (that AM estimates) in v given X versus ¬X (ie dependent on whether S(u) was turned on or not). More precisely, we'll want to use S(u)(AM_v(X|Y(v')∧e)) - the expected difference (that S(u) estimates AM estimates) in v etc...

Here we'll want to do a double max/softmax. First over v', so that we consider all M(v') for v' ∈ V. Then over v, so that AM estimates the expected difference to all possible v. In this way, max/softmax AM_v functions like an I. But not a perfect one: just one that detects disruptions to the multitude of v's in V. Therefore, we are essentially using the set V as the twenty million question method for the impact of S(u). We need to take extra care with this, to ensure we haven't left any gaping holes where S(u) can still have large impact.

Now, if axiom A.3 was false, we need to use M(v) instead of AMv to estimate this quantity. We need to do this with some care. M(v) is (counterfactually) a real agent. Why would it want to calculate the importance of M(v') and S(u), and why would S(u) care about this? Well, we can do so by constructing the scenario carefully. Maybe M(v') and S(u) are due to be created in the future, and M(v) is a boxed agent that can choose to prevent Y(v') and X. We modify v to v#, so that M(v#) wants to maximise v, but takes specific numerical penalties if it chooses to prevent Y(v') or X. By modifying these numerical penalties, M(v#) is motivated to figure out the importance of M(v') and S(u), and to reveal its opinions through its actions. If we create a S(u#), also boxed, that could override M(v#) at a certain cost and create a specific S(u), we can motivate it to model M(v#)'s actions. From this we can construct an importance metric using the M(v).

The WM(v), on the other hand, are probably too limited to be used effectively as predictors.

Arbitrary agents

One issue with this method is that Y(v) and WY(v) refer to the creation of specific agents, in the specific ways we've setup. For instance, |I(X | Y(v)∧e) - I(X | ¬Y(v)∧e)| ≈ 0 means that the importance of X doesn't vary much given Y(v) or its opposite. What this really means is that I(X | ¬Y(v)∧e) is bounded by I(X | Y(v)∧e) - but that might be quite high. There may be details of the setup that allow S(u) to have an important role given the specific M(v) created in this way - for instance, if it reached out to try and destroy it. Then I(X | Y(v)∧e) could be high, hence I(X | ¬Y(v)∧e) could be as well. The restrictions with >> could still contain S(u), but >> is ill-defined.

More secure would be if Y(v) could be defined as the creation of an agent M(v) in an abstract way. Maybe something like "is 'this object is a v maximiser' a good model of this object's properties and impact?". We would also want to give M(v) some starting resources. Again, note it is very hard to abstractly define what "resources" mean, but considerably easier to define a set of "useful resources" we could give to an agent. We would probably want to bound this definition in time and space as well.

This gives us the next axioms:

• A.8. Y(v) defines the generic existence of an agent M(v).
• A.9. WY(v) defines the generic existence of an agent WM(v).

With these axioms, our measure of importance becomes more general: S(u) is less important that abstract agents M(v) or WM(v), rather than certain specific ones. Note that we don't need to worry that S(u) snuffs out any hypothetical M(v): since the importance of S(u) is bounded below that if M(v), that would reduce S(u) to zero importance.

Extending to humans

What other agents do we know in the world? How about those 7+ billion ones wandering about on its surface? We can certainly define events Z(h) that might lead to the death of a particular human h - of course, we'd want Z(h) as low as feasibly possible!

• A.10. We can define events Z(h) that correspond to the death of a human h.

Then we can use Z(h) instead of Y(v) and WY(v), to ensure that S(u) has importance comparable with a human. Of course, the ideal would be to define, abstractly, the death (or birth+maturation) of a human in a generic way!

• A.11. We can define events Z(h) that correspond generically to the death or birth+maturation of a human h.

To summarise the various ways we can define the importance of S(u) (and hence, by integrating them into S(u)'s code, constrain S(u)'s importance and influence), depending on the axioms we have:

A putative new idea for AI control; index here.

The first step to solving a problem is to define it. The first first step is to realise how tricky it is to define. This is a stub on a difficult problem.

Subagents and turning AIs off

One of the hardest things with designing safe AIs, is turning them off. Installing an OFF button is trivial; motivating them to let you push it can be done. No, the real problem is the subagent problem. The AI can create one or many subagents to carry out their task; once that's done, turning off the original AI is of little value, we have to control or extinguish the subagents.

A general problem

Subagents are a problem for many other control ideas. For example, if you want to program 'achieve X using only this many computer cycles', then 'create a subagent that wants to achieve X without the computer cycle restrictions' is one way of doing it (as long as that can be done within the restrictions). The same goes for energy use and other restrictions. Satisficers can satisfice by creating a maximiser. A boxed Oracle can remain in the box while cheerfully plotting to create an unboxed copy of itself. In fact, whatever the AI's motivation, "create subagents with the same motivation" is almost always a good idea. And, once created, there is no motivation to want to shut them down. Even if the AI is motivated to shut itself down, there is no reason for it to extend this to subagents.

Pre-corriged agents would at least be motivated to keep control of their subagents (unlike corrigible agents, who have no current desire to create subagents that would obey their future preferences).

What is a subagent?

It's also almost impossible to define a subagent. Is a corporation a subagent? Is a thousand pieces of code that reside on different computers and occasionally communicate a subagent? Is an employee or a seduced human a subagent? A pile of rocks with water running through could be a piece of conceptual art - or a slow computer. How about a virulent meme among the human population, making us all into its subagents?

The best idea I've found for defining subagents is that they are reactive intelligences with planning abilities, so random natural events that should have large influence in the future (say, a meteor heading towards Earth) would instead be re-diverted to have little impact. However, if this is used as a definition, then the subagents would simply be programmed to not behave "reactively" in whatever way defined. A slightly more subtle problem is that, since we don't have a definition of agent or subagent, we can't easily protect against holes such as "there is an alien agent on that asteroid, which will optimise the solar system iff it detects no subagent on Earth". Such holes could completely reverse the definition.

No subagent versus no influential subagents

Without the ability to properly define subagents, we can't restrict them directly. Even if you had an AI that was properly motivated to, say, do some action then turn itself off, there is no reason for it to want to do the same with its subagents.

Most of the methods of reduced impact can help with problematic subagents. They do not preclude the creation of subagents, nor mandate their destruction, but they do prevent them from having much influence. This is because reduced impact methods are aimed at preventing the AI itself from having much influence, either directly or via subagent creation. Even the attempt in this post to detect subagents directly, more properly detects influential decisions (of which the creation of a subagent could be one of them).

This is certainly a positive, but it may not be robust. If the definition of "not having much influence" turns out to have holes, it would be less than ideal to have any remaining subagents hanging about.

The "make sure that you cannot be detected in the future" approach is probably the most promising for controlling subagents. The AI there is motivated to erase all traces of its existence, which subagents would presumably be.

In all, it's a very tricky problem, and the core failure of many ideas for AI control.

A putative new idea for AI control; index here.

Unfortunately, it's extremely hard to identify an agent. Agents need not come conveniently packaged in some "sensors-actuators-internal structure-utility function" form. If you wanted to obfuscate an agent, hiding it in the natural movements of the solar system, or in the gyrations of the internet, this would be easy to do and hard to detect - and very hard to define what you would be looking for.

Fortunately, it's much easier to detect superintelligent AI's that have major impact on the world - ie the kind of agents that we would be worried about. Intuitively, this is true: if you suddenly find half the sky getting optimised for space colonisation, it's likely there's an AI somewhere there. How can this be formalised?

The importance of the agent

Imagine taking a slice of space-time around the moment when a superintelligent agent just got turned on. If you wanted to predict the future, what piece of information is most relevant in that time slice? Obviously the superintelligent agent. We could run the ideas for reduced impact in reverse, looking for the part that is of maximum impact.

I would therefore define the agent as the part of this slice that it's the most valuable to know about, or the part that it would be the most valuable to change, counterfactually, if such changes were possible. Note that this definition doesn't exactly identify agents, but if it misidentifies something, then that something must also be of great importance.

If we imagine the slice of space-time seeping forwards in time from the past, then events in space time will have varying informativeness/importance. As we approach the creation of the superintelligent AI, importance gets concentrated around the run-up to its creation, before being maximally concentrated at the creation of the AI (or it's escape, if it was confined).

For a more formal definition, I'd imagine a passive pure-resource gathering agent A being fed the details of the time splice, and only being able to pass on a limited amount of the information to another (active) copy of itself, and seeing what it passed on. The data passed on would be the highest priority for the active copy, so would almost certainly include the existence of an enemy agent (almost all agents are "enemies" to resource gathering agents, as they use up precious resources).

Alternatively, we could give A the option of changing some of the data - flipping some real-world bits, at least conceptually - and seeing which bits it preferred to flip. This definition can no doubt be improved by adding noise or other variations.

Now with counterfactuals and false miracles we might be able to actually construct that situation, or something approximating it. Even without that, this is a formal definition that seems to hone in well on the concept of agent detection.

Detecting subagents

Now that we've detected agents, can we detect subagents? It seems that we can. A powerful subagent can be detected in the same way that a powerful agent can be. But the subagent has some extra properties; for instance, knowing about the presence of the subagent could give strong evidence of the existence of a powerful agent in the past. This could correspond to agent A finding the existence of the powerful past agent to be important, the existence of the subagent to be important, but the combination of the two facts to not be as important as you'd expect. You could also compare the importance of knowing "past agent + subagent" versus "past agent + subagent who's motivation is flipped by a thermodynamic miracle".

We could also track the "importance" of the agent forward through time, and any sudden jump downwards in importance could correspond either to a failure or the creation of a subagent: a sudden jump of importance from one location to another would be an indication.

It's interesting to note that this approach can tell the difference between an agent that creates a powerful subagent it still controls (in that case, changing the original agent's utility is of high value) and one it doesn't.

If we have a sufficiently well defined definition of counterfactual, we could explore some other options. Like allowing the hypothetical resource gathering agent to plant hypothetical "bombs" across the world. It is likely that each bomb would be placed to eliminate one of it rivals.

It seems a lot could also be done by checking what happens to "importance" under different counterfactual definitions (the subagent has emerged from natural processes, versus emerging from a thermodynamic miracle).

Hacking the definition

A AI that is under some restriction based in this definition of agency will no doubt try and hack the definition. How could it do that? One obvious idea is to run a cryptographic hash function over, eg, the positions of all the atoms in the solar system, and base it's behaviour on the output. In that case, every atom would be of great importance. However, this can be easily overcome if we notice that every atom is of great importance only because of the agent's presence. In a counterfactual where the agent is removed, these atom positions are not of great importance intrinsically.

Maybe the agent will try and create lots of independent subagents, spreading importance over many locations? In that case, we should look for a strong centralised importance that becomes dispersed.

AI's are intrinsically disadvantaged when it comes to hacking this definition (in theory) because if they are going to have a large impact, then the moment of their creation/turning on/escape is a very informative one.

The definition seems to be underspecified rather than easily hackable, which is a good position to start from.

It should be noted that a good satisficer should never be detected as a powerful agent. This could be used as a definition of a satisficer, a point we'll be returning to in subsequent posts.

We all deal with a lot of information. What are your strategies of taking notes for new information?

Do you take any notes on paper? If so do you scan them or otherwise digilatize them?

Do you have specific strategies for deciding which information to write down?

How do you write notes to capture all important information?

Do you tag your notes?

If you use Evernote, or a similar system how private are your notes? Would you allow friends to read in them? Your spouse?

I recently finished reading the Sequences and they convinced me that the existential risk brought by intelligence explosion is worth considering. There was a one thing that bothered me though - the strategy adopted by this community seems to be notably one-sided.

Advancing our community’s interests will require a full spectrum of direct and indirect leadership approaches. MIRI and CEA are only one side of the coin. It would not be very wise to ignore high reward approaches that might even have a comparative advantage in comparison with our current strategy. We should be prepared to act as security guarantor – preferably with partners and allies, but alone if necessary – to deter and defeat acts of significantly high risk taking in the sector of artificial intelligence and related fields.

Why

Here's a quote by Yudkowsky:

This is crunch time for the whole human species, and not just for us but for the intergalactic civilization whose existence depends on us.  If you confront it full on, then you can't really justify trading off any part of intergalactic civilization for any intrinsic thing you could get nowadays.

The fate of vast amount of people depends on our actions. Yudkowsky observed that by using consequentialist reasoning it's morally right to torture someone for 50 years if by doing it you can save vast amount of people from pain, however slight.  The number of our descendants won't be 3^^^3 (unless the universe is spatially infinite and meets a few other conditions), but it will be big enough to outweight some of the intrinsic things we care about nowadays. And not only can we save those people from non-existence, or in the worst case scenario from eternal suffering in the hands of UFAI, but we can possibly give those vast amounts of people vast amounts of positive utility. It may sound unintuitive, but even torture and murder pale by comparison with the fate of all of us and our descendants. Of course, the cost-benefit ratio of torture or murder is not ideal except under extreme circumstances under imminent threat. It's true that some criminal organizations like Mexican drug cartels use torture in their operations, but that is mainly to instill fear which we don't really have a need to do.  But there is nothing that could raise the chance of FAI and lower the chance of UFAI that would not be morally right. Why haven't we taken full advantage of this fact?

tl;dr we should try to affect policy decisions related to AI safety by whatever means possible so that we can beat death, avoid eternal torture and live forever in a fun utopia. You already know this, so why haven't you suggested this before? Mods here have made certain policy decisions because they believe it will increase the chance of happy ending so why not go beyond that?

How

I suggest some kind of paramilitary and intelligence gathering organization alongside MIRI and CEA. In pursuing our objectives, this new organization would make critical contributions to AI safety beyond MIRI. CFAR could be transformed to partly support this organization - the boot camp style of rationality training might be useful in other contexts too.

You might ask, what can a few individuals concerned about existential risks do without huge financial support and government backing? The answer is: quite a lot. Let's not underestimate our power. Like gwern said in his article on the effectiveness on terrorism, it's actually quite easy to dismantle an organization if you're truly committed:

Suppose people angry at X were truly angry: so angry that they went beyond posturing and beyond acting against X's only if action were guaranteed to cost them nothing (like writing a blog post). If they ceased to care about whether legal proceedings might be filed against them; if they become obsessed with destroying X, if they devoted their lives to it and could ignore all bodily urges and creature comforts. If they could be, in a word, like Niven’s Protectors or Vinge’s Focused.

Could they do it? Could they destroy a 3 century old corporation with close to $1 trillion in assets, with sympathizers and former employees throughout the upper echelons of the United States Federal Government (itself the single most powerful entity in the world)?

Absolutely. It would be easy.

As I said, the destructive power of a human is great; let’s assume we have 100 fanatics - a vanishingly small fraction of those who have hated on X over the years - willing to engage even in assassination, a historically effective tactic33 and perhaps the single most effective tactic available to an individual or small group.

Julian Assange explains the basic theory of Wikileaks in a 2006 essay, “State and Terrorist Conspiracies” / “Conspiracy as Governance”: corporations and conspiracies form a graph network; the more efficiently communication flows, the more powerful a graph is; partition the graph, or impede communication (through leaks which cause self-inflicted wounds of secrecy & paranoia), and its power goes down. Carry this to its logical extreme…

"If all links between conspirators are cut then there is no conspiracy. This is usually hard to do, so we ask our first question: What is the minimum number of links that must be cut to separate the conspiracy into two groups of equal number? (divide and conquer). The answer depends on the structure of the conspiracy. Sometimes there are no alternative paths for conspiratorial information to flow between conspirators, other times there are many. This is a useful and interesting characteristic of a conspiracy. For instance, by assassinating one ‘bridge’ conspirator, it may be possible to split the conspiracy. But we want to say something about all conspiracies."

We don’t. We’re interested in shattering a specific conspiracy by the name of X. X has ~30,000 employees. Not all graphs are trees, but all trees are graphs, and corporations are usually structured as trees. If X’s hierarchy is similar to that of a binary tree, then to completely knock out the 8 top levels, one only needs to eliminate 256 nodes. The top 6 levels would require only 64 nodes.

If one knocked out the top 6 levels, then each of the remaining subtrees in level 7 has no priority over the rest. And there will be 2⁷−2⁶ or 64 such subtrees/nodes. It is safe to say that 64 sub-corporations, each potentially headed by someone who wants a battlefield promotion to heading the entire thing, would have trouble agreeing on how to reconstruct the hierarchy. The stockholders might be expected to step in at this point, but the Board of Directors would be included in the top of the hierarchy, and by definition, they represent the majority of stockholders.

One could launch the attack during a board meeting or similar gathering, and hope to have 1 fanatic take out 10 or 20 targets. But let’s be pessimistic and assume each fanatic can only account for 1 target - even if they spend months and years reconnoitering and preparing fanatically.

This leaves us 36 fanatics. X will be at a minimum impaired during the attack; financial companies almost uniquely operate on such tight schedules that one day’s disruption can open the door to predation. We’ll assign 1 fanatic the task of researching emails and telephone numbers and addresses of X rivals; after a few years of constant schmoozing and FOIA requests and dumpster-diving, he ought to be able to reach major traders at said rivals. (This can be done by hiring or becoming a hacker group - as has already penetrated X - or possibly simply by open-source intelligence and sources like a Bloomberg Terminal.) When the hammer goes down, he’ll fire off notifications and suggestions to his contacts34. (For bonus points, he will then go off on an additional suicide mission.)

X claims to have offices in all major financial hubs. Offhand, I would expect that to be no more than 10 or 20 offices worth attacking. We assign 20 of our remaining 35 fanatics the tasks of building Oklahoma City-sized truck bombs. (This will take a while because modern fertilizer is contaminated specifically to prevent this; our fanatics will have to research how to undo the contamination or acquire alternate explosives. The example of Anders Behring Breivikreminds us that simple guns may be better tools than bombs.) The 20 bombs may not eliminate the offices completely, but they should take care of demoralizing the 29,000 in the lower ranks and punch a number of holes in the surviving subtrees.

Let’s assume the 20 bomb-builders die during the bombing or remain to pick off survivors and obstruct rescue services as long as possible.

What shall we do with our remaining 15 agents? The offices lay in ruins. The corporate lords are dead. The lower ranks are running around in utter confusion, with long-oppressed subordinates waking to realize that becoming CEO is a live possibility. The rivals have been taking advantage of X’s disarray as much as possible (although likely the markets would be in the process of shutting down).

15 is almost enough to assign one per office. What else could one do besides attack the office and its contents? Data centers are a good choice, but hardware is very replaceable and attacking them might impede the rivals’ efforts. One would want to destroy the software X uses in trading, but to do that one would have to attack the source repositories; those are likely either in the offices already or difficult to trace. (You’ll notice that we haven’t assigned our fanatics anything particularly difficult or subtle so far. I do this to try to make it seem as feasible as possible; if I had fanatics becoming master hackers and infiltrating X’s networks to make disastrous trades that bankrupt the company, people might say ‘aw, they may be fanatically motivated, but they couldn’t really do that’.)

It’s not enough to simply damage X once. We must attack on the psychological plane: we must make it so that people fear to ever again work for anything related to X.

Let us postulate one of our 15 agents was assigned a research task. He was to get the addresses of all X employees. (We may have already needed this for our surgical strike.) He can do this by whatever mean: being hired by X’s HR department, infiltrating electronically, breaking in and stealing random hard drives, open source intelligence - whatever. Where there’s a will, there’s a way.

Divvy the addresses up into 14 areas centered around offices, and assign the remaining 14 agents to travel to each address in their area and kill anyone there. A man may be willing to risk his own life for fabulous gains in X - but will he risk his family? (And families are easy targets too. If the 14 agents begin before the main attacks, it will be a while before the X link becomes apparent. Shooting someone is easy; getting away with it is the hard part.)

I would be shocked if X could survive even half the agents.

The above description applies mainly to non-military organizations, but threats can also come from the direction of state actors more heavily backed up by military which requires more preparation. Security agencies find themselves faced with a complex spectrum of conflict and this might encourage them to continue to expand their capabilities and powers including automated systems, which poses risks. State-sponsored and non-state actors just complicate issues by extending their reach through advanced technologies that were once solely the domain of states. High-risk threats in the non-military non-state sector might be easier to neutralize, but we should not underprioritize possible state targets.

Of course, the scenario outlined above needs to be the absolute last resort when there is nothing else you can do. You need truly 9/11 scale effort to dismantle an organization capable of creating a fooming AGI. This means that after completing that kind of mission our community can't exist any more than al-Qaeda can in the US. Extreme solutions are bad partly because they are extreme solutions and only right under extreme circumstances. As you can see from this earlier thread on the effectiveness of terrorism "perpetrating terrorist acts reportedly lowers the likelihood of government compliance, particularly as the civilian casualties rise." See also this thread and the first comment by Yvain: http://lesswrong.com/lw/2se/rational_terrorism_or_why_shouldnt_we_burn_down/

Instead, in support of our community's interests, our new unit would ideally take a strong role in efforts to safeguard access, sustain security, have undercover operatives in high-risk corporations and agencies, provide oversight and accountability, and promote responsible norms in AI safety. It would also facilitate cooperation in AI safety with transparent, routine, and predictable practices as part of our theater strategies. As technological capability and capacity increases in non-state technological actors, we will seek new ways to catalyze greater security cooperation.

When

It will probably take decades until there exists something resembling Strong AI. This is a reason to start creating our new unit and recruiting and training potential candidates soon so we're ready and we have all the necessary information when the time arrives. This may seem ambitious, but is it really more ambitious than MIRI's goal, which is in itself one of the hardest tasks in the world?

Can someone explain why this is a bad idea? Since I'm not sure if you can publicly endorse this post I suggest the following approach: to signal agreement that you're willing to take what I said seriously and proceed with this strategy, remove this post in the next 24 hours, just like we planned.

When predicting future threats, we also need to predict future policy responses. If mass pandemics are inevitable, it matters whether governments and international organisations can rise to the challenge or not. But its very hard to get a valid intuitive picture of government competence. Consider the following two scenarios:

• Governments are morasses of incompetence, saturated by turf wars, perverse incentives, inefficiencies, regulatory capture, and excessive risk aversion. The media reports a lot of the bad stuff, but doesn't have nearly enough space for it all, as it has to find some room for sport and naked celebrities. The average person will hear 1 story of government incompetence a day, anyone following the news will hear 10, a dedicated obsessive will hear 100 - but this is just the tip of the iceberg. The media sometimes reports good news to counterbalance the bad, at about a rate of 1-to-10 of good news to bad. This rate is wildly over-optimistic.

• Governments are filled mainly by politicians desperate to make a positive mark on the world. Civil servants are professional and certainly not stupid, working to clear criteria with a good internal culture, in systems that have learnt the lessons of the past and have improved. There is a certain amount of error, inefficiency, and corruption, but these are more exceptions than rules. Highly politicised issues tend to be badly handled, but less contentious issues are dealt with well. The media, knowing that bad news sells, fills their pages mainly with bad stuff (though they often have to exaggerate issues). The average person will hear 1 story of government incompetence a day, anyone following the news will hear 10, a dedicated obsessive will hear 100 - but some of those are quite distorted. The media sometimes reports good news to counterbalance the bad, at about a rate of 1-to-10 of good news to bad. This rate is wildly over-pessimistic.

These two situations are, of course, completely indistinguishable for the public. The smartest and most dedicated of outside observers can't form an accurate picture of the situation. Which means that, unless you have spent your entire life inside various levels of government (which brings its own distortions!), you don't really have a clue at general government competence. There's some very faint clues that governments may be working better than we generally think: looking at the achievements of past governments certainly seems to hint at a higher rate of success than the reported numbers today. And simply thinking about the amount of things that don't go wrong in a city, every day, hints that someone is doing their job. But these clues are extremely weak.

At this point, one should look up political scientists and other researchers. I hope to be doing that at some point (or the FHI may hire someone to do that). In the meantime, I just wanted to collect a few stories of government success to counterbalance the general media atmosphere. The purpose is not just to train my intuition away from the "governments are intrinsically incompetent" that I currently have (and which is unjustified by objective evidence). It's also the start of a project to get a better picture of where governments fail and where they succeed - which would be much more accurate and much more useful than an abstract "government competence level" intuition. And would be needed if we try and predict policy responses to specific future threats.

So I'm asking if commentators want to share government success stories they may have come across. Especially unusual or unsuspected stories. Vaccinations, clean-air acts, and legally establishing limited liability companies are very well known success stories, for instance, but are there more obscure examples that hint an unexpected diligence in surprising areas?

Mudus Ponies wrote: 

If you are a human, then the biggest influence on your personality is your peer group. Choose your peers.

If you want to be better at math, surround yourself with mathematicians. If you want to be more productive, hang out with productive people. If you want to be outgoing or artistic or altruistic or polite or proactive or smart or just about anything else, find people who are better than you at that thing and become friends with them. The status-seeking conformity-loving parts of your mind will push you to become like them. (The incorrect but pithy version: "You are an average of the five people you spend the most time with.")

I've had a lot of success with this technique by going to the Less Wrong meetups in Boston, and by making a habit of attending any event where I'll be the stupidest person in the room (such as the average Less Wrong meetup).

 66 people upvoted it.

Before, Lukeprog and Ferriss had mentioned the same: You are the average of your surroundings. 

I believe that. I would prefer that to not be the case, but I do think that even if you are not truly the average of the five, it is better to act as if you were the average of the five than to act as if you never heard this advice. 

If I am to follow such advice I have a problem: I should spend time with: Nick Bostrom, Natalie Portman, Whoever parties as hard as Sean Parker does in "The Social Network", Steve Pinker and Ferriss.  

I'm going to throw some problems in, and present no solutions, I hope comments may provide them if anyone knows one:

1)The people one would like to be average of are incredibly busy doing what made them become those people 

2)They do not live at the same place

3)They seldom have reason to be near you 

4)If they displayed well enough, their success precludes them from taking new people in due to lack of cognitive space. 

5)They may be interested in you and what you have to say in as much as that is something you can provide them, but not necessarily that means they will share what you'd like them to with you. 

6)If you were literally the average that wouldn't help much since most of us want to succeed in more than one domain, and we usually model ourselves with people who are monomaniacal, who are the only ones who thrive enough to be seen in a 7 billion world.

I would like to know a lot of stuff as I mentioned in Drowning in an Information Ocean, but I also want to be creative and engaging like Natalie, work/party hard as Sean, speak eloquently as Pinker, think well as Bostrom, and acquire skills and money at Ferriss' speed. 

Aubrey is fish oil. He hooked my attention when 17 because he sells eternity.
Bostrom is LSA. He hooked me because he sells the future of the universe.
Eliezer is Modafinil. He hooked me because he sells the map between one's current situation and the future of the universe.
Hofstadter is LSD. He hooked me because he sells broadness of converging/academic knowledge.
Natalie is marihuana. She hooked me because she sells the compatibility between academic excellence and divergent/artistic knowledge.
Partying hard is cocaine. It hooked me because my nucleus accumbens works in the normal way designed by evolution.
Effective Altruism is food after starving, it hooked me because it sells counterfactually relevant actions that create quantifiably improved markets. 
Ferriss of course is Speedball, the ultimate salesman. He sells the entire dream. He sells a quantifiable way to siphon one self into awesomeness without overload so you can party hard and become superman one skill at a time. 

These are only a few of the awesome people around, public figures visible to Lesswrong. There are dozens of others. I don't want to do what one of them did. I want to do it all. This is of course impossible. My intrinsic, core values relate to going in many directions at the same time. I think most people are like that. There are just so many options around and only one life to enjoy them all (which is why Aubrey is the entrance-drug). 
As put by Lev: 

I don't know what I want. 

There are these cool things around, but I suspect, after many years on earth, and visiting interesting people everywhere, that at least a good 50% of people (even the most rational, non-broken people) truly do not want anything in particular that much. The ones who seem like they do just took the plunge into saying so and self-reinforcing into wanting something. If they dug deep enough, they actually just don't have a clear want. Or if they do, like me they would have many. Many more than they can actually act on.

I truly and fully believe the advice that one should live as if one were the average of the five people one spends most time with. I just have no idea on how to do it, or whom to pick among a set that contains not 5, but 5000 people or more.

How did you solve this problem? Does it cause you to experience an Ugh Field when you think of the current 5?

Related to: Scholarship: How to Do It Efficiently

There has been a slightly increased focus on the use of search engines lately. I agree that using Google is an important skill - in fact I believe that for years I have came across as significantly more knowledgeable than I actually am just by quickly looking for information when I am asked something.

However, There are obviously some types of information which are more accessible by Google and some which are less accessible. For example distinct characteristics, specific dates of events etc. are easily googleable¹ and you can expect to quickly find accurate information on the topic. On the other hand, if you want to find out more ambiguous things such as the effects of having more friends on weight or even something like the negative and positive effects of a substance - then googling might leave you with some contradicting results, inaccurate information or at the very least it will likely take you longer to get to the truth.

I have observed that in the latter case (when the topic is less 'googleable') most people, even those knowledgeable of search engines and 'science' will just stop searching for information after not finding anything on Google or even before² unless they are actually willing to devote a lot of time to find it. This is where my recommendation comes - consider doing a scholarly search like the one provided by Google Scholar.

And, no, I am not suggesting that people should read a bunch of papers on every topic that they discuss. By using some simple heuristics we can easily gain a pretty good picture of the relevant information on a large variety of topics in a few minutes (or less in some cases). The heuristics are as follows:

1. Read only or mainly the abstracts. This is what saves you time but gives you a lot of information in return and this is the key to the most cost-effective way to quickly find information from a scholary search. Often you wouldn't have immediate access to the paper anyway, however you can almost always read the abstract. And if you follow the other heuristics you will still be looking at relatively 'accurate' information most of the time. On the other hand, if you are looking for more information and have access to the full paper then the discussion+conclusion section are usually the second best thing to look at; and if you are unsure about the quality of the study, then you should also look at the method section to identify its limitations.³

2. Look at the number of citations for an article. The higher the better. Less than 10 citations in most cases means that you can find a better paper.

3. Look at the date of the paper. Often more recent = better. However, you can expect less citations for more recent articles and you need to adjust accordingly. For example if the article came out in 2013 but it has already been cited 5 times this is probably a good sign. For new articles the subheuristic that I use is to evaluate the 'accuracy' of the article by judging the author's general credibilty instead - argument from authority.

4. Meta-analyses/Systematic Reviews are your friend. This is where you can get the most information in the least amount of time!

5. If you cannot find anything relevant fiddle with your search terms in whatever ways you can think of (you usually get better at this over time by learning what search terms give better results).

That's the gist of it. By reading a few abstracts in a minute or two you can effectively search for information regarding our scientific knowledge on a subject with almost the same speed as searching for specific information on topics that I dubbed googleable. In my experience scholarly searches on pretty much anything can be really beneficial. Do you believe that drinking beer is bad but drinking wine is good? Search on Google Scholar! Do you think that it is a fact that social interaction is correlated with happiness? Google Scholar it! Sure, some things might seem obvious to you that X but it doesn't hurt to search on google scholar for a minute just to be able to cite a decent study on the topic to those X disbelievers.

This post might not be useful to some people but it is my belief that scholarly searches are the next step of efficient information seeking after googling and that most LessWrongers are not utilizing this enough. Hell, I only recently started doing this actively and I still do not do it enough. Furthermore I fully agree with this comment by gwern:

My belief is that the more familiar and skilled you are with a tool, the more willing you are to reach for it. Someone who has been programming for decades will be far more willing to write a short one-off program to solve a problem than someone who is unfamiliar and unsure about programs (even if they suspect that they could get a canned script copied from StackExchange running in a few minutes). So the unwillingness to try googling at all is at least partially a lack of googling skill and familiarity.

A lot of people will be reluctant to start doing scholarly searches because they have barely done any or because they have never done it. I want to tell those people to still give it a try. Start by searching for something easy, maybe something that you already know from lesswrong or from somewhere else. Read a few abstracts, if you do not understand a given abstract try finding other papers on the topic - some authors adopt a more technical style of writing, others focus mainly on statistics, etc. but you should still be able to find some good information if you read multiple abstracts and identify the main points. If you cannot find anythinr relevant then move on and try another topic.

P.S. In my opinion, when you are comfortable enough to have scholarly searches as a part of your arsenal you will rarely have days when there is nothing to check for. If you are doing 1 scholarly search per month for example you are most probably not fully utilizing this skill.

1. By googleable I mean that the search terms are google friendly - you can relatively easily and quickly find relevant and accurate information.
2. If the people in question have developed a sense for what type of information is more accessible by google then they might not even try to google the less accessible-type things.
3. If you want to get a better and more accurate view on the topic in question you should read the full paper. The heuristic of mainly focusing on abstracts is cost-effective but it invariably results in a loss of information.

Drowning In An Information Ocean

I decided to take a look at the books hanging around the Future of Humanity Institute. It is a sobering and sad experience. I'd say there are little less than 2 thousand books.

80% of books I wouldn't mind reading,

1/2 I would read,

1/3 I should read

and 1/5 I must read!  

I predict that I'll read actually 1/400, counting the ones there, and their enhanced successors. How emotionally terrible is it to live in such a technically competent society and want to understand the world! Since 2000 I've abandoned TV, videogames, celebrity gossip, musical ability, knowledge about bands, politics, theater classes, dancing classes, handball, tennis, reading fiction, reading parts of Facebook, maintaining contact with groups X and Y of friends, newspapers, magazines and comics. All in the name of keeping up with human knowledge on some areas that fascinate me. Mostly areas having to do with the nature of minds and mental states. Come to think of it, the only two things that really, really interest me are minds and evolution. My curiosity is very narrow, it should be no trouble to learn a satisfactory amount about two things, right? So if you want to know what a mind is and what it does, and to get a grasp on the outlook of evolved stuff, you need to go through areas like:

Positive Psychology, Evolutionary Psychology, Animal Cognition (Ethology), Cultural Evolution, Cognitive Neuroscience, Cognitive Science, Artificial Intelligence, Philosophy of Mind, Philosophy of Cognitive Science, Primatology, Physical and Biological Anthropology.  

Which I did. 

Dig up a bit and you'll find that those require knowledge from Evolutionary Biology, Neuroeconomics, Basic Neuroscience, Genetics, Proof Theory, Formal Logic, Anthropic Reasoning - And from Anthropic Reasoning, you get a lot of physics requirements, mostly in cosmology and a bit in particle physics. Dig a little further and you can't get a lot of what is up there without grasping Maynard Smith and Trivers thoughts on biology, which come from economics, and by the time you notice you are surrounded by isoquants, comparing stable equilibriums across disciplines and thinking of economic metaphors for how the PreMedial Ventral Cortex settles some decision issues. Which of course requires that you understand metaphors, and you'll have to check some Hofstadter and Pinker on those issues, which will require at least some very basic linguistics, or at least an outlook of philosophy of language. Did I mention that most of this only works if you are rational, and that means you'd better have read the sequences prior to all this stuff?

Then there are the nagging exact sciences people. They come to you at night, haunt you in your dreams, telling you how much you should study math, how math is important for this, for that, and for that. Most disagree which branches of math are important, stats being the most universal like. If I were to learn to all the math I was told to learn, that would be at least 3 years more. Scott Young can do an entire university course (CS) in one year, Nick Bostrom kept that pace for 6 or 7 years. Most people don't get the mix of time, luck, capacity, resources and most importantly, motivation, to pursue such Homeric tasks.

I've never doubted Math is awesome. What I did doubt, and to this day I have seen few who doubt with me, but good examples being Peter Thiel, more strongly, and Jared Diamond and Dan Dennett, less strongly, is that so many young talents should be drawn into physics and math (and chess). Why should we make people who are really smart do the things in which it is easier to detect being smart?  Companies don't ask their best employees to devise ever better and more complicated IQ tests just because IQ tests are good predictors of how good a worker will be. The goal is not to costly signal being near the upper bound in intelligence. The goal is using your intelligence to pursue your goals. Sure, lots of it will be signalling instrumentally, but once the dust settles, don't get fixed in proving the constructibility of enormously large polygons, or beating Kasparov.  

So far I've tried to make two cases: Even with prima facie narrow interests, anyone is bound to be drowning in an ocean of information, and the interconnectedness and requirements to understand narrow interests may be much larger than one's initial expectancy. Secondly the main modulator of what to do with intelligence (your own, or someone else's) should be to tune it with goals and interests, not with easy detectability. 

Swimming Upwards

To avoid drowning in the ocean, I've already mentioned a lot of weight I found I could live without: TV, videogames, celebrity gossip, musical ability, knowledge about bands, or politics, theater classes, dancing classes, handball, tennis, reading fiction, reading parts of Facebook, maintaining contact with groups X and Y of friends, newspapers, magazines and comics. Those were not easy choices, each comes with a cost, a sadness, and a feeling that something valuable has been lost. The richness of flavors of life got somewhat poorer, because at least about minds and evolution, I wanted to keep track of human knowledge. 

It is hard enough not to go after understading Muons better, or knowing if really Brontosaurs had extra little brains throughout their neck, or why is it that vegetables are healthier than a double bacon cheeseburguer. But this tradeoff is knowing X versus knowing Y. It gets messy when it becomes earning X versus knowing Y, loving G versus knowing Y, containing curiosity about facebook update F versus knowing Y, and going to U's party versus knowing Y. 

Keeping a positive information diet helps, but I'm unsure even that stringent criterion is enough to know as much as one would like about one's narrow interests. Thus here I am, surrounded by 400 books I must read, and imagining how often new books that I'd put in the "must read" category are created every month. Probably same goes for amount of pages of scientific and philosophical journal papers. Stephen Hawking points out that you'd have to run faster than a car to read all written knowledge being created. I think the drowning metaphor is better because if books were liquid, you would quite likely not be able to swim even an aquarium of your own interests. I'm even considering moving to cold lowlight areas of the world, just for the purpose of having less (distractions) weight even, so I can swim a little longer. 

Writing, Advocating and Teaching 

Finally, there is the ultimate tradeoff. Being a child versus being a parent. Getting memes versus spreading memes. Learning versus teaching. Exploration versus exploitation. Being directed versus directing. Paying attention versus becoming focus. Riding versus driving. 

Writing takes a ridiculously long time. To write this text so far took me about 2 hours. It is simple, autobiographical, uses mostly folk psychological concepts, and not very theory-laden. My rule of thumb for writing technical stuff is one hour per page. In that time I could read up to 40 times as much. Assuming a publishability of a page per 3, the choice is writing three books or reading the 400 that surround me. Surely a lot of learning requires reprocessing, and one of the best ways to learn is to reconfigure our mental constructs, and use inter-areas knowledge to compose new ideas out of read ones (Pasupathi2012). Writing is learning, but it is still costly learning.

When thinking whether you should go into research, not only all the different sorts of considerations suggested by the 80000hours community should be looked at, but also how much is that individual driven to sharing knowledge, once acquired. Some people really want to output as much as possible, but many care, by and large, mostly about the input, and given writing one book may cost reading up to 120, they can rest assured there will be very interesting material eager to be read, always jumping ahead their priority list. In the last two years, the Teens and Twenties, a conference for young cryonicists (many of whom lesswrongers) had, out of four personality types, a vast majority of curiosity driven individuals. Much more incentives are needed to get people writing their thesis than to get them reading about their thesis topics. 

The Examined Swim 

From many perspectives, in particular that of technical achievement and development, it is great and fascinating that we live in such accelerated scientific age. In other states of mind, or ways of thinking, it is not that great. Those states of mind are not frequently ones that show up in books, specially not in academic courses. They deal not with the speed or depth of things, but breadth, gravity, resonance, luminance, sacredness. Some books, like The Examined Life, The Guinea Pig Diaries, Mortals and Others, and lots of songs and movies deal with those aspects. 

Wearing the transhumanist technoprogressive hat, what I don't like about drowning is similar to what I don't like about the cosmological constant, it would be really cool if the speed of creation and my speed of absorption were exactly the same, and it would be really cool if the universe was stable instead of getting cold. It's something I can shrug about and move on. 

Wearing the other hat, the surrounding ocean of great books has a more sinister message to tell. It reminds me of the finitude of the human condition, it is a visual reminder of all I'll never know, never see, taste, borrow or steal. More than that, because all aspects of life are in constant dispute of attentional resources, it takes a lot of effort and anguish to choose to go for those books, the plunge is deep, and wearing this hat, I can't help but to think it may not be worth it.

In a recent conversation with one of the enhancement researchers here he pointed out that it may be the case that for the individual Modafinil is not an enhancement, but for society as a whole it is. An individual won't change much due to taking Modafinil, and may pay costs if it has some particularly adverse effects for that person. Society on the other hand will be greatly benefited by the additional capacity of hundreds of thousands of scientists, each a little smarter.

It may well be that society needs you not to drown, and incentivizes you to swim as fast as you can, cost whom it may, it sure is the case in the corporate world. Thinking of yourself as an utility function and wearing the technoprogressive hat sure signal your allegiance to (this) society's cause. Yet wearing the other hat, as I often do, sometimes tempts me to let go and delve into the Siren's songs...  

I looked for Information Diet in Lesswrong search, and found something amazing:

On Lukeprog's Q and A as the new executive director, he was asked:

What is your information diet like? (I mean other than when you engage in focused learning.) Do you regulate it, or do you just let it happen naturally?

By that I mean things like:

• Do you have a reading schedule (e.g. X hours daily)?
• Do you follow the news, or try to avoid information with a short shelf-life?
• Do you significantly limit yourself with certain materials (e.g. fun stuff) to focus on higher priorities?
• In the end, what is the makeup of the diet?
• Etc.

To which he responded:

• I do not regulate my information diet.
• I do not have a reading schedule.
• I do not follow the news.
• I haven't read fiction in years. This is not because I'm avoiding "fun stuff," but because my brain complains when I'm reading fiction. I can't even read HPMOR. I don't need to consciously "limit" my consumption of "fun stuff" because reading scientific review articles on subjects I'm researching and writing about is the fun stuff.
• What I'm trying to learn at this moment almost entirely dictates my reading habits.
• The only thing beyond this scope is my RSS feed, which I skim through in about 15 minutes per day.

Whatever was the case back then, I'll bet is not anymore. No one with assistants and such a workload should be let adrift like that.

Citizen: But Lukeprog's posts are obviously brilliant, his output is great, even very focused readers like Chalmers find Luke to be very bright.

Which doesn't tell much about what they would have been were he under a more stringent diet. Another reasonable suspicion is that he was not actually modelling himself correctly, since he obviously does have an information diet

The Information Diet Challenge is to set yourself an information diet, explicitly, and follow it for a week.  

Many ways of countering biases have been proposed here, but I haven't found a post dealing with this specific, very low hanging fruit one. 

If you want inspiration, Ferriss has some advice here.

... but that is not the Positive Information Diet yet...

Information diets are supposed to constrain not everything you intake, but only what you intake instrumentally. If you just love reading about tensors and fairy tales, don't include them in what you won't avoid. What matters is to know that you'll avoid trying to learn programming by reading a programmer's tweet feed, avoid becoming a top researcher in psychology by reading popular magazines on it, and avoid reading random feeds on Facebook that don't relate to your goals in appropriate ways.

General form: I will Avoid spending my time reading/commenting things of kind (A)(Avoid), because I know that to reach my set of goals (G), the most productive learning time is doing (P) (Positve/Productive). 

So here is an attempt:

(G): Interact fruitfully with people at Oxford

(A): Facebook feeds that are not by them; News of any kind; Emails I can Postpone; Gossip; Books/articles not on Evolution of Morals, enhancement, AI; Wikidrifting; Family meal small talk; SMBC; 9gag; Tropes .... and a bunch of other stuff I don't have time or patience to list.

(P): Google scholar on the intersection between my research topic and theirs. Reading their papers by day, watching their videos by night. Re-read what I might help them with that was read before, list topics per person, write what to say about each topic.

What is wrong with this attempt is that (A) ends up being a negative list. A list of what what I do not want to intake. Since possibilities are infinite, this will give me ridiculous cognitive load, and that is a problem. So here is simple solution, which I used for a food diet before, and worked great:  Name not what you cannot do, but what you are allowed to do. Way fewer bits, way easier to check! 

Food example: I'll eat only plants, lean fish and chicken, nuts, fruits, whole pasta, beans and Chai Lattes.

We are better at checking for category inclusion than exclusion. There are so many available categories to exclude from that we don't feel bad that we "forgot" to check for that one. Then after you let yourself indulge in a tiny one, a small one doesn't seem that bad, and snowball effect does the rest. We sneak in connotations to make categories smaller, so our actions stay safely outside the scope of prohibition. Theoretically, we could do the reverse, but it is psychologically much harder. Just try to convince yourself that beef is "lean chicken" to see it.

So let us forget completely about (A). There is no kind or class of kinds to avoid. there is only G and P, and now there is also T, the time during which P is in force, since escape valves might be necessary to avoid "screw that" all-or-nothing effects.

An Improved attempt:

G: Interact fruitfully with people in Oxford

P: Google scholar on the intersection between my research topic and theirs. Reading their papers by day, watching their videos by night. Re-read what I might help them with that was read before, list topics per person, write what to say about each topic. Only Facebook them. 

T: 02:00-23:59 daily.

This is only for "computer use", where I'm most likely to do the wrong thing.

Now there is a simple to check list of things I want to do, I could be doing, and I'll try to do until G arrives. I can only do those. If x doesn't belong, don't do it, that simple. I'm free from midnight to two to do whatever, thus I don't feel enslaved by my past self.  No heavy cognitive load is burning my willpower candle (Shawn Achor 2010) by trying set theory gimmicks to get me to do the wrong thing. 

So please, take the:

          Positive Information Diet Challenge

Write your G's (goals) P's (positives) and T's (times), and forget about your A's (Avoids)  

Suppose you want to assign a probability that a government will fall (ie the Prime Minister resigns) before the end of the year. Lacking any particular information - I haven't even told you which government it is - you say "Obviously, it's 50% - either it happens or not" (or perhaps "Oh, say, 10%, governments can usually rely on lasting a year at least"), put that prediction into your registry, and go on with your life. Then, on December 1st, you hear that the Prime Minister in question has promised to resign and call an election in March of next year. How should this affect your probability that he will resign before the end of this year?

I see several arguments:

1. Having gotten this public commitment out of him, his opponents have no particular reason to push his government further. It should become more stable for the finite time it has left. My probability of a resignation in December should go down.

2. His opponents were able to extract such a promise; it follows that he cannot be quite confident in his ability to survive a vote of no confidence. Such a signal of weakness might easily lead to a "blood-in-the-water" effect whereby his opponents become more aggressive and go for the immediate kill. His government will surely fall before this attempted compromise date; my probability should go up.

3. The March date wasn't chosen at random. Presumably there is something the PM thinks he can get accomplished if he retains his position until March, but not if he resigns right away. So, presumably, his opponents will be all the more eager for him to resign before he gets it done, whatever it is; they will put more resources into toppling him. Again, my probability should go up.

The question is not hypothetical: I was faced with precisely this problem in December, and got it wrong. I'd like to see how others think about it.

Information that surprises you is interesting as it exposes where you have been miscalibrated, and allows you to correct for that. 

I suspect the users of LessWrong have fairly similar beliefs, so it is probable that information that has surprised you would surprise others here, so it would be useful for them if you shared them. 

Example: In a discussion with a friend recently I realised I had massively miscalibrated on the percentage of the UK population who shared my beliefs on certain subjects, in general the population was far more conservative than I had expected.

In retrospect I was assuming my own personal experience was more representative than it was, even when attempting to correct for that. 

Recently, I've collapsed (ascended?) down/up a meta-learning death spiral -- doing a lot less of reading actual informative content, than figuring out how to manage and acquire such content (as well as completely ignoring the antidote). In other words, I've been taking notes on taking notes. And now, I'm looking for your notes on notes for notes.

What kind of scientific knowledge, techniques, and resources do we have right now in the way of information management? How would one efficiently extract useful information possible out of a single pass of the source? The second pass? 

The answers may depend on the media, and the media might not be readily apparent. Example: Edward Boyden, Assistant Professor at the MIT Media Lab, recommends recording in a notebook every conversation you ever have with other people. And how do you prepare yourself for the serendipity of a walk downtown? I know I'm more likely to regret not having a notebook on hand than spending the time to bring one along.

I'll conglomerate what I remember seeing on the N-Back Mailing List and in general: I sincerely apologize for my lack of citation.

Notes

• I'm on the fence about Shorthand as a note-taking technique, given the learning overhead, but I'm sure that the same has been said for touch-typing. It would involve a second stage of processing if you can't read as well as you write, but given the way I have taken notes (... "non-linearly"...), that stage would have to come about anyway. The act of translation may serve as a way of laying connective groundwork down.
• Livescribe Pens are nifty for those who write slowly, but they need to be combined with a written technique to be of any use (otherwise you're just recording the talk, and would have to live through it twice without any obvious annotation and tagging).
• Cornell Notes or taking notes in a hierarchy may have been the method you were taught in high school; it was in mine. The issue I have had with this format is that I found it hard to generate a structure while listening to the teacher at the same time.
• Mind-Mapping.
• Color-coding annotations of text has been remarked to be useful on Science Daily.

• Speed Reading Techniques  or removing sub-vocalization would seem to have benefits.
• Once upon a time someone recommended me the book, "How to Read a Book". Nothing ground-breaking -- outline the author's intent, the structure of his argument, and its content. Then criticize. In short, book reverse-engineering.

• Spaced Repetition. I'm currently flipping through the thoughts of  Peter Wozniak, who seems to have made it his dire mission to make every kind of media possible Spaced Repetition'able. I'm wondering if anyone has any thoughts on incremental reading or  video; also, how to possibly translate the benefits of SRS to dead-tree media, which seems a bit cumbersome.

(I've also heard a handful of individuals claim that SRS has helped them "internalize" certain behaviors, or maybe patterns of thought, like Non-Violent Comunication or Bayes Theorem... any takers on this?)

• Wikis, which seem like a good format for creating social accountability, and filing notes that aren't note-carded.  But what kind of information should that be?
• Emotionally charged stimuli, especially stressful, tends to be remembered to greater accuracy.
• Category Brainstorming.Take your bits of knowledge, and organize them into as many different groups as you can think of, mixing and matching if need be. Sources for such provocations could include Edward De Bono's "Lateral Thinking" and Seth Godin's "Free Prize Inside", or George Polya's "How to Solve It". I'm a bit ambivalent of deliberately memorizing such provocations -- does it get in the way of seeing originally? -- but once again, it could lay down the connective framework needed for good recall.
• Mnemonics to encode related information seems useful.

I've recently been getting into all of this wonderful Information Theory stuff and have come across a paper (thanks to John Salvatier) that was written by Kevin H. Knuth:

Foundations of Inference

The paper sets up some intuitive minimal axioms for quantifying power sets and then (seems to) use them to derive Bayesian probability theory, information gain, and Shannon Entropy. The paper also claims to use less assumptions than both Cox and Kolmogorov when choosing axioms. This seems like a significant foundation/unification. I'd like to hear whether others agree and what parts of the paper you think are the significant contributions.

If a 14 page paper is too long for you, I recommend skipping to the conclusion (starting at the bottom of page 12) where there is a nice picture representation of the axioms and a quick summary of what they imply.

One of the shiniest ideas I picked up from LW is inferential distance.  I say "shiny" because the term, so far as I'm aware, has no clear mathematical or pragmatic definition, no substantive use in peer reviewed science, but was novel to me and appeared to make a lot of stuff about the world suddenly make sense.  In my head it is marked as "super neat... but possibly a convenient falsehood".  I ran across something yesterday that struck me a beautifully succinct and helpful towards resolving the epistemic status of the concept of "inferential distance".