There are 2  probability puzzles that I like:

1) Suppose I tell you that I have 2 children and one of them is a boy, what is the probability that I have 2 boys?

The correct answer is not 1/2 but 1/3. How can that be? Well there are 4 possible combinations, BB,GG,BG and GB but but at least one is a boy so you can get rid of GG. So all that's left is BB,BG and GB; and in only one of those 3 possibilities do I have two boys.

2) Now I tell you that I have 2 children and one of them is a boy born on a Tuesday. What is the probability that I have 2 boys?

You may think that Tuesday is not useful information in this matter so the answer would be the same as the previous example, but you would be wrong. The correct answer is 13/27. How can that be?

Well there are 14 possibilities for EACH kid:
B-Mo, B-Tu, B-We, B-Th, B-Fr, B-Sa, B-Su
G-Mo, G-Tu, G-We, G-Th, G-Fr, G-Sa, G-Su

But I told you the one of my kids (the first or the second) was a boy born on a Tuesday so that narrows down the field of possibilities to:

First child: B-Tu, second child: B-Mo, B-Tu, B-We, B-Th, B-Fr, B-Sa, B-Su, G-Mo, G-Tu, G-We, G-Th, G-Fr, G-Sa, G-Su.

Second child: B-Tu, first child: B-Mo, B-We, B-Th, B-Fr, B-Sa, B-Su, G-Mo, G-Tu, G-We, G-Th, G-Fr, G-Sa, G-Su.

No need to put B-Tu in the second row because it's already accounted for in the first row.
So now just count them out, 14+13= 27 possibilities. How many result in 2 boys? Count them out again 7+6=13. So 13 out of 27 possibilities give you 2 boys.

  John K Clark

Apparently LW does a great job on refining rationality and dissolving confusions. But is it helpful when it comes to anything apart from designing Friendly AI, apart from a purely academic treatment of rationality? I'm currently unable to benefit from what I have so far read on LW, it actually made me even more unproductive, to an extent that I get nothing done anymore. Let me explain...

You have to know that I'm still in the process of acquiring a basic education. If I say basic, I mean basic. Since I got almost no formal education, what I do know (or know about) is largely on a very low level, yet I am plagued by problems that are themselves on a level that require the intellect and education of the folks here on LW. The problem with that is that I'm yet lacking most of the skills, tools and requisite know-how while the problems in question concern me as well. This often causes me to get stuck, I can't decide what to do. It also doesn't help much that I am the kind of person who is troubled by problems others probably don't even think about. An example from when I was much younger (around the age of 13) is when I was troubled by the fact that I could accidentally squash insects when walking over grass in our garden. Since I have never been a prodigy, far from it, it was kind of an unsolvable problem at that time, especially since I am unable to concentrate for very long and other similar problems are accumulating in my mind all the time. So what happened? After a time of paralysis and distress, as it happens often, I simply became reluctant and unwilling, angry at the world. I decided that it is not my fault that the world is designed like that and that I am not smart enough to solve the problem and do what is right. I finally managed to ignore it. But this happens all the time and the result is never satisfactory. This process too often ends in simply ignoring the problem or becoming unwilling to do anything at all. What I'm doing is not effective it seems, it already stole years of my life in which I could have learnt mathematics or other important things or done what I would have liked to do. You might wonder, shouldn't this insight cause me to ignore subsequent problems and just learn something or do what I want to do, do something that is more effective? Nope, it is exactly the kind of mantra that LW teaches that always makes me think about it rather than ignoring the problem and trying to reach my goals. Namely that the low probability of a certain event might be outweighed by the possible positive or negative 'utility' that the problem implies, especially ethical considerations. What could happen if I just ignore it, if I instead pursue another goal?

It's partly the choice that is killing me, do X or Y or continue thinking about either doing X or Y, or maybe search for some superior unknown unknown activity Z? For how long should I think about a decision and how long should I think about how long I should be thinking about it? Maybe the best analogy would be the browsing of Wikipedia on a subject that is unknown to you and over your head and clicking the first link to a page that explains a certain term you don't know just to repeat that process until you end up with 10 additional problems on an entry that is only vaguely relevant to the original problem you tried to solve. The problem is still there and you've to make the decision to ignore it, pursue it further or think about what to do. 

Recently I had blood vessel crack in my eye. Nothing to worry about, but I searched for it and became subsequently worried if something like that could happen in my brain too. It turned out that about 6 out of 100 people are predisposed for such brain aneurysms, especially people with high blood pressure. Now I might have a somewhat abnormal blood pressure and additional activity might make some blood vessel in my brain leak. Should I stop doing sports, should I even stop thinking too much because it increases the blood circulation in my brain (I noticed that I hear my blood flow when thinking too hard)? But how can I decide upon it without thinking? So I looked up on how to check if I was predisposed and it turned out that all tests are too risky. But maybe it would be rational to stop doing anything that could increase the blood pressure until there are less risky tests? And so I lost a few more days without accomplishing anything I wanted to accomplish. 

How I feel about LW

LW makes me aware of various problems and tells me about how important it is to do this or that but it doesn't provide the tools to choose my instrumental goals. Thanks to LW I learnt about Solomonoff induction. Great...fascinating! But wait, I also learnt that there is a slight problem: "the only problem with Solomonoff induction is that it is incomputable" Phew, thanks for wasting my time! See what I mean? I'm not saying that there is something wrong with what LW is doing, but people like me are missing some mid-level decision procedures on how to approach all the implications. I wish LW would also be teaching utilizable rationality skills by exemplifying the application of rationality to, and the dissolving of, real-life problems via the breakdown of decision procedures.

Take for example some of the top scoring posts. I intuitively understood them, agreed and upvoted them. My initial reaction was something along the lines of "wow great, those people think like me but are able to write down all I thought to be true." Yes, great, but that doesn't help me. I'm not a politician who's going to create a new policy for dealing with diseases. Even if I was, that post would be completely useless because it is utopic and not implementable. The same could be said about most other posts. Awesome but almost completely useless when it comes to living your life. 'Confidence levels inside and outside an argument' was an really enlightening post but only made me even more uncertain. If there is often no reason to assume very low probabilities then I'm still left with the very high risks of various possibilities, just that they suddenly became much more likely in some cases.

The problem with LW is that it tells me about those low probability high risk events. But I don't know enough to trust myself enough to overpower my gut feeling and my urge to do other things. I'd like to learn math etc., but maybe I should just work as baker or street builder to earn money to donate it to the SIAI? Maybe I should read the sequences to become more certain to be able to persuade myself? But maybe I should first learn some math to be able to read the sequences? But maybe I don't need that and would waste too much time learning math when I could earn money? And how do I know what math is important without reading the sequences? And what about what I really want to do, intuitively, should I just ignore that?

The following excerpt from a recent post, Recursively Self-Improving Human Intelligence, suggests to me that it is time for a reminder of the reason LW was started.

"[C]an anyone think of specific ways in which we can improve ourselves via iterative cycles? Is there a limit to how far we can currently improve our abilities by improving our abilities to improve our abilities? Or are these not the right questions; the concept a mere semantic illusion[?]"

These are not the right questions -- not because the concept is a semantic illusion, but rather because the questions are a little too selfish. I hope the author of the above words does not mind my saying that. It is the hope of the people who started this site (and my hope) that the readers of LW will eventually turn from the desire to improve their selves to the desire to improve the world. How the world (i.e., human civilization) can recursively self-improve has been extensively discussed on LW.

Eliezer started devoting a significant portion of his time and energy to non-selfish pursuits when he was still a teenager, and in the 12 years since then, he has definitely spent more of his time and energy improving the world than improving his self (where "self" is defined to include his income, status, access to important people and other elements of his situation). About 3 years ago, when she was 28 or 29, Anna Salamon started spending most of her waking hours trying to improve the world. Both will almost certainly devote the majority of rest of their lives to altruistic goals.

Self-improvement cannot be ignored or neglected even by pure altruists because the vast majority of people are not rational enough to cooperate with an Eliezer or an Anna without just slowing them down and the vast majority are not rational enough to avoid catastrophic mistakes were they to try without supervision to wield the most potent methods for improving the world. In other words, self-improvement cannot be ignored because now that we have modern science and technology, it takes more rationality than most people have just to be able to tell good from evil where "good" is defined as the actions that actually improve the world.

One of the main reasons Eliezer started LW is to increase the rationality of altruists and of people who will become altruists. In other words, of people committed to improving the world. (The other main reason is recruitment for Eliezer's altruistic FAI project and altruistic organization). If the only people whose rationality they could hope to increase through LW were completely selfish, Eliezer and Anna would probably have put a lot less time and energy into posting rationality clues on LW and a lot more into other altruistic plans.

Most altruists who are sufficiently strategic about their altruism come to believe that improving the effectiveness of other altruists is an extremely potent way to improve the world. Anna for example spends vastly more of her time and energy improving the rationality of other altruists than she spends improving her own rationality because that is the allotment of her resources that maximizes her altruistic goal of improving the world. Even the staff of the Singularity Institute who do not have Anna's teaching and helping skills and who consequently specialize in math, science and computers spend a significant fraction of their resources trying to improve the rationality of other altruists.

In summary, although no one (that I know of) is opposed to self-improvement's being the focus of most of the posts on LW and no one is opposed to non-altruists' using the site for self-improvement, this site was founded in the hope of increasing the rationality of altruists.

http://en.wikipedia.org/wiki/Public_international_law

Simple freindliness

Friendly AI, as believes by Hanson, is doomed to failure, since if the friendliness system is too complicated, the other AI projects generally will not apply it. In addition, any system of friendliness may still be doomed to failure - and more unclear it is, the more chances it has to fail.  By fail I mean that it will not ne accepted by most succseful AI project.

Thus, the friendliness system should be simple and clear, so it can be spread as widely as possible.

I roughly figured, what principles could form the basis of a simple friendliness:

0) Any one should understood that AI can be global risks and the friendliness of the system is needed. This basic understanding should be shared by maximum number of AI-groups (I think this is already done)

1) Architecture of AI should be such that it would use rules explicitly. (I.e. no genetic algorithms or neural networks)

2) the AI should obey commands of its creator, and clearly understand who is the creator and what is the format of commands.

3) AI must comply with all existing CRIMINAL an CIVIL laws. These laws are the first attempt to create a friendly AI – in the form of state. That is an attempt to describe good, safe human life using a system of rules. (Or system of precedents). And the number of volumes of laws and their interpretation speaks about complexity of this problem - but it has already been solved and it is not a sin to use the solution.

4) the AI should not have secrets from their creator. Moreover, he is obliged to inform him of all his thoughts. This avoids rebel of AI.

5) Each seldoptimizing of AI should be dosed in portions, under the control of the creator. And after each step mustbe run a full scan of system goals and effectivness.

6) the AI should be tested in a virtual environment (such as Secnod Life) for safety and adequacy.

7) AI projects should be registrated by centralized oversight bodies and receive safety certification from it.

Such obvious steps do not create absolutely safe AI (you can figure out how to bypass it out), but they make it much safer. In addition, they look quite natural and reasonable so they could be use by any AI project with different variations.

 Most of this steps are fallable. But without them the situation would be even worse. If each steps increase safety two times, 8 steps will increase it 256 times, which is good. Simple friendliness is plan B if mathematical FAI fails. 

Friendly AI, as believes by Hanson, is doomed to failure, since if the friendliness system is too complicated, the other AI projects generally will not apply it. In addition, any system of friendliness may still be doomed to failure - and more unclear it is, the more chances it has to fail. By fail I mean that it will not be accepted by most successful AI project. Thus, the friendliness system should be simple and clear, so it can be spread as widely as possible. I roughly figured, what principles could form the basis of a simple friendliness:

1) Any one should understood that AI can be global risks and the friendliness of the system is needed. This basic understanding should be shared by maximum number of AI-groups (I think this is alrready done)

2) Architecture of AI should be such that it would use rules explicitly. (I.e. no genetic algorithms or neural networks)

3) the AI should obey commands of its creator, and clearly understand who is the creator and what is the format of commands.

4) AI must comply with all existing criminal an civil laws. These laws are the first attempt to create a friendly AI – in the form of state. That is an attempt to describe good, safe human life using a system of rules. (Or system of precedents). And the number of volumes of laws and their interpretation speaks about complexity of this problem - but it has already been solved and it is not a sin to use the solution.

5) the AI should not have secrets from their creator. Moreover, he is obliged to inform him of all his thoughts. This avoids rebel of AI.

6) Each self optimizing of AI should be dosed in portions, under the control of the creator. And after each step must be run a full scan of system goals and effectiveness.

7) the AI should be tested in a virtual environment (such as Second Life) for safety and adequacy.

8) AI projects should be registrated by centralized oversight bodies and receive safety certification from it.

Such obvious steps do not create absolutely safe AI (you can figure out how to bypass it out), but they make it much safer. In addition, they look quite natural and reasonable so they could be use by any AI project with different variations. Most of this steps are fallable. But without them the situation would be even worse. If each steps increase safety two times, 8 steps will increase it 256 times, which is good. Simple friendliness is plan B if mathematical FAI fails.

The following is destilled from the comment section of an earlier post.

Definitions

absolute and universal: Something that applies to everything and every mind.

morality (moral world): A logically consistent system of normative theories.

reality (natural world): A logically consistent system of scientific (natural) theories.

normative theory: (Almost) any English sentence in imperative or including the word "should", "must", "to be allowed to" as the verb or equivalent construction, in contrast to descriptive theories.

mind: A mind is an intelligence that has values, desires and dislikes.

moral perception: Analogous to the sensory perceptions, a moral perception is the feeling of right and wrong.

Assumptions

A normative sentence arises as a result of the mind processing its values, desires and dislikes.

Ideas exist independently from the mind. Numbers don't stop to exist just because HAL dies.

Statement

In our everyday life, we don't question the reality, due to our sensory perception. We have moral perception as much as we have a sensory perception, therefore why should we question morality?

If you believe that the natural world is absolute and universal, then there is -- I currently think -- no good reason to doubt the existence of an absolute and universal moral world.

A text diagram for illustration

Absolute moralily

The absolute moral world, I am talking about, does encompass everything, including AI and alien intelligence. It does not mean that alien intelligence will behave similarly to us. Different moral problems require different solutions, as much as different objects behave differently according to the same physical theories. Objects in vacuum behave differently than in the atmosphere. Water behaves differently than ice, but they are all governed by the same physics, so I assume.

An Edo-ero samurai and a Wall Street banker may behave perfectly moral even if they act differently to the same problem due to the social environment. Maybe it is perfectly moral for AIs to kill and annihilate all humans, as much as it is perfectly possible that 218 of Russell's teapots are revolving around Gliese 581 g.

The intersubjective consensus

There are different sets of theories regarding the natural world: the biblical view, the theories underlying TCM, the theories underlying homeopathy, the theories underlying chiropractise and the scientific view. Many of them contradict each other. The scientific view is well-established because there is an intersubjective consensus on the usefulness of the methodology.

The methods used in moral discussions are by far not so rigidly defined as in science; it's called civil discourse. The arguments must be logical consistent and the outcomes and conclusions of the normative theory must face the empirical challenge, i.e. if you can derive from your normative theories that it is permissible to kill innocent children without any benefits, then there is probably something wrong.

Using this method, we have done quite a lot so far. We have established the UN Human Rights Charta, we have an elaborated system of international law, law itself being a manifestation of morality (denying the fact, that law is based on morality is like saying that technology isn't based on science).

Not everyone might agree and some say, "I think that chattel slavery is perfectly moral." And there are people who think that praying to an almighty pasta monster and dressing up as pirates will cure all the ills of the world. Does that mean that there is no absolute reality? Maybe.

Conclusion

As long as we have values, desires, dislikes and make judgements (which all of us do and which maybe is a defining characteristic of the human being beyond the biological basics), if we want to put these values into a logical consistent system, and if we believe that other minds with moral perception exist, then we have an absolute moral world.

So if we stop having any desires and stop making any judgements, that is if we lack any moral perception, then we may still believe in morality, as much as an agnostic won't deny the existence of God, but it would be totally irrelevant to us.

To the same degree, if someone lacks all the sensory perception, then the natural world becomes totally irrelevant to him or her.

Leading singularity proponent Ray Kurzweil co-authored a book titled Fantastic Voyage: Live Long Enough to Live Forever.  A singularity believer who thinks that if he makes it to the singularity he has an excelling chance of living forever, or at least for thousands of years, should be willing to sacrifice much for a slightly higher chance of living long enough to make it to the singularity.  This is why I think singularity believers make up a vastly disproportionate percentage of members of cryonics organizations. 

According to a new scientific article there is a medical procedure that might be able to greatly extend some peoples’ lives.  Although we don’t have a huge amount of data one small study showed that several hundred people on average lived 14 years longer than those that didn’t get the procedure.

Singularity proponents should be extremely interested in the procedure.  Indeed, a way of testing whether members of the SIAI such as Eliezer really and truly believe in the singularity is whether they at least seriously consider having the procedure.

The procedure is discussed at the end of this article.

Implications of the Theory of Universal Intelligence

If you hold the AIXI theory for universal intelligence to be correct; that it is a useful model for general intelligence at the quantitative limits, then you should take the Simulation Argument seriously.

AIXI shows us the structure of universal intelligence as computation approaches infinity.  Imagine that we had an infinite or near-infinite Turing Machine.  There then exists a relatively simple 'brute force' optimal algorithm for universal intelligence. 

Armed with such massive computation, we could just take all of our current observational data and then use a particular weighted search through the subspace of all possible programs that correctly predict this sequence (in this case all the data we have accumulated to date about our small observable slice of the universe).  AIXI in raw form is not computable (because of the halting problem), but the slightly modified time limited version is, and this is still universal and optimal.

The philosophical implication is that actually running such an algorithm on an infinite Turing Machine would have the interesting side effect of actually creating all such universes.

AIXI’s mechanics, based on Solomonoff Induction, bias against complex programs with an exponential falloff ( 2^-l(p) ), a mechanism similar to the principle of Occam’s Razor.  The bias against longer (and thus more complex) programs, lends a strong support to the goal of String Theorists, who are attempting to find a simple, shorter program that can unify all current physical theories into a single compact description of our universe.  We must note that to date, efforts towards this admirable (and well-justified) goal have not born fruit.  We may actually find that the simplest algorithm that explains our universe is more ad-hoc and complex than we would desire it to be.  But leaving that aside, imagine that there is some relatively simple program that concisely explains our universe.

If we look at the history of the universe to date, from the Big Bang to our current moment in time, there appears to be a clear local telic evolutionary arrow towards greater X, where X is sometimes described as or associated with: extropy, complexity, life, intelligence, computation, etc etc.  Its also fairly clear that X (however quantified) is an exponential function of time.  Moore’s Law is a specific example of this greater pattern.

This leads to a reasonable inductive assumption, let us call it the reasonable assumption of progress: local extropy will continue to increase exponentially for the foreseeable future, and thus so will intelligence and computation (both physical computational resources and algorithmic efficiency). The reasonable assumption of progress appears to be a universal trend, a fundamental emergent property of our physics.

Simulations

If you accept that the reasonable assumption of progress holds, then AIXI implies that we almost certainly live in a simulation now.

As our future descendants expand in computational resources and intelligence, they will approach the limits of universal intelligence.  AIXI says that any such powerful universal intelligence, no matter what its goals or motivations, will create many simulations which effectively are pocket universes.  

The AIXI model proposes that simulation is the core of intelligence (with human-like thoughts being simply one approximate algorithm), and as you approach the universal limits, the simulations which universal intelligences necessarily employ will approach the fidelity of real universes - complete with all the entailed trappings such as conscious simulated entities.

The reasonable assumption of progress modifies our big-picture view of cosmology and the predicted history and future of the universe.  A compact physical theory of our universe (or multiverse), when run forward on a sufficient Universal Turing Machine, will lead not to one single universe/multiverse, but an entire ensemble of such multi-verses embedded within each other in something like a hierarchy of Matryoshka dolls.

The number of possible levels of embedding and the branching factor at each step can be derived from physics itself, and although such derivations are preliminary and necessarily involve some significant unknowns (mainly related to the final physical limits of computation), suffice to say that we have sufficient evidence to believe that the branching factor is absolutely massive, and many levels of simulation embedding are possible.

Some seem to have an intrinsic bias against the idea bases solely on its strangeness.

Another common mistake stems from the anthropomorphic bias: people tend to image the simulators as future versions of themselves.

The space of potential future minds is vast, and it is a failure of imagination on our part to assume that our descendants will be similar to us in details, especially when we have specific reasons to conclude that they will be vastly more complex.

Asking whether future intelligences will run simulations for entertainment or other purposes are not the right questions, not even the right mode of thought.  They may, they may not, it is difficult to predict future goal systems.  But those aren’t important questions anyway, as all universe intelligences will ‘run’ simulations, simply because that precisely is the core nature of intelligence itself.  As intelligence expands exponentially into the future, the simulations expand in quantity and fidelity.

The Assemble of Multiverses

Some critics of the SA rationalize their way out by advancing a position of ignorance concerning the set of possible external universes our simulation may be embedded within.  The reasoning then concludes that since this set is essentially unknown, infinite and uniformly distributed, that the SA as such thus tells us nothing. These assumptions do not hold water.

Imagine our physical universe, and its minimal program encoding, as a point in a higher multi-dimensional space.  The entire aim of physics in a sense is related to AIXI itself: through physics we are searching for the simplest program that can consistently explain our observable universe.  As noted earlier, the SA then falls out naturally, because it appears that any universe of our type when ran forward necessarily leads to a vast fractal hierarchy of embedded simulated universes.

At the apex is the base level of reality and all the other simulated universes below it correspond to slightly different points in the space of all potential universes - as they are all slight approximations of the original.  But would other points in the space of universe-generating programs also generate observed universes like our own?

We know that the fundamental constants in the current physics are apparently well-tuned for life, thus our physics is a lone point in the topological space supporting complex life: even just tiny displacements in any direction result in lifeless universes.  The topological space around our physics is thus sparse for life/complexity/extropy.  There may be other topological hotspots, and if you go far enough in some direction you will necessarily find other universes in Tegmark’s Ultimate Ensemble that support life.  However, AIXI tells us that intelligences in those universes will simulate universes similar to their own, and thus nothing like our universe.

On the other hand we can expect our universe to be slightly different from its parent due to the constraints of simulation, and we may even eventually be able to discover evidence of the approximation itself.  There are some tentative hints from the long-standing failure to find a GUT of physics, and perhaps in the future we may find our universe is an ad-hoc approximation of a simpler (but more computationally expensive) GUT theory in the parent universe.

Alien Dreams

Our   Milky Way galaxy   is vast and old, consisting of hundreds of billions of stars, some of which are more than 13 billion years old, more than three times older than our sun.  We have direct evidence of technological civilization developing in 4 billion years from simple protozoans, but it is difficult to generalize past this single example.  However, we do now have mounting evidence that planets are common, the biological precursors to life are probably common, simple life may even have had a historical presence on mars, and all signs are mounting to support the  principle of mediocrity:  that our solar system is not a precious gem, but is in fact a typical random sample.

If the evidence for the mediocrity principle continues to mount, it provides a further strong support for the Simulation Argument.  If we are not the first technological civilization to have arisen, then technological civilization arose and achieved Singularity long ago, and we are thus astronomically more likely to be in an alien rather than posthuman simulation.

What does this change?

The set of simulation possibilities can be subdivided into PHS (posthuman historical), AHS (alien historical), and AFS (alien future) simulations (as posthuman future simulation is inconsistent).  If we discover that we are unlikely to be the first technological Singularity, we should assume AHS and AFS dominate.  For reasons beyond this scope, I imagine that the AFS set will outnumber the AHS set.

Historical simulations would aim for historical fidelity, but future simulations would aim for fidelity to a 'what-if' scenario, considering some hypothetical action the alien simulating civilization could take.  In this scenario, the first civilization to reach technological Singularity in the galaxy would spread out, gather knowledge about the entire galaxy, and create a massive number of simulations.  It would use these in the same way that all universal intelligences do: to consider the future implications of potential actions.

What kinds of actions?  

The first-born civilization would presumably encounter many planets that already harbor life in various stages, along with planets that could potentially harbor life.  It would use forward simulations to predict the final outcome of future civilizations developing on these worlds.  It would then rate them according to some ethical/utilitarian theory (we don't even need to speculate on the criteria), and it would consider and evaluate potential interventions to change the future historical trajectory of that world: removing undesirable future civilizations, pushing other worlds towards desirable future outcomes, and so on.

At the moment its hard to assign apriori weighting to future vs historical simulation possibilities, but the apparent age of the galaxy compared to the relative youth of our sun is a tentative hint that we live in a future simulation, and thus that our history has potentially been altered.