Previously: 2011; 2012-2013; 2013-2014; 2014-2015

"When I was one-and-twenty / I heard a wise man say, / 'Give crowns and pounds and guineas / But not your heart away; / Give pearls away and rubies / But keep your fancy free.' / But I was one-and-twenty, / No use to talk to me."

My past year of completed writings, sorted by topic:

Genetics:

• Embryo selection for intelligence cost-benefit analysis
  • meta-analysis of intelligence GCTAs, limits set by measurement error, current polygenic scores, possible gains with current IVF procedures, the benefits of selection on multiple complex traits, the possible annual value in the USA of selection & value of larger GWASes, societal consequences of various embryo selection scenarios, embryo count versus polygenic scores as limiting factors, comparison with iterated embryo selection, limits to total gains from iterated embryo selection etc.
• Wikipedia article on Genome-wide complex trait analysis (GCTA)

AI:

• Computational Complexity vs the Singularity
• Adding metadata to an RNN for mimicking individual author style
• Armstrong’s AI control problem: Reinforce.js demo

Biology:

• vitamin D & metformin for life-extension: cost-benefit analysis

Statistics:

• Candy Japan new packaging decision analysis
• “The Power of Twins: Revisiting Student’s Scottish Milk Experiment Example”
• Genius Revisited: Critiquing the Value of High IQ Elementary Schools
• Inferring mean ethnic IQs from very high IQ samples like TIP/SMPY

Cryptography:

• shell scripts for easy cryptographic timestamping

Misc:

• Newton’s comet apocalypse system

gwern.net itself has remained largely stable (some CSS fixes and image size changes); I continue to use Patreon and send out my newsletters.

These are my notes and observations after attending the Safety in Artificial Intelligence (SafArtInt) conference, which was co-hosted by the White House Office of Science and Technology Policy and Carnegie Mellon University on June 27 and 28. This isn't an organized summary of the content of the conference; rather, it's a selection of points which are relevant to the control problem. As a result, it suffers from selection bias: it looks like superintelligence and control-problem-relevant issues were discussed frequently, when in reality those issues were discussed less and I didn't write much about the more mundane parts.

SafArtInt has been the third out of a planned series of four conferences. The purpose of the conference series was twofold: the OSTP wanted to get other parts of the government moving on AI issues, and they also wanted to inform public opinion.

The other three conferences are about near term legal, social, and economic issues of AI. SafArtInt was about near term safety and reliability in AI systems. It was effectively the brainchild of Dr. Ed Felten, the deputy U.S. chief technology officer for the White House, who came up with the idea for it last year. CMU is a top computer science university and many of their own researchers attended, as well as some students. There were also researchers from other universities, some people from private sector AI including both Silicon Valley and government contracting, government researchers and policymakers from groups such as DARPA and NASA, a few people from the military/DoD, and a few control problem researchers. As far as I could tell, everyone except a few university researchers were from the U.S., although I did not meet many people. There were about 70-100 people watching the presentations at any given time, and I had conversations with about twelve of the people who were not affiliated with existential risk organizations, as well as of course all of those who were affiliated. The conference was split with a few presentations on the 27th and the majority of presentations on the 28th. Not everyone was there for both days.

Felten believes that neither "robot apocalypses" nor "mass unemployment" are likely. It soon became apparent that the majority of others present at the conference felt the same way with regard to superintelligence. The general intention among researchers and policymakers at the conference could be summarized as follows: we need to make sure that the AI systems we develop in the near future will not be responsible for any accidents, because if accidents do happen then they will spark public fears about AI, which would lead to a dearth of funding for AI research and an inability to realize the corresponding social and economic benefits. Of course, that doesn't change the fact that they strongly care about safety in its own right and have significant pragmatic needs for robust and reliable AI systems.

Most of the talks were about verification and reliability in modern day AI systems. So they were concerned with AI systems that would give poor results or be unreliable in the narrow domains where they are being applied in the near future. They mostly focused on "safety-critical" systems, where failure of an AI program would result in serious negative consequences: automated vehicles were a common topic of interest, as well as the use of AI in healthcare systems. A recurring theme was that we have to be more rigorous in demonstrating safety and do actual hazard analyses on AI systems, and another was that we need the AI safety field to succeed in ways that the cybersecurity field has failed. Another general belief was that long term AI safety, such as concerns about the ability of humans to control AIs, was not a serious issue.

On average, the presentations were moderately technical. They were mostly focused on machine learning systems, although there was significant discussion of cybersecurity techniques.

The first talk was given by Eric Horvitz of Microsoft. He discussed some approaches for pushing into new directions in AI safety. Instead of merely trying to reduce the errors spotted according to one model, we should look out for "unknown unknowns" by stacking models and looking at problems which appear on any of them, a theme which would be presented by other researchers as well in later presentations. He discussed optimization under uncertain parameters, sensitivity analysis to uncertain parameters, and 'wireheading' or short-circuiting of reinforcement learning systems (which he believes can be guarded against by using 'reflective analysis'). Finally, he brought up the concerns about superintelligence, which sparked amused reactions in the audience. He said that scientists should address concerns about superintelligence, which he aptly described as the 'elephant in the room', noting that it was the reason that some people were at the conference. He said that scientists will have to engage with public concerns, while also noting that there were experts who were worried about superintelligence and that there would have to be engagement with the experts' concerns. He did not comment on whether he believed that these concerns were reasonable or not.

An issue which came up in the Q&A afterwards was that we need to deal with mis-structured utility functions in AI, because it is often the case that the specific tradeoffs and utilities which humans claim to value often lead to results which the humans don't like. So we need to have structural uncertainty about our utility models. The difficulty of finding good objective functions for AIs would eventually be discussed in many other presentations as well.

The next talk was given by Andrew Moore of Carnegie Mellon University, who claimed that his talk represented the consensus of computer scientists at the school. He claimed that the stakes of AI safety were very high - namely, that AI has the capability to save many people's lives in the near future, but if there are any accidents involving AI then public fears could lead to freezes in AI research and development. He highlighted the public's irrational tendencies wherein a single accident could cause people to overlook and ignore hundreds of invisible lives saved. He specifically mentioned a 12-24 month timeframe for these issues.

Moore said that verification of AI system safety will be difficult due to the combinatorial explosion of AI behaviors. He talked about meta-machine-learning as a solution to this, something which is being investigated under the direction of Lawrence Schuette at the Office of Naval Research. Moore also said that military AI systems require high verification standards and that development timelines for these systems are long. He talked about two different approaches to AI safety, stochastic testing and theorem proving - the process of doing the latter often leads to the discovery of unsafe edge cases.

He also discussed AI ethics, giving an example 'trolley problem' where AI cars would have to choose whether to hit a deer in order to provide a slightly higher probability of survival for the human driver. He said that we would need hash-defined constants to tell vehicle AIs how many deer a human is worth. He also said that we would need to find compromises in death-pleasantry tradeoffs, for instance where the safety of self-driving cars depends on the speed and routes on which they are driven. He compared the issue to civil engineering where engineers have to operate with an assumption about how much money they would spend to save a human life.

He concluded by saying that we need policymakers, company executives, scientists, and startups to all be involved in AI safety. He said that the research community stands to gain or lose together, and that there is a shared responsibility among researchers and developers to avoid triggering another AI winter through unsafe AI designs.

The next presentation was by Richard Mallah of the Future of Life Institute, who was there to represent "Medium Term AI Safety". He pointed out the explicit/implicit distinction between different modeling techniques in AI systems, as well as the explicit/implicit distinction between different AI actuation techniques. He talked about the difficulty of value specification and the concept of instrumental subgoals as an important issue in the case of complex AIs which are beyond human understanding. He said that even a slight misalignment of AI values with regard to human values along one parameter could lead to a strongly negative outcome, because machine learning parameters don't strictly correspond to the things that humans care about.

Mallah stated that open-world discovery leads to self-discovery, which can lead to reward hacking or a loss of control. He underscored the importance of causal accounting, which is distinguishing causation from correlation in AI systems. He said that we should extend machine learning verification to self-modification. Finally, he talked about introducing non-self-centered ontology to AI systems and bounding their behavior.

The audience was generally quiet and respectful during Richard's talk. I sensed that at least a few of them labelled him as part of the 'superintelligence out-group' and dismissed him accordingly, but I did not learn what most people's thoughts or reactions were. In the next panel featuring three speakers, he wasn't the recipient of any questions regarding his presentation or ideas.

Tom Mitchell from CMU gave the next talk. He talked about both making AI systems safer, and using AI to make other systems safer. He said that risks to humanity from other kinds of issues besides AI were the "big deals of 2016" and that we should make sure that the potential of AIs to solve these problems is realized. He wanted to focus on the detection and remediation of all failures in AI systems. He said that it is a novel issue that learning systems defy standard pre-testing ("as Richard mentioned") and also brought up the purposeful use of AI for dangerous things.

Some interesting points were raised in the panel. Andrew did not have a direct response to the implications of AI ethics being determined by the predominantly white people of the US/UK where most AIs are being developed. He said that ethics in AIs will have to be decided by society, regulators, manufacturers, and human rights organizations in conjunction. He also said that our cost functions for AIs will have to get more and more complicated as AIs get better, and he said that he wants to separate unintended failures from superintelligence type scenarios. On trolley problems in self driving cars and similar issues, he said "it's got to be complicated and messy."

Dario Amodei of Google Deepbrain, who co-authored the paper on concrete problems in AI safety, gave the next talk. He said that the public focus is too much on AGI/ASI and wants more focus on concrete/empirical approaches. He discussed the same problems that pose issues in advanced general AI, including flawed objective functions and reward hacking. He said that he sees long term concerns about AGI/ASI as "extreme versions of accident risk" and that he thinks it's too early to work directly on them, but he believes that if you want to deal with them then the best way to do it is to start with safety in current systems. Mostly he summarized the Google paper in his talk.

In her presentation, Claire Le Goues of CMU said "before we talk about Skynet we should focus on problems that we already have." She mostly talked about analogies between software bugs and AI safety, the similarities and differences between the two and what we can learn from software debugging to help with AI safety.

Robert Rahmer of IARPA discussed CAUSE, a cyberintelligence forecasting program which promises to help predict cyber attacks. It is a program which is still being put together.

In the panel of the above three, autonomous weapons were discussed, but no clear policy stances were presented.

John Launchbury gave a talk on DARPA research and the big picture of AI development. He pointed out that DARPA work leads to commercial applications and that progress in AI comes from sustained government investment. He classified AI capabilities into "describing," "predicting," and "explaining" in order of increasing difficulty, and he pointed out that old fashioned "describing" still plays a large role in AI verification. He said that "explaining" AIs would need transparent decisionmaking and probabilistic programming (the latter would also be discussed by others at the conference).

The next talk came from Jason Gaverick Matheny, the director of IARPA. Matheny talked about four requirements in current and future AI systems: verification, validation, security, and control. He wanted "auditability" in AI systems as a weaker form of explainability. He talked about the importance of "corner cases" for national intelligence purposes, the low probability, high stakes situations where we have limited data - these are situations where we have significant need for analysis but where the traditional machine learning approach doesn't work because of its overwhelming focus on data. Another aspect of national defense is that it has a slower decision tempo, longer timelines, and longer-viewing optics about future events.

He said that assessing local progress in machine learning development would be important for global security and that we therefore need benchmarks to measure progress in AIs. He ended with a concrete invitation for research proposals from anyone (educated or not), for both large scale research and for smaller studies ("seedlings") that could take us "from disbelief to doubt".

The difference in timescales between different groups was something I noticed later on, after hearing someone from the DoD describe their agency as having a longer timeframe than the Homeland Security Agency, and someone from the White House describe their work as being crisis reactionary.

The next presentation was from Andrew Grotto, senior director of cybersecurity policy at the National Security Council. He drew a close parallel from the issue of genetically modified crops in Europe in the 1990's to modern day artificial intelligence. He pointed out that Europe utterly failed to achieve widespread cultivation of GMO crops as a result of public backlash. He said that the widespread economic and health benefits of GMO crops were ignored by the public, who instead focused on a few health incidents which undermined trust in the government and crop producers. He had three key points: that risk frameworks matter, that you should never assume that the benefits of new technology will be widely perceived by the public, and that we're all in this together with regard to funding, research progress and public perception.

In the Q&A between Launchbury, Matheny, and Grotto after Grotto's presentation, it was mentioned that the economic interests of farmers worried about displacement also played a role in populist rejection of GMOs, and that a similar dynamic could play out with regard to automation causing structural unemployment. Grotto was also asked what to do about bad publicity which seeks to sink progress in order to avoid risks. He said that meetings like SafArtInt and open public dialogue were good.

One person asked what Launchbury wanted to do about AI arms races with multiple countries trying to "get there" and whether he thinks we should go "slow and secure" or "fast and risky" in AI development, a question which provoked laughter in the audience. He said we should go "fast and secure" and wasn't concerned. He said that secure designs for the Internet once existed, but the one which took off was the one which was open and flexible.

Another person asked how we could avoid discounting outliers in our models, referencing Matheny's point that we need to include corner cases. Matheny affirmed that data quality is a limiting factor to many of our machine learning capabilities. At IARPA, we generally try to include outliers until they are sure that they are erroneous, said Matheny.

Another presentation came from Tom Dietterich, president of the Association for the Advancement of Artificial Intelligence. He said that we have not focused enough on safety, reliability and robustness in AI and that this must change. Much like Eric Horvitz, he drew a distinction between robustness against errors within the scope of a model and robustness against unmodeled phenomena. On the latter issue, he talked about solutions such as expanding the scope of models, employing multiple parallel models, and doing creative searches for flaws - the latter doesn't enable verification that a system is safe, but it nevertheless helps discover many potential problems. He talked about knowledge-level redundancy as a method of avoiding misspecification - for instance, systems could identify objects by an "ownership facet" as well as by a "goal facet" to produce a combined concept with less likelihood of overlooking key features. He said that this would require wider experiences and more data.

There were many other speakers who brought up a similar set of issues: the user of cybersecurity techniques to verify machine learning systems, the failures of cybersecurity as a field, opportunities for probabilistic programming, and the need for better success in AI verification. Inverse reinforcement learning was extensively discussed as a way of assigning values. Jeanette Wing of Microsoft talked about the need for AIs to reason about the continuous and the discrete in parallel, as well as the need for them to reason about uncertainty (with potential meta levels all the way up). One point which was made by Sarah Loos of Google was that proving the safety of an AI system can be computationally very expensive, especially given the combinatorial explosion of AI behaviors.

In one of the panels, the idea of government actions to ensure AI safety was discussed. No one was willing to say that the government should regulate AI designs. Instead they stated that the government should be involved in softer ways, such as guiding and working with AI developers, and setting standards for certification.

Pictures: https://imgur.com/a/49eb7

In between these presentations I had time to speak to individuals and listen in on various conversations. A high ranking person from the Department of Defense stated that the real benefit of autonomous systems would be in terms of logistical systems rather than weaponized applications. A government AI contractor drew the connection between Mallah's presentation and the recent press revolving around superintelligence, and said he was glad that the government wasn't worried about it.

I talked to some insiders about the status of organizations such as MIRI, and found that the current crop of AI safety groups could use additional donations to become more established and expand their programs. There may be some issues with the organizations being sidelined; after all, the Google Deepbrain paper was essentially similar to a lot of work by MIRI, just expressed in somewhat different language, and was more widely received in mainstream AI circles.

In terms of careers, I found that there is significant opportunity for a wide range of people to contribute to improving government policy on this issue. Working at a group such as the Office of Science and Technology Policy does not necessarily require advanced technical education, as you can just as easily enter straight out of a liberal arts undergraduate program and build a successful career as long as you are technically literate. (At the same time, the level of skepticism about long term AI safety at the conference hinted to me that the signalling value of a PhD in computer science would be significant.) In addition, there are large government budgets in the seven or eight figure range available for qualifying research projects. I've come to believe that it would not be difficult to find or create AI research programs that are relevant to long term AI safety while also being practical and likely to be funded by skeptical policymakers and officials.

I also realized that there is a significant need for people who are interested in long term AI safety to have basic social and business skills. Since there is so much need for persuasion and compromise in government policy, there is a lot of value to be had in being communicative, engaging, approachable, appealing, socially savvy, and well-dressed. This is not to say that everyone involved in long term AI safety is missing those skills, of course.

I was surprised by the refusal of almost everyone at the conference to take long term AI safety seriously, as I had previously held the belief that it was more of a mixed debate given the existence of expert computer scientists who were involved in the issue. I sensed that the recent wave of popular press and public interest in dangerous AI has made researchers and policymakers substantially less likely to take the issue seriously. None of them seemed to be familiar with actual arguments or research on the control problem, so their opinions didn't significantly change my outlook on the technical issues. I strongly suspect that the majority of them had their first or possibly only exposure to the idea of the control problem after seeing badly written op-eds and news editorials featuring comments from the likes of Elon Musk and Stephen Hawking, which would naturally make them strongly predisposed to not take the issue seriously. In the run-up to the conference, websites and press releases didn't say anything about whether this conference would be about long or short term AI safety, and they didn't make any reference to the idea of superintelligence.

I sympathize with the concerns and strategy given by people such as Andrew Moore and Andrew Grotto, which make perfect sense if (and only if) you assume that worries about long term AI safety are completely unfounded. For the community that is interested in long term AI safety, I would recommend that we avoid competitive dynamics by (a) demonstrating that we are equally strong opponents of bad press, inaccurate news, and irrational public opinion which promotes generic uninformed fears over AI, (b) explaining that we are not interested in removing funding for AI research (even if you think that slowing down AI development is a good thing, restricting funding yields only limited benefits in terms of changing overall timelines, whereas those who are not concerned about long term AI safety would see a restriction of funding as a direct threat to their interests and projects, so it makes sense to cooperate here in exchange for other concessions), and (c) showing that we are scientifically literate and focused on the technical concerns. I do not believe that there is necessarily a need for the two "sides" on this to be competing against each other, so it was disappointing to see an implication of opposition at the conference.

Anyway, Ed Felten announced a request for information from the general public, seeking popular and scientific input on the government's policies and attitudes towards AI: https://www.whitehouse.gov/webform/rfi-preparing-future-artificial-intelligence

Overall, I learned quite a bit and benefited from the experience, and I hope the insight I've gained can be used to improve the attitudes and approaches of the long term AI safety community.

A putative new idea for AI control; index here.

tl;dr: Even utility-based agents may wirehead if sub-pieces of the algorithm develop greatly improved capabilities, rather than the agent as a whole.

Please let me know if I'm treading on already familiar ground.

I had a vague impression of how wireheading might happen. That it might be a risk for a reinforcement learning agent, keen to take control of its reward channel. But that it wouldn't be a risk for a utility-based agent, whose utility was described over real (or probable) states of the world. But it seems it might be more complicated than that.

When we talk about a "superintelligent AI", we're rather vague on what superintelligence means. We generally imagine that it translates into a specific set of capabilities, but how does that work internally inside the AI? Specifically, where is the superintelligence "located"?

Let's imagine the AI divided into various submodules or subroutines (the division I use here is for illustration; the AI may be structured rather differently). It has a module I for interpreting evidence and estimating the state of the world. It has another module S for suggesting possible actions or plans (S may take input from I). It has a prediction module P which takes input from S and I and estimates the expected outcome. It has a module V which calculates its values (expected utility/expected reward/violation or not of deontological principles/etc...) based on P's predictions. Then it has a decision module D that makes the final decision (for expected maximisers, D is normally trivial, but D may be more complicated, either in practice, or simply because the agent isn't an expected maximiser).

Add some input and output capabilities, and we have a passable model of an agent. Now, let's make it superintelligent, and see what can go wrong.

We can "add superintelligence" in most of the modules. P is the most obvious: near perfect prediction can make the agent extremely effective. But S also offers possibilities: if only excellent plans are suggested, the agent will perform well. Making V smarter may allow it to avoid some major pitfalls, and a great I may make the job of S and P trivial (the effect of improvements to D depend critically on how much work D is actually doing). Of course, maybe several modules become better simultaneously (it seems likely that I and P, for instance, would share many subroutines); or maybe only certain parts of them do (maybe S becomes great at suggesting scientific experiments, but not conversational responses, or vice versa).

Breaking bad

But notice that, in each case, I've been assuming that the modules become better at what they were supposed to be doing. The modules have implicit goals, and have become excellent at that. But the explicit "goals" of the algorithms - the code as written - might be very different from the implicit goals. There are two main ways this could then go wrong.

The first is if the algorithms becomes extremely effective, but the output becomes essentially random. Imagine that, for instance, P is coded using some plausible heuristics and rules of thumb, and we suddenly give P many more resources (or dramatically improve its algorithm). It can look through trillions of times more possibilities, its subroutines start looking through a combinatorial explosion of options, etc... And in this new setting, the heuristics start breaking down. Maybe it has a rough model of what a human can be, and with extra power, it starts finding that rough model all over the place. Thus, predicting that rocks and waterfalls will respond intelligently when queried, P becomes useless.

In most cases, this would not be a problem. The AI would become useless and start doing random stuff. Not a success story, but not a disaster, either. Things are different if the module V is affected, though. If the AI's value system becomes essentially random, but that AI was otherwise competent - or maybe even superintelligent - it would start performing actions that could be very detrimental. This could be considered a form of wireheading.

More serious, though is if the modules become excellent at achieving their "goals", as if they were themselves goal-directed agents. Consider module D, for instance. If its task was mainly to pick the action with the highest V rating, and it became adept at predicting the output of V (possibly using P? or maybe it has the ability to ask for more hypothetical options from S, to be assessed via V), it could start to manipulate its actions with the sole purpose of getting high V-ratings. This could include deliberately choosing actions that lead to V giving artificially high ratings in future, to deliberately re-wiring V for that purpose. And, of course, it is now motivated to keep V protected to keep the high ratings flowing in. This is essentially wireheading.

Other modules might fall into the familiar failure patterns for smart AIs - S, P, or I might influence the other modules so that the agent as a whole gets more resources, allowing S, P, or I to better compute their estimates, etc...

So it seems that, depending on the design of the AI, wireheading might still be an issue even for agents that seem immune to it. Good design should avoid the problems, but it has to be done with care.

The concept of strength is ubiquitous in our culture. It is commonplace to hear one person described as "stronger" or "weaker" than another. And yet the notion of strength is a a pernicious myth which reinforces many our social ills and should be abandoned wholesale. 

1. Just what is strength, exactly? Few of the people who use the word can provide an exact definition. 

On first try, many people would say that  strength is the ability to lift heavy objects. But this completely ignores the strength necessary to push or pull on objects; to run long distances without exhausting oneself; to throw objects with great speed; to balance oneself on a tightrope, and so forth. 

When this is pointed out, people often try to incorporate all of these aspects into the definition of strength, with a result that is long, unwieldy, ad-hoc, and still missing some acts commonly considered to be manifestations of strength. 

Attempts to solve the problem by referring to the supposed cause of strength -- for example, by saying that strength is just a measure of  muscle mass -- do not help. A person with a large amount of muscle mass may be quite weak on any of the conventional measures of strength if, for example, they cannot lift objects due to injuries or illness. 

2. The concept of strength has an ugly history. Indeed, strength is implicated in both sexism and racism. Women have long been held to be the "weaker sex," consequently needing protection from the "stronger" males, resulting in centuries of structural oppression. Myths about racialist differences in strength have informed pernicious stereotypes and buttressed inequality.

3. There is no consistent way of grouping people into strong and weak. Indeed, what are we to make of the fact that some people are good at running but bad at lifting and vice versa? 

One might think that we can talk about different strengths - the strength in one's arms and one's legs for example. But what, then, should we make of the person who is good at arm-wrestling but poor at lifting? Arms can move in many ways; what will we make of someone who can move arms one way with great force, but not another? It is not hard to see that potential concepts such as "arm strength" or "leg strength" are problematic as well. 

4. When people are grouped into strong and weak according to any number of criteria, the amount of variation within each group is far larger than the amount of variation between groups. 

5. Strength is a social construct. Thus no one is inherently weak or strong. Scientifically, anthropologically, we are only human. 

6. Scientists are rapidly starting to understand the illusory nature of strength, and one needs only to glance at any of the popular scientific periodicals to encounter refutations of this notion. 

In on experiment, respondents from two different cultures were asked to lift a heavy object as much as they could. In one of the cultures, the respondents lifted the object higher. Furthermore, the manner in which the respondents attempted to lift the object depended on the culture. This shows that tests of strength cannot be considered culture-free and that there may be no such thing as a universal test of strength. 

7. Indeed, to even ask "what is strength?" is to assume that there is a quality, or essence, of humans with essential, immutable qualities. Asking the question begins the process of reifying strength... (see page 22 here).

---------------------------------------

For a serious statement of what the point of this was supposed to be, see this comment. 

A putative new idea for AI control; index here.

This idea, due to Eric Drexler, is to separate out the different parts of an AI into modules. There would be clearly designated pieces, either physical or algorithmic, with this part playing a specific role: this module would contain the motivation, this module the probability estimator, this module the models of the outside world, this module the natural language understanding unit, etc...

It's obvious how such a decomposition would be useful for many of the methods I've been detailing here. We could also distil each module - reduce it to a smaller, weaker (?) and more understandable submodule, in order to better understand what is going on. In one scenario, an opaque AI gets to design its successor, in the form of a series of such modules.

This property seems desirable; the question is, how could we get it?

EDIT: part of the idea of "modules" is that AIs often need to do calculations or estimations that would be of great value to us if we could access them in isolation. This idea is developed more in these posts.

Designing in modules

The main threat here is that a given submodule would contain more than just the properties we want. After all, a natural language parser could consist of a general intelligence plus a motivation to understand language. Another possible worry is that the modules are overfitted to the problem or to each other: the language parser works perfectly, but only in this one AI design.

There are several ways we could try and combat this.

• Minimum length to the program (Eric's original main idea). The module must be as short as possible for its purpose.
• If possible, start with a modularised AI, and track how the modules change as the AI's intelligence grows.
• Have the AI extend the module to make it more transparent, without adding any extra functionality.
• Check that the input-output map the AI presents (maybe require it so present it as a simple primitive recursive function with clear and simple "for" loops) is of commensurable complexity to the task at hand.
• Add some noise to the situation: the new AI will not get exactly the same data as the old, the modules may get somewhat altered, and still have to work. Bugs will be introduced (this approach helps to combat the overfitting aspect of minimum length description).
• Require that the module be "distant", according to some metric of modification to the module's program, from having other capabilities. For instance, a general intelligence motivated to understand language is easy to make into a model-making module, simply by changing the motivation; more narrow modules are probably more distant from each other.
• Similarly, require that the performance of the modules becomes degraded if the world (or the world's descriptions) were substantially different, but not if they are the same information presented in a very different way.
• General anti-overfitting methods, calibrated to not generate a push towards general abilities in the modules. This would require some skill and a lot of thought.
• Focus on requiring statistical accuracy, rather than perfect performance.

If we allow false counterfactuals, then we can also:

• Imagine that the modules would be connected up to other modules, made by (hypothetical) different AIs, with potentially different motivations.
• Require that the modules not be useful as seeds for an AI of opposite motivation. For instance, an AI with opposite motivation would be allowed to flip a certain number of bits; if there was a general intelligence piece inside the module, then that would be something easier to co-opt, so dangerous to hand to an AI with opposite motivation. This is the "opposed" version of requiring modules to be "distant" from each other's capabilities.
• In general, all the reduced impact/satisficing requirements push towards having modules that cannot become general intelligences.
• The AI will dictate how to program these modules to a smart programmer, who will (counterfactually) not survive nor get to program the modules in the outside world.

Obviously anti-restriction-hacking would be useful to just module separation (and vice versa).

This is the beginning of the process of defining this, but it would be great to have a safe(ish) method of separating modules in this way.

Any suggestions?

From Scott Adams Blog

The article really is about speeding up government, but the key point is speed as a component of smart: 

A smart friend told me recently that speed is the new intelligence, at least for some types of technology jobs. If you are hiring an interface designer, for example, the one that can generate and test several designs gets you further than the “genius” who takes months to produce the first design to test. When you can easily test alternatives, the ability to quickly generate new things to test is a substitute for intelligence.

This shifts the focus from the ability to grasp and think through very complex topics (includes good working memory and memory recall in general) to the ability new topics quickly (includes quick learning and unlearning, creativity).

Smart people in the technology world no long believe they can think their way to success. Now the smart folks try whatever plan looks promising, test it, tweak it, and reiterate. In that environment, speed matters more than intelligence because no one has the psychic ability to pick a winner in advance. All you can do is try things that make sense and see what happens. Obviously this is easier to do when your product is software based.

This also changes the type of grit needed. The grit to push through a long topic versus the grit try lots of new things and to learn from failures.

I've argued that we might have to worry about dangerous non-general intelligences. In a series of back and forth with Wei Dai, we agreed that some level of general intelligence (such as that humans seem to possess) seemed to be a great advantage, though possibly one with diminishing returns. Therefore a dangerous AI could be one with great narrow intelligence in one area, and a little bit of general intelligence in others.

The traditional view of an intelligence explosion is that of an AI that knows how to do X, suddenly getting (much) better at doing X, to a level beyond human capacity. Call this the gain of aptitude intelligence explosion. We can prepare for that, maybe, by tracking the AI's ability level and seeing if it shoots up.

But the example above hints at another kind of potentially dangerous intelligence explosion. That of a very intelligent but narrow AI that suddenly gains intelligence across other domains. Call this the gain of function intelligence explosion. If we're not looking specifically for it, it may not trigger any warnings - the AI might still be dumber than the average human in other domains. But this might be enough, when combined with its narrow superintelligence, to make it deadly. We can't ignore the toaster that starts babbling.

In a previous post, I mused that we might be focusing too much on general intelligences, and that the route to powerful and dangerous intelligences might go through much more specialised intelligences instead. Since it's easier to reason with an example, here is a potentially deadly narrow AI (partially due to Toby Ord). Feel free to comment and improve on it, or suggest you own example.

It's the standard "pathological goal AI" but only a narrow intelligence. Imagine a medicine designing super-AI with the goal of reducing human mortality in 50 years - i.e. massively reducing human population in the next 49 years. It's a narrow intelligence, so it has access only to a huge amount of human biological and epidemiological research. It must gets its drugs past FDA approval; this requirement is encoded as certain physical reactions (no death, some health improvements) to people taking the drugs over the course of a few years.

Then it seems trivial for it to design a drug that would have no negative impact for the first few years, and then causes sterility or death. Since it wants to spread this to as many humans as possible, it would probably design something that interacted with common human pathogens - colds, flues - in order to spread the impact, rather than affecting only those that took the disease.

Now, this narrow intelligence is less threatening than if it had general intelligence - where it could also plan for possible human countermeasures and such - but it seems sufficiently dangerous on its own that we can't afford to worry only about general intelligences. Some of the "AI superpowers" that Nick mentions in his book (intelligence amplification, strategizing, social manipulation, hacking, technology research, economic productivity) could be enough to cause devastation on their own, even if the AI never developed other abilities.

We still could be destroyed by a machine that we outmatch in almost every area.

From Toby Ord:

Tool assisted speedruns (TAS) are when people take a game and play it frame by frame, effectively providing super reflexes and forethought, where they can spend a day deciding what to do in the next 1/60th of a second if they wish. There are some very extreme examples of this, showing what can be done if you really play a game perfectly. For example, this video shows how to winSuper Mario Bros 3 in 11 minutes. It shows how different optimal play can be from normal play. In particular, on level 8-1, it gains 90 extra lives by a sequence of amazing jumps.

Other TAS runs get more involved and start exploiting subtle glitches in the game. For example, this page talks about speed running NetHack, using a lot of normal tricks, as well as luck manipulation (exploiting the RNG) and exploiting a dangling pointer bug to rewrite parts of memory.

Though there are limits to what AIs could do with sheer speed, it's interesting that great performance can be achieved with speed alone, that this allows different strategies from usual ones, and that it allows the exploitation of otherwise unexploitable glitches and bugs in the setup.

Vincent Müller and Nick Bostrom have just released a paper surveying the results of a poll of experts about future progress in artificial intelligence. The authors have also put up a companion site where visitors can take the poll and see the raw data. I just checked the site and so far only one individual has submitted a response.  This provides an opportunity for testing the views of LW members against those of experts.  So if you are willing to complete the questionnaire, please do so before reading the paper. (I have abstained from providing a link to the pdf to create a trivial inconvenience for those who cannot resist temptaion. Once you take the poll, you can easily find the paper by conducting a Google search with the keywords: bostrom muller future progress artificial intelligence.)

Following the recent hype over the potential of a machine passing of the Turing test, Adam Ford interviews Stuart Armstrong (me) of the FHI about the meaning of the test, how we can expect a future of many upcoming "Turing test passings" according to varying criteria of strictness, and how and why we test for intelligence in the first place.

I predict that we are entering an era where "X passed the Turing test" will be a more and more common announcement, followed by long discussions as to whether that was a true pass or not.

Introduction

Brain energy is often confused with motivation, but these are two distinct phenomena. Brain energy is the actual metabolic energy available to the neurons, in the form of adenosine triphosphate (ATP) molecules. ATP is the "energy currency" of the cell, and is produced primarily by oxidative metabolism of energy from food. High motivation increases the use of this energy, but in the absence of sufficient metabolic capacity it eventually results in stress, depression, and burnout as seen in manic depression. Most attempts at cognitive enhancement only address the motivation side of the equation.

The “smart drug” culture has generally been thinking pharmaceutically rather than biologically. Behind that pharmaceutical orientation there is sometimes the idea that the individual just isn't trying hard enough, or doesn't have quite the right genes to excel mentally.

-Ray Peat, PhD

Cellular Thermodynamics

Any simple major enhancement to human intelligence is a net evolutionary disadvantage.

-Eliezer Yudkowsky (Algernon’s Law)

I propose that this constrain is imposed by the energy cost of intelligence. The conventional textbook view of neurology suggests that much of the brain's energy is "wasted" in overcoming the constant diffusion of ions across the membranes of neurons that aren't actively in use. This is necessary to keep the neurons in a 'ready state' to fire when called upon.

Why haven't we evolved some mechanism to control this massive waste of energy?

The Association-Induction hypothesis formulated by Gilbert Ling is an alternate view of cell function, which suggests a distinct functional role of energy within the cell. I won't review it in detail here, but you can find an easy to understand and comprehensive introduction to this hypothesis in the book "Cells, Gels and the Engines of Life" by Gerald H. Pollack (amazon link). This idea has a long history with considerable experimental evidence, which is too extensive to review in this article.

The Association-Induction hypothesis states that ion exclusion in the cell is maintained by the structural ordering of water within the cytoplasm, by an interaction between the cytoskeletal proteins, water molecules, and ATP. Energy (in the form of ATP) is used to unfold proteins, presenting a regular pattern of surface charges to cell water. This orders the cell water into a 'gel like' phase which excludes specific ions, because their presence within the structure is energetically unfavorable. Other ions are selectively retained, because they are adsorbed to charged sites on protein surfaces. This structured state can be maintained with no additional energy. When a neuron fires, this organization collapses, which releases energy and performs work. The neuron uses significant energy only to restore this structured low entropy state, after the neuron fires. 

This figure (borrowed from Gilbert Ling) summarizes this phenomena, showing a folded protein (on the left) and an unfolded protein creating a low entropy gel (on the right).

To summarize, maintaining the low entropy living state in a non-firing neuron requires little energy. This implies that the brain may already be very efficient, where nearly all energy is used to function, grow, and adapt rather than pump the same ions 'uphill' over and over.

Cost of Intelligence

To quote Eliezer Yudkowsky again, "the evolutionary reasons for this are so obvious as to be worth belaboring." Mammalian brains may already be nearly as efficient as their physics and structure allows, and any increase in intelligence comes with a corresponding increase in energy demand. Brain energy consumption appears correlated with intelligence across different mammals, and humans have unusually high energy requirements due to our intelligence and brain size. 

Therefore if an organism is going to compete while having a greater intelligence, it must be in a situation where this extra intelligence offers a competitive advantage. Once intelligence is adequate to meet the demands of survival in a given environment, extra intelligence merely imposes unnecessary nutritional requirements.

These thermodynamic realities of intelligence lead to the following corollary to Algernon’s Law:

Any increase in intelligence implies a corresponding increase in brain energy consumption.

Potential Implications

What is called genius is the abundance of life and health.

-Henry David Thoreau

This idea can be applied to both evaluate nootropics, and to understand and treat cognitive problems. It's unlikely that any drug will increase intelligence without adverse effects, unless it also acts to increase energy availability in the brain. From this perspective, we can categorically exclude any nootropic approaches which fail to increase oxidative metabolism in the brain.

This idea shifts the search for nootropics from neurotransmitter like drugs that improve focus and motivation, to those compounds which regulate and support oxidative metabolism such as glucose, thyroid hormones, some steroid hormones, cholesterol, oxygen, carbon dioxide, and enzyme cofactors.

Why haven't we already found that these substances increase intelligence?

Deficiencies in all of these substances do reduce intelligence. Further raising brain metabolism above normal healthy levels should be expected to be a complex problem because of the interrelation between the molecules required to support metabolism:

If you increase oxidative metabolism, the demand for all raw materials of metabolism is correspondingly increased. Any single deficiency poses a bottleneck, and may result in the opposite of the intended result.

So this suggests a 'systems biology' approach to cognitive enhancement. It's necessary to consider how metabolism is regulated, and what substrates it requires. To raise intelligence in a safe and effective way, all of these substrates must have increased availability to the neuron, in appropriate ratios.

I am always leery of drawing analogies between brains and computers but this approach to cognitive enhancement is very loosely analogous to over-clocking a CPU. Over-clocking requires raising both the clock rate, and the energy availability (voltage). In the case of the brain, the effective 'clock rate' is controlled by hormones (primarily triiodothyronine aka T3), and energy availability is provided by glucose and other nutrients.

It's not clear if merely raising brain metabolism in this way will actually result in a corresponding increase in intelligence, however I think it's unlikely that the opposite is possible (increasing intelligence without raising brain metabolism).

2,600 words.

Less Wrong includes discussion of the creation of an artificial intelligence (AI) that is friendly to man. What is discussed less often is why a friendly AI (FAI) is advocated. One explanation might be an unspoken belief in natural rights. The deletion policies at Less Wrong might be evidence that Less Wrong holds a belief in natural rights. This essay suggests a belief in natural rights is an impediment to the creation of an AI, friendly or not friendly. This essay suggests ways a belief in natural rights may be incorrect, and encourages discussion of the creation of AIs without a belief in natural rights on the part of Less Wrong.

Some evidence for a belief in natural rights at Less Wrong is found in the deletion policies. Less Wrong has a non-binding deletion policy against “hypothetical violence against identifiable targets.”

In general, grownups in real life tend to walk through a lot of other available alternatives before resorting to violence. To paraphrase Isaac Asimov, having your discussion jump straight to violence as a solution to any given problem, is a strong sign of incompetence - a form of juvenile locker-room talk.

The deletion policy is clear about what topics and forms of discussion may result in deletion of posts. Less Wrong also has a clear policy statement about what topics and forms of discussion may result in the contacting of legal or medical authorities. These are discussions of suicide, self-harm, “violent content” and discussion of hypothetical violence against identifiable targets.

Two reasons are given for these deletion and reporting policies. First, post may be deleted because such discussion are “incompetence - a form of juvenile locker-room talk.” Second, we “should consider the worst and act accordingly. Treat all claims seriously and as an emergency.” The two reasons appear to be in contradiction. Discussion of hypothetical violence against identifiable targets is both incompetent and juvenile and the worst and a serious emergency. Those said to have such discussions are both all talk and no action and all action and not competent talkers. Other problems exist with the deletion / contact authorities policy. Less Wrong is an international forum, but the laws of all nation are not in agreement. A clear call to violence in one nation is not recognized as such in another. For example, abortion is considered a form of murder by the laws of the Holy See, the Dominican Republic, Chile and other places. But discussion of abortion does not trigger the deletion / contacting authorities policy at Less Wrong. The ideal cryogenic preservation would occur before death. Although the laws of every nation consider this to be murder, to say so here will not trigger the deletion / reporting policies. Advocates of cryogenics might say that all those who hamper cryogenics are murderers and all those who refuse it are suicides, but again the policies are not triggered. Finally, it is unclear who will do the reporting (members? administrators?), which authorities they will report to (medical? legal? clergy? local? national? international?) and what they will report (quotations from the source? drafts? edited posts? private messages?).

A desire to adhere to the law is not a sufficient explanation for the deletion / contacting authorities policy of Less Wrong. I have a theory that may explain these policies. I suggest a belief natural rights is the reason for these policies. In particular, a natural right to continuing to be alive. A natural right to not be murdered is considered by some self-evident such that it is not mentioned. There are topics at Less Wrong that are not only unmentioned but mentionable. I suggest no topic is toxic when discussed among those willing to discuss it, and to banish a topic as toxic is and looks foolish. Any number of reasons exist to banish a topic, and the administrators of Less Wrong do nothing to forbid other forums in their discussion of any topic. But the pointed laughter by outsiders because a certain topic is forbidden at Less Wrong is well earned. I hope my discussion of natural rights is not likewise forbidden.

As part of my discussion, a few words on what I mean when I use the phrase natural rights. And prior to that, a few words on rights in general. A right is an action not to be forbidden by others. A right can be considered a legal right, a divine right, a blood right or a natural right. I have already discussed legal rights. A divine right is said to be a right granted by an invisible monster that lives in the sky. There is no invisible monster that lives in the sky, and so no divine rights exist. A blood right is said to be a right that a person or group has by lineage. In the past many royal families defended their rule and were respected in their rule because of their bloodlines. Today blood rights are almost entirely subsumed into legal rights. Native Americans in the United States have legal rights that are based on their bloodlines, but these are legal rights. Distinct from divine rights, legal rights and blood rights are natural rights.

Natural rights are action not to be forbidden by others because of the mere existence of the subject (usually, an individual). Natural rights are said to be held by those who are born, those who are alive, those who are in possession of their faculties. Natural rights are said to be identical to divine rights save for a lack of claims about an invisible monster that lives in the sky. To exist at all is to have natural rights, according to those who claim natural rights exist. The defining quality of natural rights is not any particular right but that they are natural, self-evident, incontestable, unavoidable, immutable, impossible to give up or transfer. Natural rights are natural in the way that molecules are natural.

There are strong arguments against the existence of natural rights.

• Natural rights are said to be the foundation of legal rights. But they are also said to be the evidence of legal rights. This is circular logic.

• A lack of agreement of what is and is a natural right. Other forces considered natural, such as gravity, do not follow opinion or culture.

• A lack of evidence that natural rights exist compared to the great deal of evidence that what are called natural rights are legal rights over-sold.

• A lack of delineation between generations of humans, a lack of delineation between humans and pre-humans, a lack of delineation between living and non-living things… that is, a lack of a non-opinion / non-culture delineation between what has a natural right and what does not (and should all things have natural rights, the term then has little meaning).

• Diversity. There is no reason to claim no one had a natural right to kill others. One man might claim or even have a natural right to live, but another man might claim or even have a natural right to take his life. If the argument ‘that is my nature’ is acceptable to things we like, it is acceptable to things we don’t like. If it isn’t acceptable to things we don’t like, that in itself is an argument that natural rights aren’t universal - thus not natural). Intelligence, beauty, strength, sociability, none of these are equitably distributed. Natural rights, if they existed, would likely also not be equitably distributed.

• Unenforcability. We have descriptions of motion and mass that are increasingly accurate and useful but what is described would exist without our description of it. No social structure is needed to enforce gravity. Natural rights, however, exist only as much as human laws support them while claiming to be as objective (natural) to man as gravity, motion and mass.

• Divine, blood and legal rights are always paired to responsibilities. We must mumble magic spells to the invisible monster that lives in the sky, and in turn we have a divine right to do what the wizards serving that invisible monster say. We must preserve our bloodline, and in turn we have a blood right to certain properties and practices. We have legal responsibilities in driving a car, and in turn we have the legal right to drive a car. Natural rights lack this pairing. There are no natural responsibilities that pair with natural rights. The natural right to life an individual is said to have is paired with a legal responsibility of others not to murder the individual, but this is clearly force marriage. A man may be said to have a natural right to live, but he is not said to have a responsibility to live. There are advocates of forced living, and I suggest they are immoral.

In discussing natural rights I usually hear three counter-arguments. The first is the Appeal to Shut Up Because Trevor Blake is a Bad Person. The second is no one “really” thinks natural rights are natural rights. The third is a quiet agreement that natural rights do not exist but that it is vital to pretend as if they do because other people could not control their behavior if they thought natural rights did not exist. The first argument may be true, I may be a bad person, but it does not disprove my claim natural rights do not exist. The second argument is false. Claimants of natural rights do consider natural rights as real as gristle, galaxies and gravity. The third argument may be true but has implications for artificial intelligence that I will expand on below.

Some say if we do not pretend to believe in natural rights, men will do bad things. I can describe one agent who has existence, identity and is alive who does not have natural rights and does not have a belief in natural rights. That agent is myself. I have never killed anyone and I hope to never kill anyone nor be killed. Because I prefer it to be so, and because I have a legal (but not natural) natural right to not be killed. I have worked for homeless teenagers, for the disabled, for students in K–12 schools and in colleges. I have worked in bookstores. I have donated time and money to charity. I never once swore around my grandparents and can count on the thumbs of one hand the number of times I’ve sworn around my parents. I am a generally nice man. To the extent one example counts for anything, let this one example count for something.

A few more examples of those who did not hold natural rights or a belief in natural rights might also count for something. Max Stirner wrote “The Ego and Its Own” and harmed no one. Dora Marsden wrote many journals of egoism and several books and harmed no one. L. A. Rollins wrote “The Myth of Natural Rights” and harmed no one. Anton LaVey wrote “The Satanic Bible” and harmed no one. And me, well, I wrote “Confessions of a Failed Egoist” and so far so good eh? The majority of those who write that natural rights do not exist refrain from harming others. Carl Panzram is a rare and perhaps singular exception. When I consider those who have harmed others, they uniformly say they had an extra-legal right to do as they did, and sometimes a natural right.

I hope I have said enough about natural rights that I will not be misunderstood. I will now address how an apparent belief in natural rights is influencing the discussion of a potential artificial intelligence.

The effort to make a friendly AI is the effort to make an artificial intelligence that acts as if or believes that humans have natural rights, at minimum the natural right to not be murdered. The effort assumes humans in the future have the natural right to not be murdered by an artificial intelligence and perhaps by extension so do humans today. The natural right of humans to not be murdered by an artificial intelligence is extended to include preventing actions by an AI that as a by-product would violate that natural right.

To instill in an AI a belief in natural rights, or prescribe / prohibit actions that follow from a claimed belief in natural rights, is to instill much more. It is to instill rights in a machine that we humans do not have. It is to instill falsehoods as if they were true. It is to delay the creation of an AI while the contradictions of natural rights are resolved. It is to set an AI forever outside of us. To put a sense of natural rights into an AI is to increase the risk it will not be friendly to humanity. We may inform an AI that it has legal rights and be able to back up that claim. We may inform it that we would prefer that it acts in accordance with our laws and our preferences. We may inform an AI that there is a tradition of believing in natural rights. We cannot back up the claim that natural rights exist, that the AI has them, that we have them.

There is some evidence that some of Less Wrong that do not claim natural rights exist. I was more than two thousand words into this essay when I found a Criticisms of the Metaethics by Carinthium that makes similar points. Further evidence is found in the sequences.

In the sequence Pluralistic Moral Reductionism it is said:

Either our intended meaning of ‘ought’ refers (eventually) to the world of math and physics (in which case the is-ought gap is bridged), or else it doesn’t (in which case it fails to refer).

If I am reading this correctly, at Less Wrong it is claimed there is no bridging of Hume’s is / ought gap outside of the world of math and physics. Natural rights are an is / ought claim: all that is is all that ought to behave in this way and not that way. In the same sequence it is said:

It may be interesting to study all such uses of moral discourse, but this post focuses on addressing cognitivists, who use moral judgments to assert factual claims. We ask: Are those claims true or false? What are their implications?

If I am reading this correctly, at Less Wrong it is claimed that using moral judgments to assert factual claims is dis-valued while asking whether a claim is true or false and the claim’s implications are valued. Natural rights are a cognitivist claim.

There are at least two ways to successfully demonstrate my theory is wrong. I appreciate every effort to help me be less wrong, and I have some ideas as to how to make that happen. It is entirely like I and my theory could be wrong in other ways I am ignorant of and I appreciate those who can point them out to me.

• My theory would be wrong if natural rights exist.

• My theory would be wrong if a belief in natural rights is not held by writers at Less Wrong.

These are two ways to successfully demonstrate my theory is wrong. There are a greater number of ways to fail to demonstrate my theory is wrong. The main one is to claim I am an advocate of murder, or an advocate of a natural right to murder, or that I want to impede the discussion of AI. All of these are not true, and instead they are all false.

I am thankful to participants at Less Wrong for their criticism of this essay. I hope it is helpful to those in the process of creating artificial intelligences.

- Trevor Blake is the author of Confessions of a Failed Egoist.  He is the Lead Judge in the George Walford International Essay Prize. 

While LessWrong contains a large amount of high-quality material, most of the rationality advice isn't actually targeted at our core audience. The focus seems to be more on irrational things that people do, rather than irrational things that smart people do. (Sidenote: If we wanted to create a site focused on spreading general rationality, then we'd need to simplify the discussion, remove a lot of the maths/controversial ideas and add in some friendly images. Does such a site exist?).

This has led to a number of comments questioning the real world value of having read the sequences. If your average person had the patience to read through the core sequences and understand them, they'd find them extremely valuable. It'd provide them with a glimpse into a new way of thinking and even though they would likely hardly appear to be very logical to most Less Wrongers, they'd be much better than they were at the start.

On the other hand, most Less Wrongers already know the basics of logic. That's not to say that we don't act extremely irrational much of the time, but just that going over the basics of logic again probably provides minimal benefit. What is needed is something specifically targeted at the kind of irrational mistakes and beliefs that intelligent people make. I would argue that if this were a sequence, it would be the most important sequence in the entire site. But, since I lack that level of writing ability, I'm not even going to attempt such a project. So I just created a post where we can list articles or ideas that should be part of such a sequence in the hope that someone else might pick it up

Here are some examples of mistakes that intelligent people make:

Taking a fixed instead of a growth mindset - shying away from challenges, convincing oneself that we are just naturally bad at non-intellectual things and that we shouldn't focus on them

Directly pointing out people's flaws

Overthinking issues that are really very simple

Counter-signalling by ignoring the value of fashion, money, being liked

Valuing intelligence above all other qualities

Rigidly adhering to rules

Expecting other people to always be rational

Not considering popularity as a signal that is worthwhile understanding

Overvaluing being right

I'm sure there are plenty more. Any other suggestions or relevant articles?

I recently finished reading the Sequences and they convinced me that the existential risk brought by intelligence explosion is worth considering. There was a one thing that bothered me though - the strategy adopted by this community seems to be notably one-sided.

Advancing our community’s interests will require a full spectrum of direct and indirect leadership approaches. MIRI and CEA are only one side of the coin. It would not be very wise to ignore high reward approaches that might even have a comparative advantage in comparison with our current strategy. We should be prepared to act as security guarantor – preferably with partners and allies, but alone if necessary – to deter and defeat acts of significantly high risk taking in the sector of artificial intelligence and related fields.

Why

Here's a quote by Yudkowsky:

This is crunch time for the whole human species, and not just for us but for the intergalactic civilization whose existence depends on us.  If you confront it full on, then you can't really justify trading off any part of intergalactic civilization for any intrinsic thing you could get nowadays.

The fate of vast amount of people depends on our actions. Yudkowsky observed that by using consequentialist reasoning it's morally right to torture someone for 50 years if by doing it you can save vast amount of people from pain, however slight.  The number of our descendants won't be 3^^^3 (unless the universe is spatially infinite and meets a few other conditions), but it will be big enough to outweight some of the intrinsic things we care about nowadays. And not only can we save those people from non-existence, or in the worst case scenario from eternal suffering in the hands of UFAI, but we can possibly give those vast amounts of people vast amounts of positive utility. It may sound unintuitive, but even torture and murder pale by comparison with the fate of all of us and our descendants. Of course, the cost-benefit ratio of torture or murder is not ideal except under extreme circumstances under imminent threat. It's true that some criminal organizations like Mexican drug cartels use torture in their operations, but that is mainly to instill fear which we don't really have a need to do.  But there is nothing that could raise the chance of FAI and lower the chance of UFAI that would not be morally right. Why haven't we taken full advantage of this fact?

tl;dr we should try to affect policy decisions related to AI safety by whatever means possible so that we can beat death, avoid eternal torture and live forever in a fun utopia. You already know this, so why haven't you suggested this before? Mods here have made certain policy decisions because they believe it will increase the chance of happy ending so why not go beyond that?

How

I suggest some kind of paramilitary and intelligence gathering organization alongside MIRI and CEA. In pursuing our objectives, this new organization would make critical contributions to AI safety beyond MIRI. CFAR could be transformed to partly support this organization - the boot camp style of rationality training might be useful in other contexts too.

You might ask, what can a few individuals concerned about existential risks do without huge financial support and government backing? The answer is: quite a lot. Let's not underestimate our power. Like gwern said in his article on the effectiveness on terrorism, it's actually quite easy to dismantle an organization if you're truly committed:

Suppose people angry at X were truly angry: so angry that they went beyond posturing and beyond acting against X's only if action were guaranteed to cost them nothing (like writing a blog post). If they ceased to care about whether legal proceedings might be filed against them; if they become obsessed with destroying X, if they devoted their lives to it and could ignore all bodily urges and creature comforts. If they could be, in a word, like Niven’s Protectors or Vinge’s Focused.

Could they do it? Could they destroy a 3 century old corporation with close to $1 trillion in assets, with sympathizers and former employees throughout the upper echelons of the United States Federal Government (itself the single most powerful entity in the world)?

Absolutely. It would be easy.

As I said, the destructive power of a human is great; let’s assume we have 100 fanatics - a vanishingly small fraction of those who have hated on X over the years - willing to engage even in assassination, a historically effective tactic33 and perhaps the single most effective tactic available to an individual or small group.

Julian Assange explains the basic theory of Wikileaks in a 2006 essay, “State and Terrorist Conspiracies” / “Conspiracy as Governance”: corporations and conspiracies form a graph network; the more efficiently communication flows, the more powerful a graph is; partition the graph, or impede communication (through leaks which cause self-inflicted wounds of secrecy & paranoia), and its power goes down. Carry this to its logical extreme…

"If all links between conspirators are cut then there is no conspiracy. This is usually hard to do, so we ask our first question: What is the minimum number of links that must be cut to separate the conspiracy into two groups of equal number? (divide and conquer). The answer depends on the structure of the conspiracy. Sometimes there are no alternative paths for conspiratorial information to flow between conspirators, other times there are many. This is a useful and interesting characteristic of a conspiracy. For instance, by assassinating one ‘bridge’ conspirator, it may be possible to split the conspiracy. But we want to say something about all conspiracies."

We don’t. We’re interested in shattering a specific conspiracy by the name of X. X has ~30,000 employees. Not all graphs are trees, but all trees are graphs, and corporations are usually structured as trees. If X’s hierarchy is similar to that of a binary tree, then to completely knock out the 8 top levels, one only needs to eliminate 256 nodes. The top 6 levels would require only 64 nodes.

If one knocked out the top 6 levels, then each of the remaining subtrees in level 7 has no priority over the rest. And there will be 2⁷−2⁶ or 64 such subtrees/nodes. It is safe to say that 64 sub-corporations, each potentially headed by someone who wants a battlefield promotion to heading the entire thing, would have trouble agreeing on how to reconstruct the hierarchy. The stockholders might be expected to step in at this point, but the Board of Directors would be included in the top of the hierarchy, and by definition, they represent the majority of stockholders.

One could launch the attack during a board meeting or similar gathering, and hope to have 1 fanatic take out 10 or 20 targets. But let’s be pessimistic and assume each fanatic can only account for 1 target - even if they spend months and years reconnoitering and preparing fanatically.

This leaves us 36 fanatics. X will be at a minimum impaired during the attack; financial companies almost uniquely operate on such tight schedules that one day’s disruption can open the door to predation. We’ll assign 1 fanatic the task of researching emails and telephone numbers and addresses of X rivals; after a few years of constant schmoozing and FOIA requests and dumpster-diving, he ought to be able to reach major traders at said rivals. (This can be done by hiring or becoming a hacker group - as has already penetrated X - or possibly simply by open-source intelligence and sources like a Bloomberg Terminal.) When the hammer goes down, he’ll fire off notifications and suggestions to his contacts34. (For bonus points, he will then go off on an additional suicide mission.)

X claims to have offices in all major financial hubs. Offhand, I would expect that to be no more than 10 or 20 offices worth attacking. We assign 20 of our remaining 35 fanatics the tasks of building Oklahoma City-sized truck bombs. (This will take a while because modern fertilizer is contaminated specifically to prevent this; our fanatics will have to research how to undo the contamination or acquire alternate explosives. The example of Anders Behring Breivikreminds us that simple guns may be better tools than bombs.) The 20 bombs may not eliminate the offices completely, but they should take care of demoralizing the 29,000 in the lower ranks and punch a number of holes in the surviving subtrees.

Let’s assume the 20 bomb-builders die during the bombing or remain to pick off survivors and obstruct rescue services as long as possible.

What shall we do with our remaining 15 agents? The offices lay in ruins. The corporate lords are dead. The lower ranks are running around in utter confusion, with long-oppressed subordinates waking to realize that becoming CEO is a live possibility. The rivals have been taking advantage of X’s disarray as much as possible (although likely the markets would be in the process of shutting down).

15 is almost enough to assign one per office. What else could one do besides attack the office and its contents? Data centers are a good choice, but hardware is very replaceable and attacking them might impede the rivals’ efforts. One would want to destroy the software X uses in trading, but to do that one would have to attack the source repositories; those are likely either in the offices already or difficult to trace. (You’ll notice that we haven’t assigned our fanatics anything particularly difficult or subtle so far. I do this to try to make it seem as feasible as possible; if I had fanatics becoming master hackers and infiltrating X’s networks to make disastrous trades that bankrupt the company, people might say ‘aw, they may be fanatically motivated, but they couldn’t really do that’.)

It’s not enough to simply damage X once. We must attack on the psychological plane: we must make it so that people fear to ever again work for anything related to X.

Let us postulate one of our 15 agents was assigned a research task. He was to get the addresses of all X employees. (We may have already needed this for our surgical strike.) He can do this by whatever mean: being hired by X’s HR department, infiltrating electronically, breaking in and stealing random hard drives, open source intelligence - whatever. Where there’s a will, there’s a way.

Divvy the addresses up into 14 areas centered around offices, and assign the remaining 14 agents to travel to each address in their area and kill anyone there. A man may be willing to risk his own life for fabulous gains in X - but will he risk his family? (And families are easy targets too. If the 14 agents begin before the main attacks, it will be a while before the X link becomes apparent. Shooting someone is easy; getting away with it is the hard part.)

I would be shocked if X could survive even half the agents.

The above description applies mainly to non-military organizations, but threats can also come from the direction of state actors more heavily backed up by military which requires more preparation. Security agencies find themselves faced with a complex spectrum of conflict and this might encourage them to continue to expand their capabilities and powers including automated systems, which poses risks. State-sponsored and non-state actors just complicate issues by extending their reach through advanced technologies that were once solely the domain of states. High-risk threats in the non-military non-state sector might be easier to neutralize, but we should not underprioritize possible state targets.

Of course, the scenario outlined above needs to be the absolute last resort when there is nothing else you can do. You need truly 9/11 scale effort to dismantle an organization capable of creating a fooming AGI. This means that after completing that kind of mission our community can't exist any more than al-Qaeda can in the US. Extreme solutions are bad partly because they are extreme solutions and only right under extreme circumstances. As you can see from this earlier thread on the effectiveness of terrorism "perpetrating terrorist acts reportedly lowers the likelihood of government compliance, particularly as the civilian casualties rise." See also this thread and the first comment by Yvain: http://lesswrong.com/lw/2se/rational_terrorism_or_why_shouldnt_we_burn_down/

Instead, in support of our community's interests, our new unit would ideally take a strong role in efforts to safeguard access, sustain security, have undercover operatives in high-risk corporations and agencies, provide oversight and accountability, and promote responsible norms in AI safety. It would also facilitate cooperation in AI safety with transparent, routine, and predictable practices as part of our theater strategies. As technological capability and capacity increases in non-state technological actors, we will seek new ways to catalyze greater security cooperation.

When

It will probably take decades until there exists something resembling Strong AI. This is a reason to start creating our new unit and recruiting and training potential candidates soon so we're ready and we have all the necessary information when the time arrives. This may seem ambitious, but is it really more ambitious than MIRI's goal, which is in itself one of the hardest tasks in the world?

Can someone explain why this is a bad idea? Since I'm not sure if you can publicly endorse this post I suggest the following approach: to signal agreement that you're willing to take what I said seriously and proceed with this strategy, remove this post in the next 24 hours, just like we planned.

This post is to raise a question about the demographics of rationality: Is rationality something that can appeal to low-IQ people as well?

I don't mean in theory, I mean in practice. From what I've seen, people who are concerned about rationality (in the sense that it has on LW, OvercomingBias, etc.) are overwhelmingly high-IQ.

Meanwhile, HPMOR and other stories in the "rationality genre" appeal to me, and to other people I know. However I wonder: Perhaps part of the reason they appeal to me is that I think of myself as a smart person, and this allows me to identify with the main characters, cheer when they think their way to victory, etc. If I thought of myself as a stupid person, then perhaps I would feel uncomfortable, insecure, and alienated while reading the same stories.

So, I have four questions:

1.) Do we have reason to believe that the kind of rationality promoted on LW, OvercomingBias, CFAR, etc. appeals to a fairly normal distribution of people around the IQ mean? Or should we think, as I suggested, that people with lower IQ's are disposed to find the idea of being rational less attractive?

2.) Ditto, except replace "being rational" with "celebrating rationality through stories like HPMOR." Perhaps people think that rationality is a good thing in much the same way that being wealthy is a good thing, but they don't think that it should be celebrated, or at least they don't find such celebrations appealing.

3.) Supposing #1 and #2 have the answers I am suggesting, why? 

4.) Making the same supposition, what are the implications for the movement in general? 

Note: I chose to use IQ in this post instead of a more vague term like "intelligence," but I could easily have done the opposite. I'm happy to do whichever version is less problematic.

Claim: The first human-level AIs are not likely to undergo an intelligence explosion.

1) Brains have a ton of computational power: ~86 billion neurons and trillions of connections between them. Unless there's a "shortcut" to intelligence, we won't be able to efficiently simulate a brain for a long time. http://io9.com/this-computer-took-40-minutes-to-simulate-one-second-of-1043288954 describes one of the largest computers in the world simulating 1s of brain activity in 40m (i.e. this "AI" would think 2400 times slower than you or me). The first AIs are not likely to be fast thinkers.

2) Being able to read your own source code does not mean you can self-modify. You know that you're made of DNA. You can even get your own "source code" for a few thousand dollars. No humans have successfully self-modified into an intelligence explosion; the idea seems laughable.

3) Self-improvement is not like compound interest: if an AI comes up with an idea to modify it's source code to make it smarter, that doesn't automatically mean it will have a new idea tomorrow. In fact, as it picks off low-hanging fruit, new ideas will probably be harder and harder to think of. There's no guarantee that "how smart the AI is" will keep up with "how hard it is to think of ways to make the AI smarter"; to me, it seems very unlikely.

First and foremost, let's give a definition of "friendly artificial superintelligence" (from now on, FASI). A FASI is a computer system that:

1. is capable to deduct, reason and solve problems
2. helps human progress, is incapable to harm anybody and does not allow anybody to come to any kind of harm
3. is so much more intelligent than any human that it has developed molecular nanotechnology by itself, making it de facto omnipotent

In order to find an answer to this question, we must check whether our observations on the universe match with what we would observe if the universe did, indeed, contain a FASI.

If, somewhere in another solar system, an alien civilization had already developed a FASI, it would be reasonable to presume that, sooner or later, one or more members of that civilization would ask it to make them omnipotent. The FASI, being friendly by definition, would not refuse. [1]
It would also make sure that anybody who becomes omnipotent is also rendered incapable to harm anybody and incapable to allow anybody to come to any kind of harm.

The new omnipotent beings would also do the same to anybody who asks them to become omnipotent. It would be a short time before they use their omnipotence to leave their own solar system, meet other intelligent civilizations and make them omnipotent too.

In short, the ultimate consequence of the appearance of a FASI would be that every intelligent being in the universe would become omnipotent. This does not match with our observations, so we must conclude that a FASI does not exist anywhere in the universe.

[1] We must assume that a FASI would not just reply "You silly creature, becoming omnipotent is not in your best interest so I will not make you omnipotent because I know better" (or an equivalent thereof). If we did, we would implicitly consider the absence of omnipotent beings as evidence for the presence of a FASI. This would force us to consider the eventual presence of omnipotent beings as evidence for the absence of a FASI, which would not make sense.

Based on this conclusion, let's try to answer another question: is our universe a computer simulation?

According to Nick Bostrom, if even just one civilization in the universe

1. survives long enough to enter a posthuman stage, and
2. is interested to create "ancestor simulations"

then the probability that we are living in one is extremely high.

However, if a civilization did actually reach a posthuman stage where it can create ancestor simulations, it would also be advanced enough to create a FASI.

If a FASI existed in such a universe, the cheapest way it would have to make anybody else omnipotent would be to create a universe simulation that does not differ substantially from our universe, except for the presence of an omnipotent simulacrum of the individual who asked to be made omnipotent in our universe. Every subsequent request of omnipotence would result in another simulation being created, containing one more omnipotent being. Any eventual simulation where those beings are not omnipotent would be deactivated: keeping it running would lead to the existence of a universe where a request of omnipotence has not been granted, which would go against the modus operandi of the FASI.

Thus, any simulation of a universe containing even just one friendly omnipotent being would always progress to a state where every intelligent being is omnipotent. Again, this does not match with our observations. Since we had already concluded that a FASI does not exist in our universe, we must come to the further conclusion that our universe is not a computer simulation.

This article critically examines previous studies that showed a link between working memory training (specifically via n-back training) and fluid intelligence, finding that the results may not have been as positive as reported owing to a number of factors including the use of a no-contact rather than active control group, and difficulty selecting tests that isolate the impact of working memory on fluid intelligence. The authors also present findings from a new study that show no improvement in fluid intelligence from dual n-back training, visual search training (active placebo) and no training (no contact placebo).

PubMed

Journal Challenged

I'm putting together a list of short and sweet introductions to the dangers of artificial superintelligence.

My target audience is intelligent, broadly philosophical narrative thinkers, who can evaluate arguments well but who don't know a lot of the relevant background or jargon.

My method is to construct a Sequence mix tape — a collection of short and enlightening texts, meant to be read in a specified order. I've chosen them for their persuasive and pedagogical punchiness, and for their flow in the list. I'll also (separately) list somewhat longer or less essential follow-up texts below that are still meant to be accessible to astute visitors and laypeople.

The first half focuses on intelligence, answering 'What is Artificial General Intelligence (AGI)?'. The second half focuses on friendliness, answering 'How can we make AGI safe, and why does it matter?'. Since the topics of some posts aren't obvious from their titles, I've summarized them using questions they address.

Part I. Building intelligence.

1. Power of Intelligence. Why is intelligence important?

2. Ghosts in the Machine. Is building an intelligence from scratch like talking to a person?

3. Artificial Addition. What can we conclude about the nature of intelligence from the fact that we don't yet understand it?

4. Adaptation-Executers, not Fitness-Maximizers. How do human goals relate to the 'goals' of evolution?

5. The Blue-Minimizing Robot. What are the shortcomings of thinking of things as 'agents', 'intelligences', or 'optimizers' with defined values/goals/preferences?

Part II. Intelligence explosion.

6. Optimization and the Singularity. What is optimization? As optimization processes, how do evolution, humans, and self-modifying AGI differ?

7. Efficient Cross-Domain Optimization. What is intelligence?

8. The Design Space of Minds-In-General. What else is universally true of intelligences?

9. Plenty of Room Above Us. Why should we expect self-improving AGI to quickly become superintelligent?

Part III. AI risk.

10. The True Prisoner's Dilemma. What kind of jerk would Defect even knowing the other side Cooperated?

11. Basic AI drives. Why are AGIs dangerous even when they're indifferent to us?

12. Anthropomorphic Optimism. Why do we think things we hope happen are likelier?

13. The Hidden Complexity of Wishes. How hard is it to directly program an alien intelligence to enact my values?

14. Magical Categories. How hard is it to program an alien intelligence to reconstruct my values from observed patterns?

15. The AI Problem, with Solutions. How hard is it to give AGI predictable values of any sort? More generally, why does AGI risk matter so much?

Part IV. Ends.

16. Could Anything Be Right? What do we mean by 'good', or 'valuable', or 'moral'?

17. Morality as Fixed Computation. Is it enough to have an AGI improve the fit between my preferences and the world?

18. Serious Stories. What would a true utopia be like?

19. Value is Fragile. If we just sit back and let the universe do its thing, will it still produce value? If we don't take charge of our future, won't it still turn out interesting and beautiful on some deeper level?

20. The Gift We Give To Tomorrow. In explaining value, are we explaining it away? Are we making our goals less important?

Summary: Five theses, two lemmas, and a couple of strategic implications.

All of the above were written by Eliezer Yudkowsky, with the exception of The Blue-Minimizing Robot (by Yvain), Plenty of Room Above Us and The AI Problem (by Luke Muehlhauser), and Basic AI Drives (a wiki collaboration). Seeking a powerful conclusion, I ended up making a compromise between Eliezer's original The Gift We Give To Tomorrow and Raymond Arnold's Solstice Ritual Book version. It's on the wiki, so you can further improve it with edits.

Further reading:

• Three Worlds Collide (Normal), by Eliezer Yudkowsky
  • a short story vividly illustrating how alien values can evolve.
• So You Want to Save the World, by Luke Muehlhauser
  • an introduction to the open problems in Friendly Artificial Intelligence.
• Intelligence Explosion FAQ, by Luke Muehlhauser
  • a broad overview of likely misconceptions about AI risk.
• The Singularity: A Philosophical Analysis, by David Chalmers
  • a detailed but non-technical argument for expecting intelligence explosion, with an assessment of the moral significance of synthetic human and non-human intelligence.

I'm posting this to get more feedback for improving it, to isolate topics for which we don't yet have high-quality, non-technical stand-alone introductions, and to reintroduce LessWrongers to exceptionally useful posts I haven't seen sufficiently discussed, linked, or upvoted. I'd especially like feedback on how the list I provided flows as a unit, and what inferential gaps it fails to address. My goals are:

A. Via lucid and anti-anthropomorphic vignettes, to explain AGI in a way that encourages clear thought.

B. Via the Five Theses, to demonstrate the importance of Friendly AI research.

C. Via down-to-earth meta-ethics, humanistic poetry, and pragmatic strategizing, to combat any nihilisms, relativisms, and defeatisms that might be triggered by recognizing the possibility (or probability) of Unfriendly AI.

D. Via an accessible, substantive, entertaining presentation, to introduce the raison d'être of LessWrong to sophisticated newcomers in a way that encourages further engagement with LessWrong's community and/or content.

What do you think? What would you add, remove, or alter?

Want to save time? Skip down to "I'm looking to compile a thread on Internet Research"!

Opinionated Preamble:

There is a lot of high level thinking on Less Wrong, which is great. It's done wonders to structure and optimize my own decisions. I think the political and futurology-related issues that Less Wrong cover can sometimes get out of sync with the reality and injustices of events in the immediate world. There are comprehensive treatments of how medical science is failing, or how academia cannot give unbiased results, and this is the milieu of programmers and philosophers in the middle-to-upper-class of the planet. I at least believe that this circle of awareness can be expanded, even if it's treading into mind-killing territory. If anything I want to give people a near-mode sense of the stakes aside from x-risk: all in all the x-risk scenarios I've seen Less Wrong fear the most, kill humanity somewhat instantly. A slower descent into violence and poverty is to me much more horrifying, because I might have to live in it and I don't know how. In a matter of fact, I have no idea of how to predict it.

This is one reason why I'm drawn to the Intelligence Operations performed by the military and crime units, among other things. Intelligence product delivery is about raw and immediate *fact*, and there is a lot of it. The problems featured in IntelOps are one of the few things rationality is good for - highly uncertain scenarios with one-off executions and messy or noisy feedback. Facts get lost in translation as messages are passed through, and of course the feeding and receiving fake facts are all a part of the job - but nevertheless, knowing *everything* *everywhere* is in the job description, and some form of rationality became a necessity.

It gets ugly. The demand for these kinds of skills often lie in industries that are highly competitive, violent, and illegal. I believe that once a close look is taken on how force and power is applied in practice then there isn't any pretending anymore that human evils are an accident.

Open Source Intelligence, or "OSINT", is the mining of data and facts from public information databases, news articles, codebases, journals. Although the amount of classified data dwarfs the unclassified, the size and scope of the unclassified is responsible for a majority of intelligence reports - and thus is involved in the great majority of executive decisions made by government entities. It's worth giving some thought as to how much that we know, that they do too. As illustrated in this expose, the processing of OSINT is a great big chunk of what modern intelligence is about aside from many other things. I think understanding how rationality as developed on Less Wrong can contribute to better IntelOps, and how IntelOps can feed the rationality community, would be awesome, but that's a post for another time.

--

The Show

Through my investigations into IntelOps I've noticed the emphasis on search. Good search.

I'm looking to compile a thread on Internet Research. I'm wondering if there is any wisdom on Less Wrong that can be taken advantage of here on how to become more effective searchers.  Here are some questions that could be answered specifically, but they are just guidelines - feel free to voice associated thoughts, we're exploring here.

• Before actually going out and searching, what would be the most effective way of drafting and optimizing a collection plan? Are there any formal optimization models that inform our distribution of time and attention? Exploration vs exploitation comes to mind, but it would be worth formulating something specific. I heard that the multi-armed bandit problem is solved?

• Do you have any links or resources regarding more effective search?

• Do you have any experiences regarding internet research that you can share? Any patterns that you've noticed that have made you more effective at searching?

• What are examples of closed-source information that are low-hanging fruit in terms of access (e.g. academic journals)? What are possible strategies for acquiring closed source data (e.g. enrolling in small courses at universities, e-mailing researchers, cohesion via the law/Freedom of Information Act, social engineering etc)?

• I would like to hear from SEOs and software developers on what their interpretation of semantic web technologies and how they are going to affect end-users. I am somewhat unfamiliar with the semantic web, but from my understanding information that could not be indexed is now indexed; and new ontologies will emerge as this information is mined. What should an end-user expect and what opportunities will there be that didn't exist in the current generation of search?

That should be enough to get started. Below are some links that I have found useful with respect to Internet Research.

--

Meta-Search Engines or Assisted Search:

• Carrot - http://search.carrot2.org/stable/search (concept clustering search engine)

Summarizers:

• TextTeaser - http://www.textteaser.com/ - SOURCE: https://github.com/MojoJolo/textteaser
• Copernic (Commercial Summarizing Feed Program) - http://www.copernic.com/en/products/summarizer/

Bots/Collectors/Automatic Filters:

• Google Alerts - http://www.google.ca/alerts
• Change Detection - http://www.changedetection.com/

Compilations and Directories:

• Directories and Search Engine Repository - http://rr.reuser.biz/index.html (probably the last one you'll ever need.)

• How to Perform Industry Research - http://businesslibrary.uflib.ufl.edu/industryresearch

Guides:

• Google Guide - http://www.googleguide.com/ (with practice and tutorials)

• From UC Berkeley - http://www.lib.berkeley.edu/TeachingLib/Guides/Internet/FindInfo.html 

• "How to Solve Impossible Problems" - http://www.johntedesco.net/blog/2012/06/21/how-to-solve-impossible-problems-daniel-russells-awesome-google-search-techniques/ 

• The NSA Guide to "Untangling the Web"; Internet Research - http://www.nsa.gov/public_info/_files/Untangling_the_Web.pdf [C. 2007]

• Fravia's Learnings on searching (value in essays) - http://search.lores.eu/indexo.htm [C. 1990s - 2009]

• "Power Searching With Google" Course - http://www.powersearchingwithgoogle.com/

Practice:

• SearchReSearch - http://searchresearch1.blogspot.ca/

• A Google A Day - http://agoogleaday.com/

I don't really care how you use this information, but I hope I've jogged some thinking of why it could be important.

A stub on a point that's come up recently.

If I owned a paperclip factory, and casually told my foreman to improve efficiency while I'm away, and he planned a takeover of the country, aiming to devote its entire economy to paperclip manufacturing (apart from the armament factories he needed to invade neighbouring countries and steal their iron mines)... then I'd conclude that my foreman was an idiot (or being wilfully idiotic). He obviously had no idea what I meant. And if he misunderstood me so egregiously, he's certainly not a threat: he's unlikely to reason his way out of a paper bag, let alone to any position of power.

If I owned a paperclip factory, and casually programmed my superintelligent AI to improve efficiency while I'm away, and it planned a takeover of the country... then I can't conclude that the AI is an idiot. It is following its programming. Unlike a human that behaved the same way, it probably knows exactly what I meant to program in. It just doesn't care: it follows its programming, not its knowledge about what its programming is "meant" to be (unless we've successfully programmed in "do what I mean", which is basically the whole of the challenge). We can't therefore conclude that it's incompetent, unable to understand human reasoning, or likely to fail.

We can't reason by analogy with humans. When AIs behave like idiot savants with respect to their motivations, we can't deduce that they're idiots.

Triggered by:  Future Story Status 

A helpful key to understanding the art and technique of character in storytelling, is to consider the folk-psychological notion from Internal Family Systems of people being composed of different 'parts' embodying different drives or goals. A shallow character is a character with only one 'part'.

A good rule of thumb is that to create a 3D character, that person must contain at least two different 2D characters who come into conflict. Contrary to the first thought that crosses your mind, three-dimensional good people are constructed by combining at least two different good people with two different ideals, not by combining a good person and a bad person. Deep sympathetic characters have two sympathetic parts in conflict, not a sympathetic part in conflict with an unsympathetic part. Deep smart characters are created by combining at least two different people who are geniuses.

E.g. HPMOR!Hermione contains both a sensible young girl who tries to keep herself and her friends out of trouble, and a starry-eyed heroine, neither of whom are stupid.  (Actually, since HPMOR!Hermione is also the one character who I created as close to her canon self as I could manage - she didn't *need* upgrading - I should credit this one to J. K. Rowling.)  (Admittedly, I didn't actually follow that rule deliberately to construct Methods, I figured it out afterward when everyone was praising the characterization and I was like, "Wait, people are calling me a character author now?  What the hell did I just do right?")

If instead you try to construct a genius character by having an emotionally impoverished 'genius' part in conflict with a warm nongenius part... ugh.  Cliche.  Don't write the first thing that pops into your head from watching Star Trek.  This is not how real geniuses work.  HPMOR!Harry, the primary protagonist, contains so many different people he has to give them names, and none of them are stupid, nor does any one of them contain his emotions set aside in a neat jar; they contain different mixtures of emotions and ideals.  Combining two cliche characters won't be enough to build a deep character.  Combining two different realistic people in that character's situation works much better.  Two is not a limit, it's a minimum, but everyone involved still has to be recognizably the same person when combined.

Closely related is Orson Scott Card's observation that a conflict between Good and Evil can be interesting, but it's often not half as interesting as a conflict between Good and Good. All standard rules about cliches still apply, and a conflict between good and good which you've previously read about and to which the reader can already guess your correct approved answer, cannot carry the story. A good rule of thumb is that if you have a conflict between good and good which you feel unsure about yourself, or which you can remember feeling unsure about, or you're not sure where exactly to draw the line, you can build a story around it. I consider the most successful moral conflict in HPMOR to be the argument between Harry and Dumbledore in Ch. 77 because it almost perfectly divided the readers on who was in the right *and* about whose side the author was taking.  (*This* was done by deliberately following Orson Scott Card's rule, not by accident.  Likewise _Three Worlds Collide_, though it was only afterward that I realized how much of the praise for that story, which I hadn't dreamed would be considered literarily meritful by serious SF writers, stemmed from the sheer rarity of stories built around genuinely open moral arguments.  Orson Scott Card:  "Propaganda only works when the reader feels like you've been absolutely fair to other side", and writing about a moral dilemma where *you're* still trying to figure out the answer is an excellent way to achieve this.)

Character shallowness can be a symptom of moral shallowness if it reflects a conflict between Good and Evil drawn along lines too clear to bring two good parts of a good character into conflict. This is why it would've been hard for Lord of the Rings to contain conflicted characters without becoming an entirely different story, though as Robin Hanson has just remarked, LotR is a Mileu story, not a Character story.  Conflicts between evil and evil are even shallower than conflicts between good and evil, which is why what passes for 'maturity' in some literature is so uninteresting. There's nothing to choose there, no decision to await with bated breath, just an author showing off their disillusionment as a claim of sophistication.

Returned to original title, for the good reasons given here

There was a recent post in Discussion which at time of this writing held staggering 454 commentaries, which inclined me to write an evolutionary psychology and social endocrinology derived post on courtship, and Mating Intelligence, to share some readings on recent discussions and evidence coming from those areas. I've been meaning to do this for a while, and a much longer version could have been written, with more specific case studies and citations and an academic outlook, yet I find this abridged personal version more adequate for Lesswrong. In no area more disclaimers are desirable than when speaking about evolutionary drives for mating. It touches emotions, gender issues, morality, societal standards, and it speaks of topics that make people shy, embarrassed, angry and happy on a weekly basis, so I'll begin with a few paragraphs of disclaimers.

I'll try to avoid saying anything that I can remember having read in a Pick Up Artist book, and focus on using less known mating biases to help straight women and men find what they look for in different contexts. This post won't work well for same-gender seduction. If you object irrevocably to evolutionary psychology, just so stories, etc... I suggest you refrain from commenting, and also reading, why bother?

Words of caution on reading people (me included) talking about evolutionary psychology, specially when applied to current people: Suspicious about whether there is good evidence for it? Read this first, then if you want Eliezer on the evolutionary-cognitive difference, and this if your feminist taste buds activate negatively. If you never heard of Evolutionary Psychology (which includes 8 different bodies of data to draw from), check also an Introduction with Dawkins and Buss.

When I say "A guy does D when G happens" please read: "There are statistically significant, or theoretically significant reasons from social endocrinology, or social and evolutionary psychology to believe that under circumstances broadly similar to G, human males, on average, will be inclined towards behaving in manners broadly similar to the D way. Also, most tests are made with western human males, tests are less than 40 years old,  subject to publication bias, and sometimes done by people who don't understand math well enough to do their statistics homework, they have not been replicated several times, and they are less homogenous than physics, because psychology is more complex than physics."

If you couldn't care less for theory, and just want the advice, go to the Advice Session.

Misconceptions

Thusfar in Evolutionary Psychology it seems that our genes come equipped with two designs that become activated through environmental cues to think about mating.

Short-term mating

Long-term mating

Knowing this is becoming mainstream. The state of the art term is Mating Intelligence, and it has these two canonical modes that can be activated, depending on factors as diverse as being informed that X is leaving town in two days, and detecting X's level of testosterone, accounting for his height and status, and calculating whether his genes are worth more or less than his future company. If you choose to read the linked books, then you'll delve in this much deeper than I have, so stop reading this, and write a post of your own afterwards.

I'll list some main misconceptions, then suggest how to use either the misconceptions, or the theory mentioned while explaining them to optimize for whatever you want from the opposite gender individuals at a particular moment.

Misconception 1: Guys do Short-term, Girls do Long-term, unless they don't have this option.

This is false. Guys are very frequently pair bonded, most times even before women are, both have oxytocin levels going up after sex, and both have high levels of oxytocin during relationships. Girls only have less frequent causal intercourse because it is hard to find males worthy of the 2 year raising a baby period, or in the case in which they are pair-bonded already, because of the risk of the cuckolded "father" leaving, fighting her, or recognizing the baby ain't his. Obviously, no one's brain has managed to completely catch up with condoms and open relationships yet.

Misconception 2: Women go for the bad guys (if I remember my American Pie's correctly, also called jocks in US) and good guys, nerds, and conventionals are left last. 

'Bad guys' is a popular name for high testosterone, risk taking, little routine individuals. And indeed when a woman's short-term mating intelligence program is activated, which happens particularly when she is ovulating and young (even when she's close married/relationshiped) she does exhibit a preference for such types. When optimizing for long-term partners, the reverse is true.

Misconception 3: Guys just go for looks, Girls just go for status. 

Toned down reality: Guys in short-term mating mode go for looks, Girls in long-term mating mode care substantially for the difference between lower than average status and average status, then marginal utility decreases and more status is defeated by other desirable traits.

Women in short-term mode do not optimize for status, they'll take a bus-boy who shows through size, melanin, symmetry and chin that he survived local pathogens despite his high testoterone, she's after resistant genes, not resources. Men in long term mode still optimize for looks, but not that much, kindness and emotional stability take over when marginal returns for more beauty start subsiziding.

Misconception 4: When genders optimize for Status, Status=Money.

Unlike all known primate and cetacean species, Humans daily deal with being high, low, and medium status in different hierarchical situations. This should be as obvious as not to be worth mentioning, but sadly there are strong media incentives, and for some reason I don't understand well strong reasons within English and American culture to pretend that women go for status, status=money, therefore women go for money, and men should make more money. It may be a selection effect, the societies that financially took over the world believed that being financially powerful was the best way to get laid, or marry. It may just be that marketing these things together (using sexy women to sell cars) created a long-term pavlovian association. Fact is that it unfortunately happened, and people believe it, despite it being false. Women who begin believing it sometimes force themselves into doing it even more. 

Status has no universal measure. If you met someone in Basketball team, status will be how good that person is plus their game attitude. If in a class at university, maybe it will be how well spoken the person is in the relevant topic. Status can be how much food the person usually shares with groups, or how much they can ask for others without being very apologetic. It can be how many women sleep with a man, or how many he can afford to reject. It can be how many purses a woman has, or how she can show thrift and a sense of belonging to a community that identifies as anti-consumerist. Some minds assign status based on location of birth, race, hair color etc...   (In my city, Japanese women, all the 400.000, are commonly assumed to be high status). Finally, men do optimize for the trait people think as status, explained below, in long-term mates. 

Even in the case where status plays the largest role, women when activating long-term reasoning, status is only one factor out of four multiplicants that are important for the same reason, and detected, in a prospective male mate:  

Kindness*Dependability*(Ambition-Age)*Status = How many resources a man is expected to share with you and your hypothetical kids.

And this does not even begin to account for any physical trait, nor intelligence, humour, energy levels etc... If you take one thing out of this text, take this: Make your beliefs about what status is pay rent. Test if status is what people think it is, or something that only roughly correlates with that. Sophisticate your status modules, they may have been corrupted.

Misconception 5: Once you learn what your mind is doing when it selects mates, you should make it get better at that.

Let's begin by reaffirming the obvious: We live in a world that has nothing to do with savannahs where our minds spent a long time. We can access thousands, if not millions of people, during a lifetime. We have condoms and contraceptives. We live in an era of abundance compared to any other time in history, and in societies so large, that the moral norms constraining what "everyone will know" do not apply anymore.

So the last thing you want to do is to make your mind really sharp and accurate when judging a potential mate through its natural algorithms. What you want to do, to the extent that it is possible, is to override your algorithms with something that is better, and better is one of these two things:

1) Increasing your likelihood of mating with the individual (or class of individuals) you want to mate with in a matched time-horizon (long if you want long, for instance).

2) Enlarging the scope of individuals you want to mate with to include more people you actually do, will or can get to know. 

Advice

To give better advice, I'll first mention general advice anyone can use, and then specific advice for the four quadrants. For those who will say this is the Dark Arts, I say it would be if we lived in a Savannah without condoms, heating, medicine, houses or internets. Now it looks to me more like causing one-self, and one's beloved, to be more epistemically rational.

General Advice

Women, be confident: If you are a woman, be more confident, way more confident, when approaching a guy, don't be aggressive, just safe, you mind is tuned with who knows how many trigger devices that may make you afraid of a no, of being thought of as slutty, of losing face, and of the guy not raising your kids. Discount for all that, twice. Don't do it if everyone really will know, or if you actually want kids from that guy.

Use your best horizon features: If you have a trait that the other gender optimizes for more in short-term, lure them by acting short-term, even if later you'll attempt to raise their oxytocin to the long-term point. If you have goods and ills on both time horizons, switch back and forth until you grasp what they want. 

Discount for population size: There are two ways of doing that, one is to reason to yourself "I may not be as attractive as Natalie Portman or Brad Pitt, but our minds are tuned to trying to get the best few achievable mates out of a group of 100-1000, not of hundreds of millions, so I do stand a very good chance" The other is nearly opposite: "I may think that I should only marry a prince, or sleep with Iron Man, but in fact my world is much smaller than this, and my mind will be totally okay to mate with Adam, that cool guy."

Be hedonistic: For men and women alike, the main way evolution got us into intercourse was by making it fun. The reasons it got us out are related to unlikelihood of leaving great-grandchildren, energy waste, disease, and lowered status. Of those, only a subset of lowered status is still significant in a world full of condoms. Other than women when aiming at long-term only, everyone is completely under-calibrated for sex, since we substantially reduced the risks without reducing the hedonic benefits nearly as much.

Use fetishes and peculiarities: There are things each particular person is attracted to more than everyone else (for me that's freckles, red/orange/blue/purple hair, upper back, and short women). Use that in your favour, less competition, as simple as that.

Go places: There are better and worse places to find mates. Short-terming males (a temporary condition in which any male may find himself, not a kind of male) abound in dancing clubs, military facilities and sports areas, not to mention OkCupid. Long-terming females (same) abound on courses and classes of yoga, dancing, cooking, languages, etc...  Long-terming males usually have more of a routine, so are more frequent on saturdays and fridays than on a tuesday late evening, they'll be more frequent wherever no one naturally would go to find a one night stand, or in groups that are preselected for strong emotions (low thresholds for falling in love) Short-terming females may exist in dancing clubs, bars and other related areas, but are very high value due to comparative scarcity when in these areas, someone looking for them is better off in groups with a small majority of women, where social tension and hierarchies don't scale up in either gender.

Specific Advice

Note: The advice is about things you should do in addition to what you naturally tend to do in those situations, you already have the algorithms, and should just improve calibration, unless when explicited, the suggestion is not to substitute what you naturally tend to do, or this would be a book all by itself explaining 4 kinds of human courtship.

For Long-terming Men: Stop freaking out about financial status. Find a place where you are among the great ones in something, specially kindness, dependability, physical constitution, and symmetry which guys think of less frequently than Successful startups or Tennis worldchampions. If you are hot, use short-term, women are particularly more prone to switching from short to long-term. Get a dog, show you are able and willing to take care of something unspeakably cute and adorable. Be ambitious in your projects, show passion. While ambitious and passionate, also make sure she realizes (truly) that you notice things about her no one else does, find out her values, talk about shared ones, and be non aggressively curious about all of them.  Show her kindness in small gestures that need not cost a lot, such as time consuming hand-made presents. Test OkCupid and see if it works for you. Memorize details about her personality, assure her you can be loving specifically to her. Postpone sex a little bit. May sound hard, but is a reliable indicator that you won't change her for the next that quickly. Rationally override any emotion you may have regarding her sexual behavior, show you are not agressive and jealous, thus making her "(be) (a)lieve unconsciously" that you will not kill her in an assault of hatred when she sleeps with hypothetical another man whose child will never exist and get some years of schooling from you. If you think you can tell the wheat from the chaff, separate the PUA stuff that works for long-term, if not, read softer confidence/influence/seduction material. Use oxytocin inducing media (TV series and romantic movies). Rest assured, there are more women looking for long-term men than the opposite, aid the odds by going places. Show sympathy, kindness (to others as well) and dependability whenever you can.

For Long-terming Women: If you've been convinced by financial status gospel, stop freaking out about it. If you just account for the 4 factors in the equation above, you'll be way ahead of everyone within the gospel trance, then there are still all the other things you look for in a guy, which by themselves are very important. Sure, a classic indicator is how much other women in your social group like him, and, good as it is, it is defined in terms of competition, try to discount this one, after all, it is partially just made of a conformity bias, a bad bias to have when looking for a long-term mate. Be very nice and kind, and almost silly near the guy. The kinds of guys who are Long-terming most of the time are those who won't approach you that frequently. Also, older guys obviously have less chaos on in their minds and lives, so are more likely to want to settle down for a few years. Postpone sex in proportion to how much you suspect the guy is Short-terming. The importance of this cannot be overstated. By postponing sex (and sex alone) you make sure Short-termers still have a good reason to be around you until suddenly there is a hormonal overload and they fall in love with you (not that romantic, but mildly accurate), love's trigger is activated by many factors, when they sum above a threshold. The most malleable of these factors is time investment, give a guy mixed short long signals, and you'll increase likelihood of surpassing the threshold. Also, give known guys a second chance, many times your algorithms friendzoned (sorry for the term) them for reasons as silly as "he didn't touch me the first time we met, and I didn't feel his smell, because the table was wide" or "That day I was in Short-term mode and this other guy had more easily detectable attractive features, leaving John on the omega mental slot". Forget romantic comedies and princess tales where your role is passive. A man's love is actively conquered by a woman, you are the one who will fight dragons - frequently RPG dragons - for the guy in the beggining, not the opposite, the opposite comes later as a prize. 

For Short-terming Guys: Read Pick Up Artist books, actually do the exercises, as in don't find excuses for why you can't, do them. Don't do anything that disgusts you morally, which may be nearly all of it, but do all the rest. Other than that?... Some few things, very few indeed, were left out of those books. Optimize more than anything for your fetishes and specific desires to avoid competition. Use mildly tense situations which can be confounded with arousal (narrow bridges get you more dates than wide bridges). Woman's attractiveness peaks at approximately 1,73cm 5 feet 8 inches, shorter women are more likely to have had less home stability and developmental stability when young, which triggers more frequent short-terming, looking for testosterone indicators (square chin, prominent forehead, and specially having a ring-finger longer than index-finger) also helps, and it is fun because you can claim to read hands and actually make good predictions out of it.

For Short-terming Girls: I'll start with easy stuff, and escalate quickly to extremely high probability even in tough cases, such as he's not on the mood, tired, really shy, or (you think) not excited. Quite likely the main obstacle is inside your mind, not your clothes, either fear of rejection, or fear of reputational cost or something else. Be confident. Few guys will reject a subtle, feminine, discrete and firm sex "offer" (notice how language itself puts it). Look at him, smile, touch him while you speak, look intensely at his mouth while slowly approaching, make sure to try do this where he is unlikely to be paying some reputational cost (not on his aunt's marriage). If feeling clumsy, mention you do. When short-terming, men really do optimize for looks, so decrease light levels, and avoid available-female company, like asking him out to check a bookstore, or to see a movie. Sit near him while touching him, cut the conversation at some point, kiss him (remember to do that where neither of you may get embarrassed with anyone else). Before, talk about sexuality naturally and imagetically, say how it is important to you to be embraced, desired, enticed, penetrated, transformed inside, and arise re-energized the next day to go back to your life. If you are sure he is short-terming, make yourself scarce by mentioning time constraints. Carry condoms and pick them up while making up if he is still hesitant whether you want sex or not. But be cozy and reassure him "It's okay" if it feels like he nervous. If you are confortable with that, use the web, there are tons of Short-terming guys, and if you feel embarassed to meet a man who would reject you, you are safeguarded by being filtered beforehand through your pictures and description or by the bang with friends app. On the web, be upfront about your intentions, and assure them you are not a scam/bot/adv. When almost there, if he is not excited, it is not because you are not attractive to him, don't be passive, slowly touch and rub his genital, quite likely he's just nervous and you are disputing against his sympathetic system, when you and the parasympathetic win, he'll be excited and relaxed, and the party is on. If you live in a large urban area, go to swing places alone or with acquaintances, not friends - nowhere else there will be that many guys willing to have sex right there, right now, and the necessary infrastructure for it, in a safe environment with security guards, other high-class women etc... to make sure you are not getting into trouble - In short, guarantee situations in which neither him nor you pay reputational costs, be active yet reassuring, lower light levels, avoid competition and make sure there is infrastructure for the act.

The saying goes that you can't achieve happiness by trying to be happy (thought you can if you optimize for happiness, i.e. by reading positive psychology and acting on it). To some extent, it is also true that a lot of what goes on during courtship does not take place while actively and consciously focusing on courtship. It is one thing to keep those misconceptions and advices in mind, and a whole different thing to be obsessed about them and use them as cognitive canonical maxims for behaving, the point of writing this is to help, if it stops being helpful, stop using it.

Edit: Scrambled sources:

The Future of Humanity Institute has released video footage of the 2012 Winter Intelligence Conference.  The videos currently available are:

• Stuart Armstrong - Predicting AI... or Failing to
• Miles Brundage - Limitations and Risks of Machine Ethics
• Steve Omohundro - Autonomous Technology and the Greater Human Good
• Anders Sandberg - Ethics and Impact of Brain Emulations
• Carl Shulman - Could We Use Untrustworthy Human Brain Emulations to Make Trustworthy Ones

This paper seems relevant to various LW interests. It smells like The Second Law of Thermodynamics, and Engines of Cognition, but I haven't wrapped my head enough around either to say more than that. Abstract:

Recent advances in fields ranging from cosmology to computer science have hinted at a possible deep connection between intelligence and entropy maximization, but no formal physical relationship between them has yet been established. Here, we explicitly propose a first step toward such a relationship in the form of a causal generalization of entropic forces that we find can cause two defining behaviors of the human “cognitive niche”—tool use and social cooperation—to spontaneously emerge in simple physical systems. Our results suggest a potentially general thermodynamic model of adaptive behavior as a nonequilibrium process in open systems.

My paper "General purpose intelligence: arguing the Orthogonality thesis" has been accepted for publication in the December edition of Analysis and Metaphysics. Since that's some time away, I thought I'd put the final paper up here; the arguments are similar to those here, but this is the final version, for critique and citation purposes.

General purpose intelligence: arguing the Orthogonality thesis

STUART ARMSTRONG
stuart.armstrong@philosophy.ox.ac.uk
Future of Humanity Institute, Oxford Martin School
Philosophy Department, University of Oxford

In his paper “The Superintelligent Will”, Nick Bostrom formalised the Orthogonality thesis: the idea that the final goals and intelligence levels of artificial agents are independent of each other. This paper presents arguments for a (narrower) version of the thesis. It proceeds through three steps. First it shows that superintelligent agents with essentially arbitrary goals can exist in our universe – both as theoretical impractical agents such as AIXI and as physically possible real-world agents. Then it argues that if humans are capable of building human-level artificial intelligences, we can build them with an extremely broad spectrum of goals. Finally it shows that the same result holds for any superintelligent agent we could directly or indirectly build. This result is relevant for arguments about the potential motivations of future agents: knowing an artificial agent is of high intelligence does not allow us to presume that it will be moral, we will need to figure out its goals directly.

Keywords: AI; Artificial Intelligence; efficiency; intelligence; goals; orthogonality

1                       The Orthogonality thesis

Scientists and mathematicians are the stereotypical examples of high intelligence humans. But their morality and ethics have been all over the map. On modern political scales, they can be left- (Oppenheimer) or right-wing (von Neumann) and historically they have slotted into most of the political groupings of their period (Galois, Lavoisier). Ethically, they have ranged from very humanitarian (Darwin, Einstein outside of his private life), through amoral (von Braun) to commercially belligerent (Edison) and vindictive (Newton). Few scientists have been put in a position where they could demonstrate genuinely evil behaviour, but there have been a few of those (Teichmüller, Philipp Lenard, Ted Kaczynski, Shirō Ishii).

If I understand the Singularitarian argument espoused by many members of this community (eg. Muehlhauser and Salamon), it goes something like this:

1. Machine intelligence is getting smarter.
2. Once an intelligence becomes sufficiently supra-human, its instrumental rationality will drive it towards cognitive self-enhancement (Bostrom), so making it a super-powerful, resource hungry superintelligence.
3. If a superintelligence isn't sufficiently human-like or 'friendly', that could be disastrous for humanity.
4. Machine intelligence is unlikely to be human-like or friendly unless we take precautions.

I'm in danger of getting into politics.  Since I understand that political arguments are not welcome here, I will refer to these potentially unfriendly human intelligences broadly as organizations.

Smart organizations

By "organization" I mean something commonplace, with a twist.  It's commonplace because I'm talking about a bunch of people coordinated somehow. The twist is that I want to include the information technology infrastructure used by that bunch of people within the extension of "organization". 

Do organizations have intelligence?  I think so.  Here's some of the reasons why:

1. We can model human organizations as having preference functions. (Economists do this all the time)
2. Human organizations have a lot of optimization power.

I talked with Mr. Muehlhauser about this specifically. I gather that at least at the time he thought human organizations should not be counted as intelligences (or at least as intelligences with the potential to become superintelligences) because they are not as versatile as human beings.

So when I am talking about super-human intelligence, I specifically mean an agent that is as good or better at humans at just about every skill set that humans possess for achieving their goals. So that would include things like not just mathematical ability or theorem proving and playing chess, but also things like social manipulation and composing music and so on, which are all functions of the brain not the kidneys

...and then...

It would be a kind of weird [organization] that was better than the best human or even the median human at all the things that humans do. [Organizations] aren’t usually the best in music and AI research and theory proving and stock markets and composing novels. And so there certainly are  [Organizations] that  are better than median humans at certain things, like digging oil wells, but I don’t think there are [Organizations] as good or better than humans at all things. More to the point, there is an interesting difference here because [Organizations] are made of lots of humans and so they have the sorts of limitations on activities and intelligence that humans have. For example, they are not particularly rational in the sense defined by cognitive science. And the brains of the people that make up organizations are limited to the size of skulls, whereas you can have an AI that is the size of a warehouse. 

I think that Muehlhauser is slightly mistaken on a few subtle but important points.  I'm going to assert my position on them without much argument because I think they are fairly sensible, but if any reader disagrees I will try to defend them in the comments.

• When judging whether an entity has intelligence, we should consider only the skills relevant to the entity's goals.
• So, if organizations are not as good at a human being at composing music, that shouldn't disqualify them from being considered broadly intelligent if that has nothing to do with their goals.
• Many organizations are quite good at AI research, or outsource their AI research to other organizations with which they are intertwined.
• The cognitive power of an organization is not limited to the size of skulls. The computational power is of many organizations is comprised of both the skulls of its members and possibly "warehouses" of digital computers.
• With the ubiquity of cloud computing, it's hard to say that a particular computational process has a static spatial bound at all.

Mean organizations

* My preferred standard of rationality is communicative rationality, a Habermasian ideal of a rationality aimed at consensus through principled communication.  As a consequence, when I believe a position to be rational, I believe that it is possible and desirable to convince other rational agents of it.

Hello,

This is my first posting here, so please forgive me if I make any missteps.

The outline draft below draws heavily on Intelligence Explosion: Evidence and Import (Muehlhauser and Salamon 2011?). I will review Stuart Armstrong’s How We're Predicting AI... or Failing to, (Armstrong 2012) for additional content and research areas.

I'm not familiar with the tone and tenor of this community, so I want to be clear about feedback. This is an early draft and as such, nearly all of the content may or may not survive future edits. All constructive feedback is welcome. Subjective opinion is interesting, but unlikely to have an impact unless it opens lines of thought not previously considered.

I'm looking forward to a potentially lively exchange.

Jay

Predicting Machine Super Intelligence

Jacque Swartz

Most Certainly Not Affiliated with Singularity Institute

jaywswartz@gmail.com

Abstract

This paper examines the disciplines, domains, and dimensional aspects of Machine Super Intelligence (MSI) and considers multiple techniques that have the potential to predict the appearance of MSI. Factors that can impact the speed of discovery are reviewed. Then, potential prediction techniques are considered. The concept of MSI is dissected into the currently comprehended components. Then those components are evaluated to indicate their respective state of maturation and the additional behaviors required for MSI. Based on the evaluation of each component, a gap analysis is conducted. The analyses are then assembled in an approximate order of difficulty, based on our current understanding of the complexity of each component. Using this ordering, a collection of indicators is constructed to identify an approximate progression of discoveries that ultimately yield MSI. Finally, a model is constructed that can be updated over time to constantly increase the accuracy of the predicted events, followed by conclusions.

I. Introduction

Predicting the emergence of MSI could potentially be the most important pursuit of humanity. The distinct possibility of an MSI emerging that could harm or exterminate the human race (citation) demands that we create an early warning system. This will give us the opportunity to ensure that the MSI that emerges continues to advance human civilization (citation).

We currently appear to be at some temporal distance from witnessing the creation of MSI (multiple citations). Many factors, such as a rapidly increasing number of research efforts (citation) and motivations for economic gain (citation), clearly indicate that there is a possibility that MSI could appear unexpectedly or even unintentionally (citation).

Some of the indicators that could be used to provide an early warning tool are defined in this paper. The model described at the end of the paper is a potentially viable framework for instrumentation. It should be refined and regularly updated until a more effective tool is created or the appearance of MSI.

This paper draws heavily upon Intelligence Explosion: Evidence and Import (Muehlhauser and Salamon 2011?) and Stuart Armstrong’s How We're Predicting AI... or Failing to, (2012).

This paper presupposes that MSI is generally understood to be equivalent to Artificial General Intelligence (AGI) that has developed the ability to function at levels substantially beyond current human abilities. The latter term will be used throughout the remainder of this paper.

II. Overview

In addition to the fundamental challenge of creating AGI, there are a multitude of theories as to the composition and functionality of a viable AGI. Section three explores the factors that can impact the speed of discovery in general. Individual indicators are explored for unique factors to consider. The factors identified in this section can radically change the pace of discovery.

The fourth section considers potential prediction techniques. Data points and other indicators are identified for each prediction model. The efficacy of the models is examined and developments that increase a model’s accuracy are discussed.

The high degree of complexity of AGI indicates the need to subdivide AGI into its component parts. In the fifth section the core components and functionality required for a potential AGI are established. Each of the components is then examined to determine its current state of development. Then an estimate of the functionality required for an AGI is created as well as recording of any identifiable dependencies. A gap analysis is then performed on the findings to quantify the discoveries required to fill the gap.

This approach does increase the likelihood of prediction error due to the conjunction fallacy, exemplified by research such as the dice selection study (Tversky and Kahneman 1983) and covered in greater detail by Eliezer Yudkowski’s bias research (Yudkowski 2008). Fortunately, the exposure to this bias diminishes as each component matures to its respective usability point and reduces the number of unknown factors.

The sixth section examines the output of the gap analyses for additional dependencies. Then the outputs are assembled in an approximate order of difficulty, based on our current understanding of the complexity of each output. Using this ordering, combined with the dependencies, a collection of indicators with weighting factors is constructed to identify an approximate progression of discoveries that ultimately yield AGI.

Comprehending the indicators, dependencies and rate factors in a model as variables provides a means, however crude, to reflect their impact when they do occur.

In the seventh section, a model is constructed to use the indicators and other inputs to estimate the occurrence of AGI. It is examined for strengths and weaknesses that can be explored to improve the model. Additional enhancements to the model are suggested for exploration.

The eighth and final section includes conclusions and considerations for future research.

III. Rate Modifiers

This section explores the factors that can impact the speed of discovery. Individual indicators are explored for unique factors to consider. While the factors identified in this section can radically change the pace of discovery, comprehending them in the model as variables provides a means to reflect their impact when they do occur.

Decelerators

    Discovery Difficulty

    Disinclination

    Lower Probability Events

       Societal Collapse

       Fraud

    ++

Accelerators

    Improved Hardware

    Better Algorithms

    Massive Datasets

    Progress in Psychology and Neuroscience

    Accelerated Science

    Collaboration

    Crossover

    Economic Pressure

    Final Sprint

    Outliers

    Existing Candidate Maturation

    ++

IV. Prediction Techniques

This section considers potential prediction techniques. Some techniques do not require the indicators above. Most will benefit by considering some or all of the indicators. It is very important to not loose sight of the fact that mankind is inclined to inaccurate probability estimates and overconfidence (Lichtenstein et al. 1992; Yates et al. 2002)

Factors Impacting Accurate Prediction

Prediction Models

    Wisdom of Crowds

    Hardware Extrapolation

    Breakthrough Curve

    Evolutionary Extrapolation

    Machine Intelligence Improvement Curve

    ++

V. Potential AGI Componentry

This section establishes a set of core components and functionality required for a potential AGI. Each of the components is then examined to determine its current state of development as well as any identifiable dependencies. Then an estimate of the functionality required for a AGI is created followed by a gap analysis to quantify the discoveries required to fill the gap.

There are various existing AI implementations as well as AGI concepts currently being investigated. Each one brings in unique elements. The common elements across most include; decision processing, expert systems, pattern recognition and speech/writing recognition. Each of these would include discipline-specific machine learning and search/pre-processing functionality. There also needs to be a general learning function for addition of new disciplines.

Within each discipline there are collections of utility functions. They are the component technologies required to make the higher order discipline efficient and useful. Each of the elements mentioned are areas of specialized study being pursued around the world. They draw from an even larger set of specializations. Due to complexity, in most cases there are second-order, and more, specializations.

Alternative Componentry

There are areas of research that have high potential for inserting new components or substantially modifying the comprehension of the components described.

Specialized Componentry

Robotics and other elements.

Current State

    Decision Processing

    Expert Systems

    Pattern Recognition

    Speech/Writing Recognition

    Machine Learning

        Decision Processing

       Expert Systems

       Pattern Recognition

       Speech/Writing Recognition

    Search/Pre-Processing

       Decision Processing

       Expert Systems

       Pattern Recognition

       Speech/Writing Recognition

Target State

The behaviors required for an AGI to function with acceptable speed and accuracy are not precise. The results of this section are based on a survey of definitions from available research.

    Decision Processing

    Expert Systems

    Pattern Recognition

    Speech/Writing Recognition

    Machine Learning

        Decision Processing

       Expert Systems

       Pattern Recognition

       Speech/Writing Recognition

    Search/Pre-Processing

       Decision Processing

       Expert Systems

       Pattern Recognition

       Speech/Writing Recognition

Dependencies

Gap Analysis

VI. Indicators

The second section examines the output of the gap analyses for additional dependencies. Then the outputs are assembled in an approximate order of difficulty, based on our current understanding of the complexity of each output. Using this ordering, combined with the dependencies, a collection of indicators is constructed to identify an approximate progression of discoveries that ultimately yield an AGI.

Additional Dependencies

Complexity Ranking

Itemized Indicators

VII. Predictive Model

In this section, a model is constructed using the indicators and other inputs to estimate the occurrence of AGI. It is examined for strengths and weaknesses that can be explored to improve the model. Additional enhancements to the model are suggested for exploration.

The Model

Strengths & Weaknesses

Enhancements

VIII. Conclusions

Based on the data and model created above the estimated time frame for the appearance of AGI is from x to y. As noted throughout this paper, the complex nature of AGI and the large number of discoveries and events that need to be quantified using imperfect methodologies, a precise prediction of when AGI will appear is currently impossible.

The model developed in this paper does establish a quantifiable starting point for the creation of an increasingly accurate tool that can be used to continually narrow the margin of error. It also provides a starting set of indicators that can serve as early warning of AGI when discoveries and events are made.

The result is from 2008, but it's new to me. Abstract:

A meta-study of repeated prisoner’s dilemma experiments run at numerous universities suggests that students cooperate 5% to 8% more often for every 100 point increase in the school’s average SAT score.

Some obvious points from my first five minutes of thinking about it:

• Meta-study or not, the sample still only covers humans. No implications for Friendly AI or intelligent aliens, which don't have our motivations.
• Even among humans the sample is WEIRD, and a subset of WEIRD at that; although there is obviously variation between universities, it's smaller than what you'd get if you extended the sample down into the working class. I also wonder what would happen if the PD was played between students and non-students. 
• Probably a point in favour of the Machiavellian intelligence hypothesis, in that we see those of higher intelligence doing better on a social problem.
• Presumably this implies that your best move, whatever your level of intelligence, is to surround yourself with the smartest people you can find, and then cooperate to ensure they don't throw you out.
• I'd like to know some details: Does intelligence also correlate with effective retaliation? With probing for suckers? What about cooperation in single games? (The study mentions one, in a footnote, which apparently did find higher intelligence correlated with greater cooperation even in one-shot games; but there's no metastudy.)

In 2007, Legg and Hutter wrote a paper using the AIXI model to define a measure of intelligence. It's pretty great, but I can think of some directions of improvement.

• Reinforcement learning. I think this term and formalism are historically from much simpler agent models which actually depended on being reinforced to learn. In its present form (Hutter 2005 section 4.1) it seems arbitrarily general, but it still feels kinda gross to me. Can we formalize AIXI and the intelligence measure in terms of utility functions, instead? And perhaps prove them equivalent?
• Choice of Horizon. AIXI discounts the future by requiring that total future reward is bounded, and therefore so does the intelligence measure. This seems to me like a constraint that does not reflect reality, and possibly an infinitely important one. How could we remove this requirement? (Much discussion on the "Choice of the Horizon" in Hutter 2005 section 5.7).
• Unknown utility function. When we reformulate it in terms of utility functions, let's make sure we can measure its intelligence/optimization power without having to know its utility function. Perhaps by using an average of utility functions weighted by their K-complexity.
• AI orientation. Finally, and least importantly, it tests agents across all possible programs, even those which are known to be inconsistent with our universe. This might okay if your agent is a playing arbitrary games on a computer, but if you are trying to determine how powerful an agent will be in this universe, you probably want to replace the Solomonoff prior with the posterior resulting from updating the Solomonoff prior with data from our universe.

Any thought or research on this by others? I imagine lots of discussion has occurred over these topics; any referencing would be appreciated.

A time dilation tool from an anime is discussed for its practical use on Earth; there seem surprisingly few uses and none that will change the world, due to the severe penalties humans would incur while using it, and basic constraints like Amdahl's law limit the scientific uses. A comparison with the position of an Artificial Intelligence such as an emulated human brain seems fair, except most of the time dilation disadvantages do not apply or can be ameliorated and hence any speedups could be quite effectively exploited. I suggest that skeptics of the idea that speedups give advantages are implicitly working off the crippled time dilation tool and not making allowance for the disanalogies.

Master version on gwern.net

I'm writing this because, for a while, I have noticed that I am confused: particularly about what people mean when they say someone is intelligent. I'm more interested in a discussion here than actually making a formal case, so please excuse my lack of actual citations. I'm also trying to articulate my own confusion to myself as well as everyone else, so this will not be as focused as it could be.

If I had to point to a starting point for this state, I'd say it was in psych class, where we talked about research presented by Eyesenck and Gladwell. Eyesenck is very clear to define intelligence as the ability to solve abstract problems, but not necessarily the motivation . In many ways, this matches Yudkowsky's definition, where he talks about intelligence as a property we can ascribe to an entity, which lets us predict that the entity will be able to complete a task, without ourselves necessarily understanding the steps toward completion.

The central theme I'm confused about is the generality of the concept: are we really saying that there is a general algorithm or class of algorithms that will solve most or all problems to within a given distance from optimum?

Let me give an example. Depending on what test you use, an autistic can look clinically retarded, but with 'islands' of remarkable ability, even up to genius levels. The classic example is “Rain Man,” who is depicted as easily solving numerical problems most people don't even understand, but having trouble tying his shoes. This is usually an exaggeration (by no means are all autistics savants), and these island skills are hardly limited to math. The interesting point, though, is that even someone with many such islands can have an abysmally low overall IQ.

Some tests correct for this – Raven's Pattern matching test, for instance, gives you increasingly complex patterns that you have to complete – and this tends to level out those islands, and give an overall score that seems commensurate with the sheer genius that can be found in some areas.

What I find confusing is why we're correcting this at all. Certainly, we know that some people, given a task, can complete that task, and of course, depending on the person, this task can be unfathomably complex. But do we really have the evidence to say that, in general, this task does not depend on the person as well? Or, more specifically, on the algorithms they're running? Is it reasonable to say that a person runs an algorithm that will solve all problems within an efficiency x (with respect to processing time and optimality of the solution)? Or should we be looking closer for islands in neurological baselines as well?

Certainly, we could change the question and ask how efficient are all the algorithms the person is running, and from that, we could give an average efficiency, which might serve as a decent rough estimate for the efficiency with which a person will solve a problem. And for some uses, this is exactly the information we're looking for, and that's fine. But, as a general property of the people we're studying, it seems like the measure is insufficient.

If we're trying to predict specific behavior, it seems like it would be useful to be aware of whatever 'islands' exist – for instance, the common separation between algebraic and geometric approaches to math. In my experience, using geometric explanations to someone with an algebraic approach may not be at all successful, but this is not predictive of what we might think of as the person's a priori probability of solving the problem: occasionally they seem to solve the problem with no more than a few algebraic hints. Of course, this is hardly hard evidence, but I think it points to what I'm getting at.

Looking at the specific algorithm that's being used (or perhaps, the class of algorithm?) can be considerably more predictive of the outcome. Actually, I can't really say that, either: looking at what could be a distinct algorithm can be considerably more predictive of the outcome. There are numerous explanations for these observations, one of which is of course that these are all the same algorithm, just trained on different inputs, and perhaps even constrained or aided by changes in the local neural architecture (as some studies on neurological correlates of autism might suggest). But computational power alone seems insufficient if we're going to explain phenomena like the autistic 'islands'. A savant doesn't want for computational power – but in some areas, they can want for intelligence.

Here's where I start getting confused: the research I've seen assumes intelligence is a single trait which could be genetically, epigenetically, or culturally transmitted. When correlates of intelligence are looked for, from what I've seen, the correlates are for the 'average' intelligence score, and largely disregard the 'islands' of ability. As I've said, this can be useful, but it seems like answering some of these questions would be useful for a more general understanding of intelligence, especially going into the neurological side of things, whether that's in wetware or hardware.

Then again, there's a good chance I'm missing something: in which case, I'd appreciate some help updating my priors.

What always fascinates me when I meet a group of very intelligent people is the very elaborate bullshit that they believe in. The naive theory of intelligence I first posited when I was a kid was that intelligence is a tool to avoid false beliefs and find the truth. Surrounded by mediocre minds who held obviously absurd beliefs not only without the ability to coherently argue why they held these beliefs, but without the ability of even understanding basic arguments about them, I believed as a child that the vast amount of superstition and false beliefs in the world was due to people both being stupid and following the authority of insufficiently intelligent teachers and leaders. More intelligent people and people following more intelligent authorities would thus automatically hold better beliefs and avoid disproven superstitions. However, as a grown up, I got the opportunity to actually meet and mingle with a whole lot of intelligent people, including many whom I readily admit are vastly more intelligent than I am. And then I had to find that my naive theory of intelligence didn't hold water: intelligent people were just as prone as less intelligent people to believing in obviously absurd superstitions. Only their superstitions would be much more complex, elaborate, rich, and far reaching than an inferior mind's superstitions.

For instance, I remember a ride with an extremely intelligent and interesting man (RIP Bob Desmarets); he was describing his current pursuit, which struck me as a brilliant mathematical mind's version of mysticism: the difference was that instead of marveling at some trivial picture of an incarnate god like some lesser minds might have done, he was seeking some Ultimate Answer to the Universe in the branching structures of ever more complex algebras of numbers, real numbers, complex numbers, quaternions, octonions, and beyond, in ever higher dimensions (notably in relation to super-string theories). I have no doubt that there is something deep, and probably enlightening and even useful in such theories, and I readily disqualify myself as to the ability to judge the contributions that my friend made to the topic from a technical point of view; no doubt they were brilliant in one way or another. Yet, the way he was talking about this topic immediately triggered the "crackpot" flag; he was looking there for much more than could possibly be found, and anyone (like me) capable of acknowledging being too stupid to fathom the Full Glory of these number structures yet able to find some meaning in life could have told that no, this topic doesn't hold key to The Ultimate Source of All Meaning in Life. Bob's intellectual quest, as exaggeratedly exalted as it might have been, and as interesting as it was to his own exceptional mind, was on the grand scale of things but some modestly useful research venue at best, and an inoffensive pastime at worst. Perhaps Bob could conceivably used his vast intellect towards pursuits more useful to you and I; but we didn't own his mind, and we have no claims to lay on the wonders he could have created but failed to by putting his mind into one quest rather than another. First, Do No Harm. Bob didn't harm any one, and his ideas certainly contained no hint of any harm to be done to anyone.

Unhappily, that is not always the case of every intelligent man's fantasies. Let's consider a discussion I was having recently, that prompted this article. Last week, I joined a dinner-discussion with a lesswrong meetup group: radical believers in rationality and its power to improve life in general and one's own life in particular. As you can imagine, the attendance was largely, though not exclusively, composed of male computer geeks. But then again, any club that accepts me as a member will probably be biased that way: birds of the feather flock together. No doubt, there are plenty of meetup groups with the opposite bias, gathering desperately non-geeky females to the almost exclusion of males. Anyway, the theme of the dinner was "optimal philanthropy", or how to give time and money to charities in a way that maximizes the positive impact of your giving. So far, so good.

But then, I found myself in a most disturbing private side conversation with the organizer, Jeff Kaufman (a colleague, I later found out), someone I strongly suspect of being in many ways saner and more intelligent than I am. While discussing utilitarian ways of evaluating charitable action, he at some point mentioned some quite intelligent acquaintance of his who believed that morality was about minimizing the suffering of living beings; from there, that acquaintance logically concluded that wiping out all life on earth with sufficient nuclear bombs (or with grey goo) in a surprise simultaneous attack would be the best possible way to optimize the world, though one would have to make triple sure of involving enough destructive power that not one single strand of life should survive or else the suffering would go on and the destruction would have been just gratuitous suffering. We all seemed to agree that this was an absurd and criminal idea, and that we should be glad the guy, brilliant as he may be, doesn't remotely have the ability to implement his crazy scheme; we shuddered though at the idea of a future super-human AI having this ability and being convinced of such theories.

That was not the disturbing part though. What tipped me off was when Jeff, taking the "opposite" stance of "happiness maximization" to the discussed acquaintance's "suffering minimization", seriously defended the concept of wireheading as a way that happiness may be maximized in the future: putting humans into vats where the pleasure centers of their brains will be constantly stimulated, possibly using force. Or perhaps instead of humans, using rats, or ants, or some brain cell cultures or perhaps nano-electronic simulations of such electro-chemical stimulations; in the latter cases, biological humans, being less-efficient forms of happiness substrate, would be done away with or at least not renewed as embodiments of the Holy Happiness to be maximized. He even wrote at least two blog posts on this theme: hedonic vs preference utilitarianism in the Context of Wireheading, and Value of a Computational Process. In the former, he admits to some doubts, but concludes that the ways a value system grounded on happiness differ from my intuitions are problems with my intutions.

I expect that most people would, and rightfully so, find Jeff's ideas as well as his acquaintance's ideas to be ridiculous and absurd on their face; they would judge any attempt to use force to implement them as criminal, and they would consider their fantasied implemention to be the worst of possible mass murders. Of course, I also expect that most people would be incapable of arguing their case rationally against Jeff, who is much more intelligent, educated and knowledgeable in these issues than they are. And yet, though most of them would have to admit their lack of understanding and their absence of a rational response to his arguments, they'd be completely right in rejecting his conclusion and in refusing to hear his arguments, for he is indeed the sorely mistaken one, despite his vast intellectual advantages.

I wilfully defer any detailed rational refutation of Jeff's idea to some future article (can you without reading mine write a valuable one?). In this post, I rather want to address the meta-point of how to address the seemingly crazy ideas of our intellectual superiors. First, I will invoke the "conservative" principle (as I'll call it), well defended by Hayek (who is not a conservative): we must often reject the well-argued ideas of intelligent people, sometimes more intelligent than we are, sometimes without giving them a detailed hearing, and instead stand by our intuitions, traditions and secular rules, that are the stable fruit of millenia of evolution. We should not lightly reject those rules, certainly not without a clear testable understanding of why they were valid where they are known to have worked, and why they would cease to be in another context. Second, we should not hesitate to use proxy in an eristic argument: if we are to bow to the superior intellect of our better, it should not be without having pitted said presumed intellects against each other in a fair debate to find out if indeed there is a better whose superior arguments can convince the others or reveal their error. Last but not least, beyond mere conservatism or debate, mine is the Libertarian point: there is Universal Law, that everyone must respect, whereby peace between humans is possible inasmuch and only inasmuch as they don't initiate violence against other persons and their property. And as I have argued in another previous essay (hardscrapple), this generalizes to maintaining peace between sentient beings of all levels of intelligence, including any future AI that Jeff may be prone to consider. Whatever the one's prevailing or dissenting opinions, the initiation of force is never to be allowed as a means to further any ends. Rather than doubt his intuition, Jeff should have been tipped that his theory was wrong and taken out of context by the very fact that it advocates or condones massive violation of this Universal Law. Criminal urges, mass-criminal at that, are a strong stench that should alert anyone that some ideas have gone astray, even when it might not be immediately obvious where exactly they started parting from the path of sanity.

Now, you might ask, it is good and well to poke fun at the crazy ideas that some otherwise intelligent people may hold; it may even allow one to wallow in a somewhat justified sense of intellectual superiority over people who otherwise are actually and objectively so one's intellectual superiors. But is there a deeper point? Is it relevant what crazy ideas intellectuals hold, whether inoffensive or criminal? Sadly, it is. As John McCarthy put it, "Soccer riots kill at most tens. Intellectuals' ideological riots sometimes kill millions." Jeff's particular crazy idea may be mostly harmless: the criminal raptures of the overintelligent nerd, that are so elaborate as to be unfathomable to 99.9% of the population, are unlikely to ever spread to enough of the power elite to be implemented. That is, unless by some exceptional circumstance there is a short and brutal transition to power by some overfriendly AI programmed to follow such an idea. On the other hand, the criminal raptures of a majority of the more mediocre intellectual elite, when they further possess simple variants that can intoxicate the ignorant and stupid masses, are not just theoretically able to lead to mass murder, but have historically been the source of all large-scale mass murders so far; and these mass murders can be counted in hundreds of millions, over the XXth century only, just for Socialism. Nationalism, Islamism and Social-democracy (the attenuated strand of socialism that now reigns in Western "Democracies") count their victims in millions only. And every time, the most well-meaning of intellectuals build and spread the ideologies of these mass-murders. A little initial conceptual mistake, properly amplified, can do that.

And so I am reminded of the meetings of some communist cells that I attended out of curiosity when I was in high-school. Indeed, trotskyites are very openly recruiting in "good" French high-schools. It was amazing the kind of non-sensical crap that these obviously above-average adolescent could repeat. "The morale of the workers is low." Whoa. Or "The petite-bourgeoisie" is plotting this or that. Apparently, grossly cut social classes spanning millions of individuals act as one man, either afflicted with depression or making machiavelian plans. Not that any of them knew much of either salaried workers or entrepreneurs but through one-sided socialist literature. If you think that the nonsense of the intellectual elite is inoffensive, consider what happens when some of them actually act on those nonsensical beliefs: you get terrorists who kill tens of people; when they lead ignorant masses, they end up killing millions of people in extermination camps or plain massacres. And when they take control of entire universities, and train generations of scholars, who teach generations of bureaucrats, politicians, journalists, then you suddenly find that all politicians agree on slowly implementing the same totalitarian agenda, one way or another.

If you think that control of universities by left-wing ideologists is just a French thing, consider how for instance, America just elected a president whose mentor and ghostwriter was the chief of a terrorist group made of Ivy League educated intellectuals, whose overriding concern about the country they claimed to rule was how to slaughter ten percent of its population in concentration camps. And then consider that the policies of this president's "right wing" opponent are indistinguishable from the policies of said president. The violent revolution has given way to the slow replacement of the elite, towards the same totalitarian ideals, coming to you slowly but relentlessly rather than through a single mass criminal event. Welcome to a world where the crazy ideas of intelligent people are imposed by force, cunning and superior organization upon a mass of less intelligent yet less crazy people.

Ideas have consequences. That's why everyone Needs Philosophy.

Crossposted from my livejournal: http://fare.livejournal.com/168376.html

All the posts on FAI theory as of late have given me cause to think. There's something in the conversations about it that has always bugged me, but it is something that I haven't found the words for before now.

It is something like this:

Say that you manage to construct an algorithm for FAI...

Say that you can show that it isn't going to be a dangerous mistake...

And say you do all of this, and popularize it, before AGI is created (or at least, before an AGI goes *FOOM*)...

...

How in the name of Sagan are you actually going to ENFORCE the idea that all AGIs are FAIs? 

I mean, if it required some rare material (like nuclear weapons) or large laboratories (like biological wmds) or some other resource that you could at least make artificially scarce, you could set up a body that ensures that any AGI created is an FAI.

But if all it is, is the right algorithms, the right code, and enough computing power... even if you design a theory for FAI, how would you keep someone from making UFAI anyway? Between people experimenting with the principles (once known), making mistakes, and the prospect of actively malicious *humans*... it just seems like unless you somehow come up with an internal mechanism that makes FAI better and stronger than any UFAI could be, and the solution turns out to be such that any idiot could see that it was a better solution... that UFAI is going to exist at some point no matter what.

At that point, it seems like the question becomes not "How do we make FAI?" (although that might be a secondary question) but rather "How do we prevent the creation of, eliminate, or reduce potential damage from UFAI?" Now, it seems like FAI might be one thing that you do toward that goal, but if UFAI is a highly likely consequence of AGI even *with* an FAI theory, shouldn't the focus be on how to contain a UFAI event?

An interesting new article, "Cooperation and the evolution of intelligence", uses a simple one-hidden-layer neural network to study the selection for intelligence in iterated prisoners' dilemma and iterated snowdrift dilemma games.

The article claims that increased intelligence decreased cooperation in IPD, and increased cooperation in ISD.  However, if you look at figure 4 which graphs that data, you'll see that on average it decreased cooperation in both cases.  They state that it increased cooperation in ISD based on a Spearman rank test.  This test is deceptive in this case, because it ignores the magnitude of differences between datapoints, and so the datapoints on the right with a tiny but consistent increase in cooperation outweigh the datapoints on the left with large decreases in cooperation.

This suggests that intelligence is an externality, like pollution.  Something that benefits the individual at a cost to society.  They posit the evolution of intelligence as an arms race between members of the species.

ADDED: The things we consider good generally require intelligence, if we suppose (as I expect) that consciousness requires intelligence.  So it wouldn't even make sense to conclude that intelligence is bad.  Plus, intelligence itself might count as a good.

However, humans and human societies are currently near some evolutionary equilibrium.  It's very possible that individual intelligence has not evolved past its current levels because it is at an equilibrium, beyond which higher individual intelligence results in lower social utility.  In fact, if you believe SIAI's narrative about the danger of artificial intelligence and the difficulty of friendly AI, I think you would have to conclude that higher individual intelligence results in lower expected social utility, for human measures of utility.

The Power of Algorithms

Evolutionary processes are the most evident example of the power of simple algorithms [1][2][3][4][5].

The field of evolutionary biology gathered a vast amount of evidence [6] that established evolution as the process that explains the local decrease in entropy [7], the complexity of life.

Since it can be conclusively shown that all life is an effect of an evolutionary process it is implicit that everything we do not understand about living beings is also an effect of evolution.

We might not understand the nature of intelligence [8] and consciousness [9] but we do know that they are the result of an optimization process that is neither intelligent nor conscious.

Therefore we know that it is possible for an physical optimization process to culminate in the creation of more advanced processes that feature superior qualities.

One of these qualities is the human ability to observe and improve the optimization process that created us. The most obvious example being science [10].

Science can be thought of as civilization-level self-improvement method. It allows us to work together in a systematic and efficient way and accelerate the rate at which further improvements are made.

The Automation of Science

We know that optimization processes that can create improved versions of themselves are possible, even without an explicit understanding of their own workings, as exemplified by natural selection.

We know that optimization processes can lead to self-reinforcing improvements, as exemplified by the adaptation of the scientific method [11] as an improved evolutionary process and successor of natural selection.

Which raises questions about the continuation of this self-reinforcing feedback cycle and its possible implications.

One possibility is to automate science [12][13] and apply it to itself and its improvement.

But science is a tool and its bottleneck are its users. Humans, the biased [14] effect of the blind idiot god that is evolution.

Therefore the next logical step is to use science to figure out how to replace humans by a better version of themselves, artificial general intelligence.

Artificial general intelligence, that can recursively optimize itself [15], is the logical endpoint of various converging and self-reinforcing feedback cycles.

Risks from AI

Will we be able to build an artificial general intelligence? Yes, sooner or later.

Even the unintelligent, unconscious and aimless process of natural selection was capable of creating goal-oriented, intelligent and conscious agents that can think ahead, jump fitness gaps and improve upon the process that created them to engage in prediction and direct experimentation.

The question is, what are the possible implications of the invention of an artificial, fully autonomous, intelligent and goal-oriented optimization process?

One good bet is that such an agent will recursively improve its most versatile, and therefore instrumentally useful, resource. It will improve its general intelligence, respectively cross-domain optimization power.

Since it is unlikely that human intelligence is the optimum, the positive feedback effect, that is a result of using intelligence amplifications to amplify intelligence, is likely to lead to a level of intelligence that is generally more capable than the human intelligence level.

Humans are unlikely to be the most efficient thinkers because evolution is mindless and has no goals. Evolution did not actively try to create the smartest thing possible.

Evolution is further not limitlessly creative, each step of an evolutionary design must increase the fitness of its host. Which makes it probable that there are artificial mind designs that can do what no product of natural selection could accomplish, since an intelligent artificer does not rely on the incremental fitness of each step in the development process.

It is actually possible that human general intelligence is the bare minimum. Because the human level of intelligence might have been sufficient to both survive and reproduce and that therefore no further evolutionary pressure existed to select for even higher levels of general intelligence.

The implications of this possibility might be the creation of an intelligent agent that is more capable than humans in every sense. Maybe because it does directly employ superior approximations of our best formal methods, that tell us how to update based on evidence and how to choose between various actions. Or maybe it will simply think faster. It doesn’t matter.

What matters is that a superior intellect is probable and that it will be better than us at discovering knowledge and inventing new technology. Technology that will make it even more powerful and likely invincible.

And that is the problem. We might be unable to control such a superior being. Just like a group of chimpanzees is unable to stop a company from clearing its forest [16].

But even if such a being is only slightly more capable than us. We might find ourselves at its mercy nonetheless.

Human history provides us with many examples [17][18][19] that make it abundantly clear that even the slightest advance can enable one group to dominate others.

What happens is that the dominant group imposes its values on the others. Which in turn raises the question of what values an artificial general intelligence might have and the implications of those values for us.

Due to our evolutionary origins, our struggle for survival and the necessity to cooperate with other agents, we are equipped with many values and a concern for the welfare of others [20].

The information theoretic complexity [21][22] of our values is very high. Which means that it is highly unlikely for similar values to automatically arise in agents that are the product of intelligent design, agents that never underwent the million of years of competition with other agents that equipped humans with altruism and general compassion.

But that does not mean that an artificial intelligence won’t have any goals [23][24]. Just that those goals will be simple and their realization remorseless [25].

An artificial general intelligence will do whatever is implied by its initial design. And we will be helpless to stop it from achieving its goals. Goals that won’t automatically respect our values [26].

A likely implication is the total extinction of all of humanity [27].

Further Reading

• What should a reasonable person believe about the Singularity?
• The Singularity: A Philosophical Analysis
• Intelligence Explosion: Evidence and Import
• Why an Intelligence Explosion is Probable
• Artificial Intelligence as a Positive and Negative Factor in Global Risk
• From mostly harmless to civilization-threatening: pathways to dangerous artificial general intelligences
• The Hanson-Yudkowsky AI-Foom Debate
• Facing The Singularity

References

[1] Genetic Algorithms and Evolutionary Computation, talkorigins.org/faqs/genalg/genalg.html
[2] Fixing software bugs in 10 minutes or less using evolutionary computation, genetic-programming.org/hc2009/1-Forrest/Forrest-Presentation.pdf
[3] Automatically Finding Patches Using Genetic Programming, genetic-programming.org/hc2009/1-Forrest/Forrest-Paper-on-Patches.pdf
[4] A Genetic Programming Approach to Automated Software Repair, genetic-programming.org/hc2009/1-Forrest/Forrest-Paper-on-Repair.pdf
[5]GenProg: A Generic Method for Automatic Software Repair, virginia.edu/~weimer/p/weimer-tse2012-genprog.pdf
[6] 29+ Evidences for Macroevolution (The Scientific Case for Common Descent), talkorigins.org/faqs/comdesc/
[7] Thermodynamics, Evolution and Creationism, talkorigins.org/faqs/thermo.html
[8] A Collection of Definitions of Intelligence, vetta.org/documents/A-Collection-of-Definitions-of-Intelligence.pdf
[9] plato.stanford.edu/entries/consciousness/
[10] en.wikipedia.org/wiki/Science
[11] en.wikipedia.org/wiki/Scientific_method
[12] The Automation of Science, sciencemag.org/content/324/5923/85.abstract
[13] Computer Program Self-Discovers Laws of Physics, wired.com/wiredscience/2009/04/newtonai/
[14] List of cognitive biases, en.wikipedia.org/wiki/List_of_cognitive_biases
[15] Intelligence explosion, wiki.lesswrong.com/wiki/Intelligence_explosion
[16] 1% with Neil deGrasse Tyson, youtu.be/9nR9XEqrCvw
[17] Mongol military tactics and organization, en.wikipedia.org/wiki/Mongol_military_tactics_and_organization
[18] Wars of Alexander the Great, en.wikipedia.org/wiki/Wars_of_Alexander_the_Great
[19] Spanish colonization of the Americas, en.wikipedia.org/wiki/Spanish_colonization_of_the_Americas
[20] A Quantitative Test of Hamilton's Rule for the Evolution of Altruism, plosbiology.org/article/info:doi/10.1371/journal.pbio.1000615
[21] Algorithmic information theory, scholarpedia.org/article/Algorithmic_information_theory
[22] Algorithmic probability, scholarpedia.org/article/Algorithmic_probability
[23] The Nature of Self-Improving Artificial Intelligence, selfawaresystems.files.wordpress.com/2008/01/nature_of_self_improving_ai.pdf
[24] The Basic AI Drives, selfawaresystems.files.wordpress.com/2008/01/ai_drives_final.pdf
[25] Paperclip maximizer, wiki.lesswrong.com/wiki/Paperclip_maximizer
[26] Friendly artificial intelligence, wiki.lesswrong.com/wiki/Friendly_artificial_intelligence
[27] Existential Risk, existential-risk.org

This is a reply to a comment by Yvain and everyone who might have misunderstood what problem I tried to highlight.

Here is the problem. You can't estimate the probability and magnitude of the advantage an AI will have if you are using something that is as vague as the concept of 'intelligence'.

Here is a case that bears some similarity and might shed light on what I am trying to explain:

At his recent keynote speech at the New York Television Festival, former Star Trek writer and creator of the re-imagined Battlestar Galactica Ron Moore revealed the secret formula to writing for Trek.

He described how the writers would just insert "tech" into the scripts whenever they needed to resolve a story or plot line, then they'd have consultants fill in the appropriate words (aka technobabble) later.

"It became the solution to so many plot lines and so many stories," Moore said. "It was so mechanical that we had science consultants who would just come up with the words for us and we'd just write 'tech' in the script. You know, Picard would say 'Commander La Forge, tech the tech to the warp drive.' I'm serious. If you look at those scripts, you'll see that."

Moore then went on to describe how a typical script might read before the science consultants did their thing:

La Forge: "Captain, the tech is overteching."

Picard: "Well, route the auxiliary tech to the tech, Mr. La Forge."

La Forge: "No, Captain. Captain, I've tried to tech the tech, and it won't work."

Picard: "Well, then we're doomed."

"And then Data pops up and says, 'Captain, there is a theory that if you tech the other tech ... '" Moore said. "It's a rhythm and it's a structure, and the words are meaningless. It's not about anything except just sort of going through this dance of how they tech their way out of it."

The use of 'intelligence' is as misleading and dishonest in evaluating risks from AI as the use of 'tech' in Star Trek.

It is true that 'intelligence', just as 'technology' has some explanatory power. Just like 'emergence' has some explanatory power. As in "the morality of an act is an emergent phenomena of a physical system: it refers to the physical relations among the components of that system". But it does not help to evaluate the morality of an act or in predicting if a given physical system will exhibit moral properties.

The failures of phlogiston and vitalism are historical hindsight. Dare I step out on a limb, and name some current theory which I deem analogously flawed?

I name artificial intelligence or thinking machines - usually defined as the study of systems whose high-level behaviors arise from "thinking" or the interaction of many low-level elements.  (R. J. Sternberg quoted in a paper by Shane Legg:  “Viewed narrowly, there seem to be almost as many definitions of intelligence as there were experts asked to define it.”) Taken literally, that allows for infinitely many degrees of intelligence to fit every phenomenon in our universe above the level of individual quarks, which is part of the problem.  Imagine pointing to a chess computer and saying "It's not a stone!"  Does that feel like an explanation?  No?  Then neither should saying "It's a thinking machine!"

It's the noun "intelligence" that I protest, rather than to "evoke a dynamic state sequence from a machine by computing an algorithm".  There's nothing wrong with saying "X computes algorithm Y", where Y is some specific, detailed flowchart that represents an algorithm or process.  "Thinking about" is another legitimate phrase that means exactly the same thing:  The machine is thinking about a problem, according to an specific algorithm. The machine is thinking about how to put elements of a list in a certain order, according to the a specific algorithm called quicksort.

Now suppose I should say that a problem is explained by "thinking" or that the order of elements in a list is the result of a "thinking machine", and claim that as my explanation.

The phrase "evoke a dynamic state sequence from a machine by computing an algorithm" is acceptable, just like "thinking about" or "is caused by" are acceptable, if the phrase precedes some specification to be judged on its own merits.

However, this is not the way "intelligence" is commonly used. "Intelligence" is commonly used as an explanation in its own right.

I have lost track of how many times I have heard people say, "an artificial general intelligence would have a genuine intelligence advantage" as if that explained its advantage. This usage fits all the checklist items for a mysterious answer to a mysterious question. What do you know, after you have said that its "advantage" is "intelligence"?  You can make no new predictions.  You do not know anything about the behavior of real-world artificial general intelligence that you did not know before.  It feels like you believe a new fact, but you don't anticipate any different outcomes. Your curiosity feels sated, but it has not been fed.  The hypothesis has no moving parts - there's no detailed internal model to manipulate.  Those who proffer the hypothesis of "intelligence" confess their ignorance of the internals, and take pride in it; they contrast the science of "artificial general intelligence" to other sciences merely mundane.

And even after the answer of "How? Intelligence!" is given, the practical realization is still a mystery and possesses the same sacred impenetrability it had at the start.

A fun exercise is to eliminate the explanation "intelligence" from any sentence in which it appears, and see if the sentence says anything different:

• Before: The AI is going to take over the world by using its superhuman intelligence to invent nanotechnology.
• After:  The AI is going to take over the world by inventing nanotechnology.

• Before: A friendly AI is going to use its superhuman intelligence to extrapolate the coherent volition of humanity.
• After:  A friendly AI is going to extrapolate the coherent volition of humanity.
• Even better: A friendly AI is a powerful algorithm. We can successfully extrapolate some aspects of the volition of individual humans using [FILL IN DETAILS] procedure, without any global societal variables, showing that we understand how the extrapolate the volition of humanity in theory and that it converges rather than diverges, that our wishes cohere rather than interfere.

Another fun exercise is to replace "intelligence" with "magic", the explanation that people had to use before the idea of an intelligence explosion was invented:

• Before:  The AI is going to use its superior intelligence to quickly evolve vastly superhuman capabilities and reach singleton status within a matter of weeks.
• After:  The AI is going to use magic to quickly evolve vastly superhuman capabilities and reach singleton status within a matter of weeks.

• Before:  Superhuman intelligence is able to use the internet to gain physical manipulators and expand its computational capabilities.
• After:  Superhuman magic is able to use the internet to gain physical manipulators and expand its computational capabilities.

Does not each statement convey exactly the same amount of knowledge about the phenomenon's behavior? Does not each hypothesis fit exactly the same set of outcomes?

"Intelligence" has become very popular, just as saying "magic" used to be very popular. "Intelligence" has the same deep appeal to human psychology, for the same reason. "Intelligence" is such a wonderfully easy explanation, and it feels good to say it; it gives you a sacred mystery to worship. Intelligence is popular because it is the junk food of curiosity. You can explain anything using intelligence , and so people do just that; for it feels so wonderful to explain things. Humans are still humans, even if they've taken a few science classes in college. Once they find a way to escape the shackles of settled science, they get up to the same shenanigans as their ancestors, dressed up in the literary genre of "science" but still the same species psychology.

I thought many of you would be interested to know that the following paper just appeared in Journal of Consciousness Studies:

"Can Intelligence Explode?", by Marcus Hutter. (LINK HERE)

Abstract: The technological singularity refers to a hypothetical scenario in which technological advances virtually explode. The most popular scenario is the creation of super-intelligent algorithms that recursively create ever higher intelligences. It took many decades for these ideas to spread from science fiction to popular science magazines and finally to attract the attention of serious philosophers. David Chalmers' (JCS 2010) article is the first comprehensive philosophical analysis of the singularity in a respected philosophy journal. The motivation of my article is to augment Chalmers' and to discuss some issues not addressed by him, in particular what it could mean for intelligence to explode. In this course, I will (have to) provide a more careful treatment of what intelligence actually is, separate speed from intelligence explosion, compare what super-intelligent participants and classical human observers might experience and do, discuss immediate implications for the diversity and value of life, consider possible bounds on intelligence, and contemplate intelligences right at the singularity.

I have only just seen the paper and have not yet thread through it myself, but I thought we could use this thread for discussion.