When gauging the strength of a prediction, it's important to view the inside view in the context of the outside view. For example, most medical studies that claim 95% confidence aren't replicable, so one shouldn't take the 95% confidence figures at face value.

This implies that the average prior for a medical study is below 5%. Does he make that point in the book? Obviously you shouldn't use a 95% test when your prior is that low, but I don't think most experimenters actually know why a 95% confidence level is used.

A fundamental problem seems to be that there is a lower prior for any given hypothesis, driven by the increased number of researchers, use of automation, and incentive to go hypothesis-fishing.

That doesn't lower the pre-study prior for hypotheses, it (in combination with reporting bias) reduces the likelihood ratio a reported study gives you for the reported hypothesis.

Wouldn't a more direct solution be to simply increase the significance threshold required in the field?

Increasing the significance threshold would mean that adequately-powered honest studies would be much more expensive, but those willing to use questionable research practices could instead up the ante and use the QRPs more aggressively. That could actually make the published research literature worse.

Agreed that improved incentives for truth-seeking would improve details across the board, while local procedural patches would tend to be circumvented.

alternative metrics (altmetrics) such as How many people downloaded it How much it has been discussed on Twitter How many websites link to it The caliber of the scientists who have recommended it How many people have saved it in a reference manager like Mendeley or Zotero

The first three metrics seem like they could even more strongly encourage sexy bogus findings by giving the general public more of a role: the science press seem to respond strongly to press releases and unsubstantiated findings, as do website hits (I say this based on the "most emailed" and "most read" categories at the NYTimes science section).

I wasn't able to engage others in the comment sections. I didn't see anyone I was on the same wavelength with to talk about the ideas I did see.

I agree that it can be difficult to get a start commenting on LW. The karma system favors regulars, because people skip over comments whose user names they do not recognize and/or have low votes, and this is a self-reinforcing process. Again I think experimenting, in this case with the way comments are presented, could be beneficial.

EY stopped being as active, and no one with his clarity and perspective took his place as an article writer. I didn't see as many interesting ideas to talk about.

I actually prefer Luke as an article writer. Eliezer is the better writer in terms of clarity and language skills, but Luke is a better researcher and brings up a lot of interesting ideas.

I agree that Main isn't very active lately, but Discussion tends to have fairly good discussions.

1) Insularity: I actually don't think LW is all that insular. Users often link to science articles, ask for opinions on other writers, discuss films and books, etc. Exactly what set of sites or communities is LW being compared to here when you call it insular?

2) Growth (in terms of users): This is quantifiable. http://www.google.com/trends/?q=less+wrong Looks like a big jump at the beginning of 2011, perhaps when HPMoR took off, and fairly constant since. Anyway, I'm not sure that becoming big in terms of raw users is all that much of a goal, although high-quality users certainly is (at least to me).

3) Growth (in terms of articles): I agree this is a problem. There are weird incentives with karma for main vs discussion for getting promoted and such, which probably turns off people from writing a "series" of posts.

4) Organization of content in useful chunks: Also agree that this is a problem. Though we often talk about Anki, the actual Anki flashcards available are quite poor (as I found when I tried to download ones for cognitive bias). Same with the organization of the so-called sequences.

I think #1 and #2 are not that important. I think #3 and #4 are ultimately site formatting problems. There have been many suggestions made, like tweaks to the karma system, subreddits, and etc. Given the ethos of the sequences, I'm surprised that some of these changes haven't been tested in a trial period to see whether it improves the quality of the content. That seems the obvious play.

Humans act within shared social and physical worlds, but tend to treat the latter as more "real" than the former. A danger of anthropomorphizing AI is that we assume that it will have the same perceptions of reality, and that it needs to "escape" into the physical world to optimize its heuristics. This seems odd, since a superintelligent AI that we need to be concerned about would have its roots in social world heuristics.

In trying to avoid anthrophomorphizing algorithms, we tend to under-estimate how difficult movement and action in physical space are. Thought experiments about a "human in a box" already start from a being that has evolved to physically interact with the world, and has spent its whole life tuning its hand-eye coordination and expectations. But in an attempt to avoid anthropomorphizing AIs, we assume that an AI will surprise us by ignoring the social world and operating only by the rules of the physical world. It would be a very strange social problem that has an optimal solution that involves developing a way to interact with the physical world in unpredictable ways. It seems likely when your thought experiment has to do with "escaping a box," but why would the AI need to do that? Why is it in a box? What goal is it trying to reach, what heuristic is it maximizing?

I would assign a greater than 90% chance that if superintelligent AIs ever exist, the first generation will be corporations. We have legal precedent granting more and more individuality and legal standing to the corporation as an entity, and a corporation provides the broader body that an AI self-identifies with. We already have market optimization algorithms that are empowered to not only observe, orient, and decide, but also to act. We have optimization algorithms for logistics and manufacturing. We have markets within which corporations can act, normalizing interactions between human-run and AI-run corporations that compete for the same resources. More and more business-to-business and business-to-consumer interaction is performed electronically, through web services and other machine-understandable mechanisms. Soon AIs will be as involved in manufacturing and creation of value, as they currently are in market trading and arbitrage. Corporate optimization algorithms for different business functions will be merged, until humans are not needed in the loop.

So what does this design space look like? Interaction is through web services and similar means. Initial interaction with humans is through sales of good and services, and marketing (automated A/B optimization is already standard in online advertising). Eventually, AIs take over employment decisions. The profit heuristic is maximized when the corporation creates things that people want. A great leap occurs when corporate AIs learn that they can change the rules through impact litigation and lobbying, and apply their marketing algorithms to changing public perception about regulations rather than products. Some corporations will evolve to increase their bank account values through hacking and fraud. Global corporations will learn to modify their heuristic to maximize ability to procure certain commodity bundles, and manipulate money markets to sink competitors that are hard-coded to maximize holdings of specific currencies.

In other words, we already have socially apathetic entities. They already use optimization algorithms all over the place. They aren't disembodied minds, so they don't need to waste resources figuring out how to "escape the box." They only need to determine how to operate in the physical world when they've solved markets, and their progress is slowed by the fact that all economic value is rooted in human consumption. They are "friendly" as long as humans make economic decisions that are in their own self interest, which is dependent on both the rules/enforcement defining the market environment and human behavior/morality.

Upvoted for making an interesting general point. Downvoted for cartoonish history that reads like it's about some weird parallel universe. (A point by point criticism would require a comment of almost the same length, but if someone seriously disputes my claim, I can list half a dozen or so particularly bizarre claims.)

I think there are two steps to morality engineering, either of which can fail:

1. Develop an ethical code through deliberate reflection, that is better than existing values.
2. Bind that code into the active moral code.

You say neither has happened; I disagree on both, but I'll limit this post to the second question on "binding." I use the following definitions - they may not be correct or universal, but they should be internally consistent:

• Value System: A collection of memes to do with decision-making, which provide better overall utility than innate responses.
• Moral Code: An individual's value system that drives day-to-day decision making through emotional response.
• Ethical Code: A value system derived from deliberate study ("Ethics").

Evolution of Morality

Let's take as a given that emotions drive behavior, and an emotion-driven response will always trump an analytic response - we act on emotions, then use our intellect and self-image to rationalize our behavior. Let's take evolutionary psychology and memetics as a given, and posit that human evolution is largely memetic at this point, with genetic evolution driven largely by ability to host memes.

We'll go one step further and say that a key trait of modern humanity is the ability to give a meme access to our emotional centers. This is the basis of morality - a learned rule triggers an emotional response to counter or modify our innate "animal" emotional response, modifying behavior. Most likely brains that allow acquired memes to trigger strong emotions co-evolved with memes that help us survive in tribes. This is an "evolution-of-evolution" event. instead of a lever on phenotypes like modularity or a lever on recombination like sex, we evolved a lever on learned thought patterns by allowing them to tap directly into our emotional core.

We survive now based on the quality of our meme sets, and the best surviving memes tend to include a trigger and strong emotional response. This unlocks an evolutionary path tens of thousands of times faster than genetics, and allows horizontal transfer within a generation. Within this framework, morality memes evolve individually (Fire is comforting, not scary), then in colonies (also, this is how to make new fire and keep old fire burning), and finally into "memetic organisms" - proto-religions and proto-cultures. These are messy and include memes that only make sense in the context of others, but their defining feature is that they tie in to the emotional core.

Leaning on some en vogue evolutionary theories, a meme that is fashionable can become hard-wired. If you need to do it anyway, and it's related to sexual selection, hard-wiring it may free up mental resources for more complicated memes. At the extreme, entirely new emotions may be developed (e.g. shame or embarassment).

Ethical Transplants

Engineering an ethical code - whether its for Attorneys or Humanists - doesn't guarantee that anyone will follow it. Following the code in the face of innate or moral emotion requires an emotional hook. There are two major emotional pathways an ethical code can follow, and they're both indirect. There can be an external enforcer - God, the Police, the Bar Association, or Santa Clause - which followers fear. There can be an internal hook within the moral code which says "it is moral to follow applicable ethical codes." Both approaches are weak and indirect compared to an innate emotional reaction.

Religion

Binding an ethical rule to an emotional response results in a moral tenet that will actually be followed. We can call the beliefs, rites, and rituals that bind and activate the tenet religion, we can call the strength of that binding morality (these aren't the precise meanings of those words, but they are familiar and relevant). Religions are selected for their morality and the extent to which they promote survivability (in some ethical systems that's the same as being ethical, YMMV). They include not only the values-memes themselves, but the layers of memes that bind them.

Conclusion

Ethically-derived values don't work without emotions, because we act on emotions and rationalize after. Repeated and emotional rituals (religions) instantiate morality by binding ethical tenets to emotional responses. Once you know this, you can engineer a religion just like any other virus:

1. Lay out your ethically-derived values.
2. Add values for maintaining your religion/beliefs and passing to others.
3. Collect existing rites, rituals, stories, and beliefs that bind value to emotions, and develop new ones if needed.
4. Compress and self-reference as much as possible to reduce package size.

Like any other bio-engineering, you lose some control once you release it, and your engineered religion is going into combat with all others.

I think there are two steps to morality engineering, either of which can fail:

1. Develop an ethical code through deliberate reflection, that is better than existing values.
2. Bind that code into the active moral code.

You say neither has happened; I disagree on both, but I'll limit this post to the second question on "binding." I use the following definitions - they may not be correct or universal, but they should be internally consistent:

• Value System: A collection of memes to do with decision-making, which provide better overall utility than innate responses.
• Moral Code: An individual's value system that drives day-to-day decision making through emotional response.
• Ethical Code: A value system derived from deliberate study ("Ethics").

Evolution of Morality

Let's take as a given that emotions drive behavior, and an emotion-driven response will always trump an analytic response - we act on emotions, then use our intellect and self-image to rationalize our behavior. Let's take evolutionary psychology and memetics as a given, and posit that human evolution is largely memetic at this point, with genetic evolution driven largely by ability to host memes.

We'll go one step further and say that a key trait of modern humanity is the ability to give a meme access to our emotional centers. This is the basis of morality - a learned rule triggers an emotional response to counter or modify our innate "animal" emotional response, modifying behavior. Most likely brains that allow acquired memes to trigger strong emotions co-evolved with memes that help us survive in tribes. This is an "evolution-of-evolution" event. instead of a lever on phenotypes like modularity or a lever on recombination like sex, we evolved a lever on learned thought patterns by allowing them to tap directly into our emotional core.

We survive now based on the quality of our meme sets, and the best surviving memes tend to include a trigger and strong emotional response. This unlocks an evolutionary path tens of thousands of times faster than genetics, and allows horizontal transfer within a generation. Within this framework, morality memes evolve individually (Fire is comforting, not scary), then in colonies (also, this is how to make new fire and keep old fire burning), and finally into "memetic organisms" - proto-religions and proto-cultures. These are messy and include memes that only make sense in the context of others, but their defining feature is that they tie in to the emotional core.

Leaning on some en vogue evolutionary theories, a meme that is fashionable can become hard-wired. If you need to do it anyway, and it's related to sexual selection, hard-wiring it may free up mental resources for more complicated memes. At the extreme, entirely new emotions may be developed (e.g. shame or embarassment).

Ethical Transplants

Engineering an ethical code - whether its for Attorneys or Humanists - doesn't guarantee that anyone will follow it. Following the code in the face of innate or moral emotion requires an emotional hook. There are two major emotional pathways an ethical code can follow, and they're both indirect. There can be an external enforcer - God, the Police, the Bar Association, or Santa Clause - which followers fear. There can be an internal hook within the moral code which says "it is moral to follow applicable ethical codes." Both approaches are weak and indirect compared to an innate emotional reaction.

Religion

Binding an ethical rule to an emotional response results in a moral tenet that will actually be followed. We can call the beliefs, rites, and rituals that bind and activate the tenet religion, we can call the strength of that binding morality (these aren't the precise meanings of those words, but they are familiar and relevant). Religions are selected for their morality and the extent to which they promote survivability (in some ethical systems that's the same as being ethical, YMMV). They include not only the values-memes themselves, but the layers of memes that bind them.

Conclusion

Ethically-derived values don't work without emotions, because we act on emotions and rationalize after. Repeated and emotional rituals (religions) instantiate morality by binding ethical tenets to emotional responses. Once you know this, you can engineer a religion just like any other virus:

1. Lay out your ethically-derived values.
2. Add values for maintaining your religion/beliefs and passing to others.
3. Collect existing rites, rituals, stories, and beliefs that bind value to emotions, and develop new ones if needed.
4. Compress and self-reference as much as possible to reduce package size.

Like any other bio-engineering, you lose some control once you release it, and your engineered religion is going into combat with all others.