CFAR’s new focus, and AI Safety

AnnaSalamon

51 CFAR’s new focus, and AI Safety

by AnnaSalamon

3rd Dec 2016

4 min read

51

A bit about our last few months:

We’ve been working on getting a simple clear mission and an organization that actually works. We think of our goal as analogous to the transition that the old Singularity Institute underwent under Lukeprog (during which chaos was replaced by a simple, intelligible structure that made it easier to turn effort into forward motion).
As part of that, we’ll need to find a way to be intelligible.
This is the first of several blog posts aimed at causing our new form to be visible from outside. (If you're in the Bay Area, you can also come meet us at tonight's open house.) (We'll be talking more about the causes of this mission-change; the extent to which it is in fact a change, etc. in an upcoming post.)

Here's a short explanation of our new mission:

We care a lot about AI Safety efforts in particular, and about otherwise increasing the odds that humanity reaches the stars.
Also, we[1] believe such efforts are bottlenecked more by our collective epistemology, than by the number of people who verbally endorse or act on "AI Safety", or any other "spreadable viewpoint" disconnected from its derivation.
Our aim is therefore to find ways of improving both individual thinking skill, and the modes of thinking and social fabric that allow people to think together. And to do this among the relatively small sets of people tackling existential risk.

To elaborate a little:

Existential wins and AI safety

By an “existential win”, we mean humanity creates a stable, positive future. We care a heck of a lot about this one.

Our working model here accords roughly with the model in Nick Bostrom’s book Superintelligence. In particular, we believe that if general artificial intelligence is at some point invented, it will be an enormously big deal.

(Lately, AI Safety is being discussed by everyone from The Economist to Newsweek to Obama to an open letter from eight thousand. But we’ve been thinking on this, and backchaining partly from it, since before that.)

Who we’re focusing on, why

Our preliminary investigations agree with The Onion’s; despite some looking, we have found no ultra-competent group of people behind the scenes who have fully got things covered.

What we have found are:

AI and machine learning graduate students, researchers, project-managers, etc. who care; who can think; and who are interested in thinking better;
Students and others affiliated with the “Effective Altruism” movement, who are looking to direct their careers in ways that can do the most good;
Rationality geeks, who are interested in seriously working to understand how the heck thinking works when it works, and how to make it work even in domains as confusing as AI safety.

These folks, we suspect, are the ones who can give humanity the most boost in its survival-odds per dollar of CFAR’s present efforts (which is a statement partly about us, but so it goes). We’ve been focusing on them.

(For the sake of everyone. Would you rather: (a) have bad rationality skills yourself; or (b) be killed by a scientist or policy-maker who also had bad rationality skills?)

Brier-boosting, not Signal-boosting

Everyone thinks they’re right. We do, too. So we have some temptation to take our own favorite current models of AI Safety strategy and to try to get everyone else to shut up about their models and believe ours instead.

This understandably popular activity is often called “signal boosting”, “raising awareness”, or doing “outreach”.

At CFAR, though, we force ourselves not to do “signal boosting” in this way. Our strategy is to spread general-purpose thinking skills, not our current opinions. It is important that we get the truth-seeking skills themselves to snowball across relevant players, because ultimately, creating a safe AI (or otherwise securing an existential win) is a research problem. Nobody, today, has copyable opinions that will get us there.

We like to call this “Brier boosting”, because a “Brier score” is a measure of predictive accuracy.

(song by CFAR alum Ray Arnold)

[1] By "We believe X", we do not mean to assert that every CFAR staff member individually believes X. (Similarly for "We care about Y). We mean rather that CFAR as an organization is planning/acting as though X is true. (Much as if CFAR promises you a rationality T-shirt, that isn't an individual promise from each of the individuals at CFAR; it is rather a promise from the organization as such.)

If we're going to build an art of rationality, we'll need to figure out how to create an organization where people can individually believe whatever the heck they end up actually believing as they chase the evidence, while also having the organization qua organization be predictable/intelligible.

ETA:

You may also want to check out two documents we posted in the days since this post:

Further discussion of CFAR’s focus on AI safety, and the good things folks wanted from “cause neutrality”
CFAR's mission statement (link post, linking to our website).

Center for Applied Rationality (CFAR)Project Announcement

Frontpage

51

New Comment

Rendering 0/88 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 10:00 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Moderation Log

51 CFAR’s new focus, and AI Safety

by AnnaSalamon

3rd Dec 2016

4 min read

51

A bit about our last few months:

We’ve been working on getting a simple clear mission and an organization that actually works. We think of our goal as analogous to the transition that the old Singularity Institute underwent under Lukeprog (during which chaos was replaced by a simple, intelligible structure that made it easier to turn effort into forward motion).
As part of that, we’ll need to find a way to be intelligible.
This is the first of several blog posts aimed at causing our new form to be visible from outside. (If you're in the Bay Area, you can also come meet us at tonight's open house.) (We'll be talking more about the causes of this mission-change; the extent to which it is in fact a change, etc. in an upcoming post.)

Here's a short explanation of our new mission:

We care a lot about AI Safety efforts in particular, and about otherwise increasing the odds that humanity reaches the stars.
Also, we[1] believe such efforts are bottlenecked more by our collective epistemology, than by the number of people who verbally endorse or act on "AI Safety", or any other "spreadable viewpoint" disconnected from its derivation.
Our aim is therefore to find ways of improving both individual thinking skill, and the modes of thinking and social fabric that allow people to think together. And to do this among the relatively small sets of people tackling existential risk.

To elaborate a little:

Existential wins and AI safety

By an “existential win”, we mean humanity creates a stable, positive future. We care a heck of a lot about this one.

Who we’re focusing on, why

Our preliminary investigations agree with The Onion’s; despite some looking, we have found no ultra-competent group of people behind the scenes who have fully got things covered.

What we have found are:

AI and machine learning graduate students, researchers, project-managers, etc. who care; who can think; and who are interested in thinking better;
Students and others affiliated with the “Effective Altruism” movement, who are looking to direct their careers in ways that can do the most good;
Rationality geeks, who are interested in seriously working to understand how the heck thinking works when it works, and how to make it work even in domains as confusing as AI safety.

(For the sake of everyone. Would you rather: (a) have bad rationality skills yourself; or (b) be killed by a scientist or policy-maker who also had bad rationality skills?)

Brier-boosting, not Signal-boosting

This understandably popular activity is often called “signal boosting”, “raising awareness”, or doing “outreach”.

We like to call this “Brier boosting”, because a “Brier score” is a measure of predictive accuracy.

(song by CFAR alum Ray Arnold)

ETA:

You may also want to check out two documents we posted in the days since this post:

Further discussion of CFAR’s focus on AI safety, and the good things folks wanted from “cause neutrality”
CFAR's mission statement (link post, linking to our website).

Center for Applied Rationality (CFAR)Project Announcement

Frontpage

51

Mentioned in

92The Craft & The Community - A Post-Mortem & Resurrection

65What exactly is the "Rationality Community?"

65Further discussion of CFAR’s focus on AI safety, and the good things folks wanted from “cause neutrality”

20The Story CFAR

17Extreme Rationality: Still Not That Great

New Comment

Rendering 0/88 comments, sorted by

top scoring

(show more) Click to highlight new comments since: Today at 10:00 PM

Some comments are truncated due to high volume. (⌘F to expand all)Change truncation settings

Moderation Log

More from AnnaSalamon

Curated and popular this week

88Comments

Comment Permalink

John_Maxwell10y270

There's a difference between optimizing for truth and optimizing for interestingness. Interestingness is valuable for truth in the long run because the more hypotheses you have, the better your odds of stumbling on the correct hypothesis. But naively optimizing for truth can decrease creativity, which is critical for interestingness.

I suspect "having ideas" is a skill you can develop, kind of like making clay pots. In the same way your first clay pots will be lousy, your first ideas will be lousy, but they will get better with practice.

...creation is embarrassing. For every new good idea you have, there are a hundred, ten thousand foolish ones, which you naturally do not care to display.

Source.

If this is correct, this also gives us clues about how to solve Less Wrong's content problem.

Online communities do not have a strong comparative advantage in compiling and presenting facts that are well understood. That's the sort of thing academics and journalists are already paid to do. If online communities have a comparative advantage, it's in exploring ideas that are neglected by the mainstream--things like AI risk, or CFARish techniques for being more effective.

Unfortunately, LW's culture has historically been pretty antithetical to creativity. It's hard to tell in advance whether an idea you have is a good one or not. And LW has often been hard on posts it considers bad. This made the already-scary process of sharing new ideas even more fraught with the possibility of embarrassment.

If a single individual present is unsympathetic to the foolishness that would be bound to go on at such a [brainstorming] session, the others would freeze. The unsympathetic individual may be a gold mine of information, but the harm he does will more than compensate for that. It seems necessary to me, then, that all people at a session be willing to sound foolish and listen to others sound foolish.

Same source.

I recommend recording ideas in a private notebook. I've been doing this for a few years, and I now have way more ideas than I know what to do with.

Elon Musk

Relevant: http://waitbutwhy.com/2015/11/the-cook-and-the-chef-musks-secret-sauce.html

Lumifer10y20

There's a difference between optimizing for truth and optimizing for interestingness.

Oh yes. For example, Physical Review Letters is mostly interested in the former, while HuffPo -- in the latter.

the more hypotheses you have, the better your odds of stumbling on the correct hypothesis

That's not true because you must also evaluate all these hypotheses and that's costly. For a trivial example, given a question X, would you find it easier to identify a correct hypothesis if I presented you with five candidates or with five million candidates?

I suspec

... (read more)

0plethora10y

But academics write for other academics, and journalists don't and can't. (They've tried. They can't. Remember Vox?) AFAIK, there isn't a good outlet for compilations of facts intended for and easily accessible by a general audience, reviews of books that weren't just written, etc. Since LW isn't run for profit and is run as outreach for, among other things, CFAR, whose target demographic would be interested in such an outlet, this could be a valuable direction for either LW or a spinoff site; but, given the reputational risk (both personally and institutionally) inherent in the process of generating new ideas, we may be better served by pivoting LW toward the niche I'm thinking of -- a cross between a review journal, SSC, and, I don't know, maybe CIA (think World Factbook) or RAND -- and moving the generation and refinement of ideas into a separate container, maybe an anonymous blog or forum.

See in context