Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

[link] MIRI's 2015 in review

9 Kaj_Sotala 03 August 2016 12:03PM

https://intelligence.org/2016/07/29/2015-in-review/

The introduction:

As Luke had done in years past (see 2013 in review and 2014 in review), I (Malo) wanted to take some time to review our activities from last year. In the coming weeks Nate will provide a big-picture strategy update. Here, I’ll take a look back at 2015, focusing on our research progress, academic and general outreach, fundraising, and other activities.

After seeing signs in 2014 that interest in AI safety issues was on the rise, we made plans to grow our research team. Fueled by the response to Bostrom’s Superintelligence and the Future of Life Institute’s “Future of AI” conference, interest continued to grow in 2015. This suggested that we could afford to accelerate our plans, but it wasn’t clear how quickly.

In 2015 we did not release a mid-year strategic plan, as Luke did in 2014. Instead, we laid out various conditional strategies dependent on how much funding we raised during our 2015 Summer Fundraiser. The response was great; we had our most successful fundraiser to date. We hit our first two funding targets (and then some), and set out on an accelerated 2015/2016 growth plan.

As a result, 2015 was a big year for MIRI. After publishing our technical agenda at the start of the year, we made progress on many of the open problems it outlined, doubled the size of our core research team, strengthened our connections with industry groups and academics, and raised enough funds to maintain our growth trajectory. We’re very grateful to all our supporters, without whom this progress wouldn’t have been possible.

Comment author: qmotus 21 July 2016 05:14:13PM 0 points [-]

Will your results ultimately take the form of blog posts such as those, or peer-reviewed publications, or something else?

I think FRI's research agenda is interesting and that they may very well work on important questions that hardly anyone else does, but I haven't yet supported them as I'm not certain about their ability to deliver actual results or the impact of their research, and find it a tad bit odd that it's supported by effective altruism organizations, since I don't see any demonstration of effectiveness so far. (No offence though, it looks promising.)

Comment author: Kaj_Sotala 22 July 2016 07:24:13AM 0 points [-]

The final output of this project will be a long article, either on FRI's website or a peer-reviewed publication or both; we haven't decided on that yet.

Comment author: rmoehn 20 July 2016 06:52:40AM 1 point [-]

Thanks! I hadn't come across the Foundational Research Institute yet.

Though, hmm, not plenty of experience? If there's talk about PhDs as an advantage, it sounds to me like they're looking for people with PhD-level experience. I'm far from that. But unless you say »oh well then maybe not«, I'll apply. Who knows what will come out of it.

Comment author: Kaj_Sotala 21 July 2016 05:58:13AM 2 points [-]

I don't have a PhD either, and I know of at least one other person who'd been discussing working for them who was also very far from having that level of experience.

Comment author: Kaj_Sotala 19 July 2016 12:19:39PM *  5 points [-]

I'm currently working on an AI strategy project for the Foundational Research Institute; they are hiring and do not require plenty of experience:

Requirements

  • Language requirement is research proficiency in English.
  • We anticipate that an applicant is dedicated to alleviating and preventing suffering, and considers it the top global priority.
  • A successful applicant will probably have a background in quantitative topics such as game theory, decision theory, computer science, physics, or math. But we welcome applicants regardless of background.
  • Peer-reviewed publications or a track record of completed comparable research output is not required, but a plus.
  • There is no degree requirement, although a PhD is an advantage, all else equal.

Their open research questions include a number of AI-related ones, and I expect many of them to still have plenty of low-hanging fruit. I'm working on getting a better handle on hard takeoff scnearios in general; most of the my results so far can be found on my website under the "fri-funded" tag. (Haven't posted anything new in a while, because I'm working on a larger article that's been taking some time.)

Comment author: Kaj_Sotala 03 July 2016 07:46:55AM 0 points [-]

I thought Mark Manson explained it well:

I want you to think of vulnerability in a more broad way. Not just emotional vulnerability (although we’ll get to that), but physical vulnerability, social vulnerability.

For instance, making yourself vulnerable doesn’t just mean being willing to share your fears or insecurities. It can mean putting yourself in a position where you can be rejected, saying a joke that may not be funny, asserting an opinion that may offend others, joining a table of people you don’t know, telling a woman that you like her and want to date her. All of these things require you to stick your neck out on the line emotionally in some way. You’re making yourself vulnerable when you do them.

In this way, vulnerability represents a form of power, a deep and subtle form of power. A man who’s able to make himself vulnerable is saying to the world, “I don’t care what you think of me; this is who I am, and I refuse to be anyone else.” He’s saying he’s not needy and that he’s high status.

Most people think of a man who’s vulnerable as a man who cowers in the corner and begs others to accept him or not hurt him. This is not vulnerability, this is weakness and neediness.

Think of it this way, there are two men. One stands tall, looks straight ahead. Looks people in the eye when he speaks to them. Says what he thinks and is unconcerned with what others think of him. When he makes a mistake, he shrugs it off and maybe apologizes. When he sucks at something, he admits it. He’s unafraid to express his emotions, even if that means he gets rejected. He has no problem moving on to people who don’t reject him, but like him for who he is.

Now, the second man hunches over, eyes dart around and is unable to look someone in the eye without getting uncomfortable. He puts on a cool persona that is always aloof. He avoids saying things that may upset others, and sometimes even lies to avoid conflict. He’s always trying to impress people. When he makes a mistake, he tries to blame others or pretend like it didn’t happen. He hides his emotions and will smile and tell everyone he’s fine even when he’s not. He’s scared to death of rejection. And when he is rejected, it sends him reeling, angry, and desperate to find a way to win back the affection of the person who doesn’t like him.

Which one of these two men is more powerful? Which one is more vulnerable? Which one is more comfortable with himself? Which one do you think women would be more attracted to?

Going back to the evolutionary perspectives we discussed in Chapter 1, vulnerability makes perfect sense as an indicator to women of a male’s status and fitness. Let’s say there’s a tribe of 20 men, all hunter gatherers, all men with more or less equal possessions (or lack thereof).

Some of the men in the tribe are constantly reactive to what the other men tell them. They don’t admit faults. They change their behavior and what they say to win the approval of the other men. When something doesn’t go their way, they look to blame someone else. What would this say about their status in their tribe? If they’re basing all of their behavior on the approval of the other men and are constantly covering for their weaknesses, it says that they’re low status, not trustworthy, needy, and probably not going to be a dependable father.

Now imagine other men in the same tribe who are unfazed by the neediness or temper tantrums of the other men around them. They focus purely on their task at hand and don’t change their behavior based on what others think of them. When challenged, they stand up for themselves, but when wrong they also admit their fault, as they see no reason to hide their weakness. They have a sense of honor. They don’t react to any of the other men around them, rather, the other men react to him.

This behavior implies high status, a man who is dependable, comfortable in his strengths and weaknesses, a man who can be counted on and who is likely to rise through the ranks and provide for his family.

Comment author: Pimgd 08 June 2016 09:27:35AM 0 points [-]

Would you recommend going if it cost $3000 in total? As a healthy "couple-of-years-out-of-college" male I seem like the the type to not get a discount for anything - you're a member of the working force now, you have a job, you have money, it's the standard full price for you, sir! Not that I'm complaining - it makes sense that I'm in that bracket.

For me, $1000 would be a waste if I lost it, but it'd be easy to bounce back from. I lost $1000 in bad decisions before and was affected about as much as if I had publicly with friends embrassed myself - my brain likes to pull it up from time to time but I think of the lesson learned as more valuable than the money lost.

$4000 or $5500 or numbers in that range would be financially not a problem (I lose numbers, but not physical objects like cars or the roof over my head or the office I work at), but emotionally a huge setback.

Comment author: Kaj_Sotala 08 June 2016 01:38:58PM 3 points [-]

CFAR workshops have a money-back guarantee if you're not satisfied:

And if you conclude the benefits to your happiness and effectiveness don’t recoup the investment, we’ll refund your money up to a year after you attend the workshop.

(from their workshops page)

Comment author: Kaj_Sotala 20 May 2016 02:42:47PM 6 points [-]

I made a new blog on Tumblr. It has photos of smiling people! With more to come!

Why? Previously I happened to need pictures of smiles for a personal project. After going through an archive of photos for a while, I realized that looking at all the happy people made me feel happy and good. So I thought that I might make a habit out of looking at photos of smiling people, and sharing them.

Follow for a regular extra dose of happiness!

Comment author: lukeprog 22 April 2016 02:43:10PM 2 points [-]
Comment author: Kaj_Sotala 23 April 2016 05:13:17AM 1 point [-]

Neat, thanks!

[link] Simplifying the environment: a new convergent instrumental goal

4 Kaj_Sotala 22 April 2016 06:48AM

http://kajsotala.fi/2016/04/simplifying-the-environment-a-new-convergent-instrumental-goal/

Convergent instrumental goals (also basic AI drives) are goals that are useful for pursuing almost any other goal, and are thus likely to be pursued by any agent that is intelligent enough to understand why they’re useful. They are interesting because they may allow us to roughly predict the behavior of even AI systems that are much more intelligent than we are.

Instrumental goals are also a strong argument for why sufficiently advanced AI systems that were indifferent towards human values could be dangerous towards humans, even if they weren’t actively malicious: because the AI having instrumental goals such as self-preservation or resource acquisition could come to conflict with human well-being. “The AI does not hate you, nor does it love you, but you are made out of atoms which it can use for something else.

I’ve thought of a candidate for a new convergent instrumental drive: simplifying the environment to make it more predictable in a way that aligns with your goals.

Comment author: TheAncientGeek 11 April 2016 03:47:28PM 0 points [-]

When you say autonomous AIs, do you mean AIs that are autonomous and superinteligent?

AIs that are initially autonomous and non-superintelligent, then gradually develop towards superintelligence

If you believe in the conjunction of claims that people are motivated to create autonomous, not just agentive, AIs, and that pretty well any AI can evolve into dangerous superintelligence, then the situation is dire, because you cannot guarantee to get in first with an AI policeman as a solution to AI threat.

The situation is better, but only slightly better with legal restraint as a solution to AI threat, because you can lower the probability of disaster by banning autonomous AI...but you can only lower it, not eliminate it, because no ban is 100% effective.

And how serious are you about the threat level? Compare with micro biological research. It could be the case that someone will accidentally create an organism that spells doom for the human race, it cannot be ruled out, but no one is panicing now because there is no specific reason to rule it in, no specific pathway to it. It is a remote possibility, not a serious one.

Someone who sincerely believed that rapid self improvement towards autonomous AI could happen at any time, because there are no specific precondition or precursors for it, is someone who effectively believes it could happen now. But someone who genuinely believes an AI apocalypse could happen now is someone who would e revealing their belief in their behaviour by heading for the hills, or smashing every computer they see.

(With the important caveat that it's unclear whether an AI needed to be generally superintelligent in order to pose a major risk for society.

Narrow superintelligences may well be less dangerous than general superintelligences, and if you are able to restrict the generality of an AI, that could be a path to incremental safety.

But if the path to some kind of spontaneous superintelligence in an autonomous AI is also a path to spontaneous generality, that is hopeless. -- if the one can happen for no particular reason, so can the other. But is the situation really bad, or are these scenarios remote possibilities, like genetically engineered super plagues?

Do you think they could he deployed by basement hackers, or only by large organisations?

Hard to say. The way AI has developed so far, it looks like the capability might be restricted to large organizations with lots of hardware resources at first, but time will likely drive down the hardware requirements.

But by the time the hardware requirements have been driven down for entry level AI, the large organizations will already have more powerful systems, and they will dominate for better or worse. If benevolent, they will supress dangerous AIs coming out of basements, if dangerous they will suppress rivals. The only problematic scenario is where the hackers get in first, since they are less likely to partition agency from intelligence, as I have argued a large organisation would.

But the one thing we know for sure about AI is that it is hard.The scenario where a small team hits on the One Weird Trick to achieve ASI is the most worrying, but also the least likely.

Do you think an organisation like the military or business has a motivation to deploy [autonomous AI]?

Yes.

Which would be what?

Do you agree that there are dangers to an FAI project that goes wrong?

Yes.

Do you have a plan B to cope with a FAI that goes rogue?

Such a plan would seem to require lots of additional information about both the specifics of the FAI plan, and also the state of the world at that time, so not really.

But building an FAI capable of policing other AIs is potentially dangerous, since it would need to be both a general intelligence and super intelligence.

Do you think that having a AI potentially running the world is an attractive idea to a lot of people?

Depends on how we're defining "lots",

For the purposes of the current argument, a democratic majority.

but I think that the notion of a benevolent dictator has often been popular in many circles, who've also acknowledged its largest problems to be that 1) power tends to corrupt 2) even if you got a benevolent dictator, you also needed a way to ensure that all of their successors were benevolent. Both problems could be overcome with an AI,

There are actually three problems with benevolent dictators. As well. as power corrupting, and successorship, there is the problem of ensuring or detecting benevolence in the first place.

You have conceded that Gort AI is potentially dangerous. The danger is that it is fragile in a specific way: a near miss to a benevolent value system is a dangerous one,

so on that basis at least I would expect lots of people to find it attractive. I'd also expect it to be considered more attractive in e.g. China, where people seem to be more skeptical towards democracy than they are in the West.

Additionally, if the AI wouldn't be the equivalent of a benevolent dictator, but rather had a more hands-off role that kept humans in power and only acted to e.g. prevent disease, violent crime, and accidents, then that could be attractive to a lot of people who preferred democracy

That also depends on both getting it right, and convincing people you have got it right

Comment author: Kaj_Sotala 19 April 2016 10:06:48AM 0 points [-]

If you believe in the conjunction of claims that people are motivated to create autonomous, not just agentive, AIs, and that pretty well any AI can evolve into dangerous superintelligence, then the situation is dire, because you cannot guarantee to get in first with an AI policeman as a solution to AI threat.

The situation is better, but only slightly better with legal restraint as a solution to AI threat,

Indeed.

And how serious are you about the threat level? Compare with micro biological research. It could be the case that someone will accidentally create an organism that spells doom for the human race, it cannot be ruled out, but no one is panicing now because there is no specific reason to rule it in, no specific pathway to it. It is a remote possibility, not a serious one.

Someone who sincerely believed that rapid self improvement towards autonomous AI could happen at any time, because there are no specific precondition or precursors for it, is someone who effectively believes it could happen now. But someone who genuinely believes an AI apocalypse could happen now is someone who would e revealing their belief in their behaviour by heading for the hills, or smashing every computer they see.

I don't think that rapid self-improvement towards a powerful AI could happen at any time. It'll require AGI, and we're still a long way from that.

Narrow superintelligences may well be less dangerous than general superintelligences, and if you are able to restrict the generality of an AI, that could be a path to incremental safety.

It could, yes.

But by the time the hardware requirements have been driven down for entry level AI, the large organizations will already have more powerful systems, and they will dominate for better or worse.

Assuming they can keep their AGI systems in control.

Do you think an organisation like the military or business has a motivation to deploy [autonomous AI]?

Yes.

Which would be what?

See my response here and also section 2 in this post.

But building an FAI capable of policing other AIs is potentially dangerous, since it would need to be both a general intelligence and super intelligence. [...] You have conceded that Gort AI is potentially dangerous. The danger is that it is fragile in a specific way: a near miss to a benevolent value system is a dangerous one,

Very much so.

View more: Next