Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Halfwitz 23 November 2014 10:24:07PM *  8 points [-]

To be honest, I had you pegged as being stuck in a partisan spiral. The fact that you are willing to do this is pretty cool. Have some utils on the house. I don’t know if officially responding to your blog is worth MIRI’s time; it would imply some sort of status equivalence.

Also, you published some very embarrassing quotes from Yudkowsky. I’m guessing you caused him quite a bit of distress, so he’s probably not inclined to do you any favors. Mining someone’s juvenilia for outrageous statements is not productive – I mean he was 16 when he wrote some of the stuff you quote. I would remove those pages. Same with the usenet stuff – I know it was posted publicly but it feels like furtively-recorded conversations to me all these years later. Stick to arguments against positions MIRI and Yudkowsky currently hold. Personally I’ve moved from highly-skeptical of MIRI to moderately approving. I made this comment a year ago:

The fact that MIRI is finally publishing technical research has impressed me. A year ago it seemed, to put it bluntly, that your organization was stalling, spending its funds on the full-time development of Harry Potter fanfiction and popular science books. Perhaps my intuition there was uncharitable, perhaps not. I don't know how much of your lead researcher's time was spent on said publications, but it certainly seemed, from the outside, that it was the majority. Regardless, I'm very glad MIRI is focusing on technical research. I don't know how much farther you have to walk, but it's clear you're headed in the right direction.

And MIRI has stayed on course and is becoming a productive think tank with three full-time researchers and, it seems to me, a highly competent CEO. It is a very different organization now than the one you started out criticizing.

Comment author: lukeprog 23 November 2014 11:50:07PM 9 points [-]

For the record, I genuinely object to being thought of as a "highly competent CEO." I think "non-natural CEO working hard and learning fast and picking up lots of low-hanging fruit but also making lots of mistakes along the way because he had no prior executive experience" is more accurate. The good news is that I've been learning even more quickly since Matt Fallshaw joined the Board, since he's able and willing to put in the time to transfer to me what he's learned from launching and running multiple startups.

Comment author: Natha 05 November 2014 07:16:32AM *  1 point [-]

Subject: Animal Behavior (Ethology)

Recommendation: Animal Behavior: An Evolutionary Approach (6th Edition, 1997) Author: John Alcock

This is an excellent, well organized, engagingly written textbook. It may be a tiny bit denser than the comparison texts I give below, but I found it to be far and away the most rewarding of the three (I've just read the three). The natural examples he gives to illustrate the many behaviors are perfectly curated for the book. Also, he uses Tinbergen's four questions to frame these discussions, which ensured a rich description of each behavior. The author gives a cogent defense of sociobiology in the last chapter, which was icing on the cake.

Other #1: Principles of Animal Behavior (1st Edition, 2003) Author: Lee Alan Dugatkin

This was one I had to read for a class; it's a bit shorter than Alcock, and maybe it has been improved upon since this inaugural edition, but I found the fluff-to-substance ratio to be concerningly high. It was much more basic than Alcock, perhaps better suited for a high school audience. The chapters were written like works of fiction and the author maintained this style throughout, which I found distracting (though others may like it). Bottom line: If you have had a decent college level class in biology, you would definitely be better off going straight to an older edition of Alcock.

Other #2: Animal Behavior: An Evolutionary Approach (9th Edition, 2009) Author: John Alcock

I read through this edition too (I think there's a 10th out now) while writing my undergraduate thesis to make sure I hadn't missed any important updates in the field (I hadn't). The new edition had ~100 fewer pages; it was long on pictures (quite a few more than its predecessor) and short on content. It's been several years now and I can't remember exactly the ways in which it differed, but “watered down” comes to mind. I would highly recommend picking up an older edition unless this one is specifically required.

Comment author: lukeprog 23 November 2014 02:47:22AM 0 points [-]

Added, thanks!

Comment author: pjeby 23 November 2014 02:01:41AM 1 point [-]

Would you mind redacting your quotes of the transcript, so that people can instead enjoy the episode in context? I was intentionally vague about the parts you've chosen to excerpt or talk about, specifically not to ruin people's enjoyment of the episode. (Also, reading a transcript is a very different experience than the actual episode, lacking as it does the timing, expressions, and body language that suggest what the show's makers want us to think.)

It also seems to me that you are not interpreting the quotes particularly charitably. For example, when I saw the episode, I interpreted "can't be accounted for" as shorthand for "emergent behavior we didn't explicitly ask for", not "AI is magic". Likewise, while Mason implies that hostility is inevitable, his reward-channel takeover explanation grounds this presumption in at least one example of how an AI would come to display behavior humans would interpret as "hostile". I took this as shorthand for "there are lots of ways you can end up with a bad result from AI", not "AI is hostile and this is just one example."

Bella is not actually presented as a hostile creature who maliciously kills its creator. Heck, Bella is mostly made to seem less anthropomorphic than even Siri or Google Now! (Despite the creepy-doll choice of avatar.) The implication by Bella's co-creator that Bella might have decided to "alter a variable" by killing someone doesn't imply what a human would consider hostility. Sociopathic amorality, perhaps, but not hostility.

And while Holmes at times seems to be operating from a "true AI = magic" perspective, I also interpreted the episode as making fun of him for having this perspective, such as his pointless attempts at a Turing test that Bella essentially failed hard at in the first 30 seconds. One thing you might miss if you're not a regular of the show, is that one of Holmes' character quirks is going off on these obsessive digressions that don't always work out the way he insists they will. (Unlike the literary Sherlock, this Holmes is often wrong, even about things he states his absolute certainty about... and Watson's role is often to prod his thinking into more productive channels.)

Anyway, his extended "testing" of Bella, and the subsequent remark from Watson to Kitty about using a fire extinguisher on him if he starts hitting things, is a strong signal that we are expected to humor his pointless obsession, as all the people around him are thoroughly unimpressed by Bella right away, and don't need to spend hours questioning it to "prove" it's not "really" intelligent.

Is it possible for somebody to view the episode through their existing trope-filled worldview and not learn anything? Sure. But I don't think it would've been practical to cover the entire inferential distance in just the "A" story of a 44-minute murder mystery TV show, so I applaud the writers for actually giving it a shot, and the artful choices made to simplify their presentation without dumbing things down to the point of being actually wrong, or committing any of the usual howling blunders. For a show intended purely as entertainment, they did a better job of translating the ideas than many journalists do.

OTOH, perhaps it's an illusion of transparency on my part, and only someone already exposed to the bigger picture would be able to grasp any of it from what was put in the show, and the average person will not in fact see anything differently after watching it. But even if that is the case, I think the show's makers still deserve credit -- and lots of praise -- just for trying.

Comment author: lukeprog 23 November 2014 02:05:44AM *  3 points [-]

I've edited my post a bit in response to your concerns. I don't think I should redact all the quotes, though.

Comment author: lukeprog 23 November 2014 01:28:15AM *  4 points [-]

WARNING: SPOILERS FOLLOW. You may want to enjoy the episode yourself before reading the below transcript excerpts.


From the transcript, here's a bit about AI not doing what it's programmed to do:

Computer scientist: "[Our AI program named Bella] performs better than we expected her to."

Holmes: "Explain that."

Computer scientist: "A few weeks back, she made a request that can't be accounted for by her programming."

Holmes: "Impossible."

Holmes' assistant: "What's impossible? For the computer to ask for something?"

Holmes: "If it made a request, it did so because that's what it was programmed to do. He's claiming true machine intelligence. If he's correct in his claims, he has made a scientific breakthrough of the very highest order."

Another trope: At one point a young computer expert says "Everybody knows that one day intelligent machines are going to evolve to hate us."

Here's the bit about reward-channel takeover:

"What's the 'button-box' thing?"

"It's a scenario somebody blue-skyed at an AI conference. Imagine there's a computer that's been designed with a big red button on its side. The computer's been programmed to help solve problems, and every time it does a good job, its reward is that someone presses its button. We've programmed it to want that... so at first, the machine solves problems as fast as we can feed them to it. But over time, it starts to wonder if solving problems is really the most efficient way of getting its button pressed. Wouldn't it be better just to have someone standing there pressing its button all the time? Wouldn't it be even better to build another machine that could press its button faster than any human possibly could?"

"It's just a computer, it can't ask for that."

"Well, sure it can. If it can think, and it can connect itself to a network, well, theoretically, it could command over anything else that's hooked onto the same network. And once it starts thinking about all the things that might be a threat to the button-- number one on that list, us-- it's not hard to imagine it getting rid of the threat. I mean, we could be gone, all of us, just like that."

"That escalated quickly."

There's also a think tank called the Existential Threat Research Association (ETRA):

"[ETRA is] one of several institutions around the world which exists solely for the purpose of studying the myriad ways in which the human race can become extinct... and within this think tank, there is a small, but growing school of thought that holds that the single greatest threat to the human race... is artificial intelligence... Now, imagine their quandary. They have pinpointed a credible threat, but it sounds outlandish. The climate-change people, they can point to disastrous examples. The bio-weapons alarmists, they have a compelling narrative to weave. Even the giant comet people sound more serious than the enemies of AI.

"So... these are the people at ETRA who think AI is a threat? You think one of them killed Edwin Borstein, one of the top engineers in the field, and made it look like Bella did it, all so they could draw attention to their cause?

"A small-scale incident, something to get the media chattering."

One ETRA person is suspiciously Stephen Hawking-esque:

"Isaac Pike is a professor of computer science. He's also a vocal alarmist when it comes to artificial intelligence. Pike was born with spina bifida. Been confined to a wheelchair his entire life. For obvious reasons, he could not have executed the plan... but his student..."


Isaac Pike ends up being (probably) responsible for murdering Edwin Borstein via a computer virus installed on Bella. He says: "You're talking about nothing less than the survival of the species. Surely that's worth compromising one's values for?"

In response to Weekly LW Meetups
Comment author: lukeprog 22 November 2014 03:22:26AM 0 points [-]

Thanks again, Frank!

Comment author: cursed 18 November 2014 02:25:06AM 1 point [-]

Do you mind revealing what Shane's timelines are, and the probability that he thinks that he'll play a role in AGI?

Comment author: lukeprog 18 November 2014 02:52:29AM 8 points [-]
Comment author: lukeprog 17 November 2014 01:44:10AM 12 points [-]

Fascinating. Maybe he's been talking to Shane Legg of DeepMind, who also has much sooner timelines than I do.

In response to MIRI Research Guide
Comment author: Capla 11 November 2014 03:11:29PM 0 points [-]

Can I access the original version somehow? In particular, it included a quote at the beginning (something to the effect of “you may feel like your not so talented compared to your uber-amazing classmates in at you prestigious program, but we very rarely find individuals who are familiar with all or most of these fields.") that I wanted to share with a math-friend who expressed a fear (which he knows to be largely irrational) that he's not smart enough to work on interesting problems.

In response to comment by Capla on MIRI Research Guide
Comment author: lukeprog 11 November 2014 03:15:46PM 2 points [-]
Comment author: singularitard 21 October 2014 06:57:37PM *  1 point [-]

This sounds familiar. Are you aware of other similar concepts previously communicated elsewhere? I feel certain I've read something along these lines before. By all means, claim it's original though.

Comment author: lukeprog 02 November 2014 11:14:07PM 0 points [-]

Not sure if this is what you're thinking of, but there's a research area called "adjustable autonomy" and a few other names, which superficially sounds similar but isn't actually getting at the problem described here, which comes about due to convergent instrumental values in sufficiently advanced agents.

Comment author: AndreInfante 02 November 2014 08:48:47PM 2 points [-]

That sounds fascinating. Could you link to some non-paywalled examples?

Comment author: lukeprog 02 November 2014 11:09:21PM 3 points [-]

View more: Next