Ouch (ch 68):
Minerva was going over the Transfiguration parchment due Monday, and had just marked down to negative two hundred points a fifth-year parchment with an error that could have potentially killed someone. During her first year as a professor she'd been indignant at the folly of older students, now she was just resigned. Some people not only never learned, they never noticed that they were hopeless, they stayed bright and eager and kept on trying. Sometimes they believed you when you told them, before they left Hogwarts, that they must never try anything unusual, give up free Transfiguration and use the art only through established Charms; and sometimes... they didn't.
"It's better if you don't even try" - I wonder what inspired that sentiment.
When I read that I assumed it was a critique of people who are bright enough to plausibly create an AGI but not bright enough to make sure it is friendly.
They shouldn't "try anything unusual".
Update: Discussion has moved on to a new thread.
The load more comments links are getting annoying (at least if you're not logged in), so it's time for a new Harry Potter and the Methods of Rationality discussion thread. We're also approaching the traditional 500-comment mark, but I think that hidden comments provide more appropriate joints to carve these threads at. So as of chapter 67, this is the place to share your thoughts about Eliezer Yudkowsky's Harry Potter fanfic.
The first 5 discussion threads are on the main page under the harry_potter tag. Threads 6 and on (including this one) are in the discussion section using its separate tag system. Also: one, two, three, four, five, six. The fanfiction.net author page is the central author-controlled HPMOR clearinghouse with links to the RSS feed, pdf version, TV Tropes pages, fan art, and more, and AdeleneDawner has kept an archive of Author's Notes.
As a reminder, it's often useful to start your comment by indicating which chapter you are commenting on.
Spoiler Warning: this thread is full of spoilers. With few exceptions, spoilers for MOR and canon are fair game to post, without warning or rot13. More specifically: