Relevant quote from Altman after the firing:

“I think this will be the most transformative and beneficial technology humanity has yet invented,” Altman said, adding later, “On a personal note, four times now in the history of OpenAI, the most recent time was just in the last couple of weeks, I’ve gotten to be in the room when we push … the veil of ignorance back and the frontier of discovery forward.”

However, uploading seems to offer a third way: instead of making alignment researchers more productive, we "simply" run them faster.

When I think about uploading as an answer to AI, I don't think of it as speeding up alignment research necessarily, but rather just outpacing AI. You won't get crushed by an unaligned AI if you're smarter and faster than it is, with the same kind of access to digital resources.

The breeding process would adjust that if it was a limiting factor.

The problem with this is that one day you'll see someone who has the same flaw you've been trying to suppress in yourself, and they just completely own it, taking pride in it, focusing on its advantages, and never once trying to change it. And because they are so self-assured about it, the rest of the world buys in and views it as more of an interesting quirk than a flaw.

When you encounter that person, you'll feel like you threw away something special.

How about this one? Small group or single individual manages to align the first very powerful AGI to their interests. They conquer the world in a short amount of time and either install themselves as rulers or wipe out everyone else.

Oh, I see your other graph now. So it just always guesses 100 for everything in the vicinity of 100.

This is a cool idea. I wonder how it's able to do 100, 150, and 200 so well. I also wonder what are the exact locations of the other spikes?

You can deduce a lot about someone's personality from the shape of his face.

I don't know if this is really that controversial. The people who do casting for movies clearly understand it.

On the question of morality, objective morality is not a coherent idea. When people say "X is morally good," it can mean a few things:

• Doing X will lead to human happiness
• I want you to do X
• Most people want you to do X
• Creatures evolving under similar conditions as us will typically develop a preference for X
• If you don't do X, you'll be made to regret it
• etc...

But believers in objective morality will say that goodness means more than all of these. It quickly becomes clear that they want their own preferences to be some kind of cosmic law, but they can't explain why that's the case, or what it would even mean if it were.

On the question of consciousness, our subjective experiences are fully explained by physics. 

The best argument for this is that our speech is fully explained by physics. Therefore physics explains why people say all of the things they say about consciousness. For example, it can explain why someone looks at a sunset and says, "This experience of color seems to be occurring on some non-physical movie screen." If physics can give us a satisfying explanation for statements like that, it's safe to say that it can dissolve any mysteries about consciousness.

The problem isn't that he's overly sure about "contentious topics." These are easy questions that people should be sure about. The problem is that he's sure in the wrong direction.