People working on friendly AI probably assume that the odds of inventing a friendly AI is higher than establishing a world order in which research associated with existential risks is generally banned. Why is that? Is the reasoning that our civilization is likely to end without significant technological progress (due to reasons like nuclear war, climate change and societal collapse), so we should give it at least a try?

But why do you want it in the first place?

Emotionally -- for the feeling that something new and great is happening here, and I can see it growing.

Reflecting on this: I should not optimize for my emotions (wireheading), but the emotions are important and should reflect reality. If great things are not happening, I want to know that, and I want to fix that. But if great things are happening, then I would like a mechanism that aligns my emotions with this fact.

Okay, what exactly are the "great things" I am thinking about here? What was the referent of this emotion when Eliezer was writing the Sequences?

When Eliezer was writing the Sequences, merely the fact that "there will exist a blog about rationality; without Straw Vulcanism, without Deep Wisdom" seemed like a huge improvement of the world, because it seemed that when such blog will exist, rational people will be able to meet there and conspire to optimize the universe. Did this happen? Well, we have MIRI and CFAR, meetups in various countries (I really appreciate not having to travel across the planet just to meet people with similar values). Do they have impact other than providing people a nice place to chat? I hope so.

Maybe the lowest-hanging fruit was already picked. If someone tried to write Sequences 2.0, what would it be about? Cognitive biases that Eliezer skipped? Or the same ones, perhaps more nicely written, with better examples? Both would be nice things to have, but their awesomeness would probably be smaller than going from zero to Sequences 1.0. (Although, if the Sequences 2.0 would be written so well that they would become a bestseller, and thousands of students outside of existing rationalist communities would read them, then I would rate that as more awesome. So the possibility is there. It just requires very specialized skills.) Or maybe explaining some mathematical or programming concepts in a more accessible way. I mean those concepts that you can use in thinking about probability or how human brain works.

Internet vs real life -- things happening in the real world are usually more awesome than things happening merely online. For example, a rationalist meetup is usually better than reading an open thread on LW. The problem is visibility. The basic rule of bureaucracy -- if it isn't documented, it didn't happen -- is important here, too. When given a choice between writing another article and doing something in the real world, please choose the latter (unless the article is really exceptionally good). But then, please also write an article about it, so that your fellow rationalists who were not able to participate personally can share the experience. It may inspire them to do something similar.

By the way, if you are unhappy about the "decline" of LW because it will make a worse impression on new people you would like to introduce to LW culture -- point them towards the book instead.

Do you care about rationality? Then research rationality and write about it, here or anywhere else. Do you enjoy the community of LWers? Then participate in meetups, discuss random things in OTs, have nice conversations, etc. Do you want to write more rationalist fiction? Do it. And so on.

Adding: if you would like to see a rationalist community growing, research and write about creating and organizing communities. (That is an advice for myself, when I will have more free time.)

Judging from the recent decline of LW, it seems that the initial success of LW wasn't due to rationality, but rather due to Eliezer's great writing. If we want LW to become a fun place again, we should probably focus on writing skills instead of rationality skills. Not everyone can be as good as Eliezer or Yvain, but there's probably a lot of low hanging fruit. For example, we pretty much know what kind of fiction would appeal to an LWish audience (HPMOR, Worm, Homestuck...) and writing more of it seems like an easier task than writing fiction with mass-market appeal.

Does anyone else feel that it might be a promising direction for the community? Is there a more structured way to learn writing skills?

What is the point of having separated Open Threads and Stupid Questions threads, instead of allowing "stupid questions" in OTs and making OTs more frequent?

Maybe not for that reason. But the opportunity cost of having kids, for example in terms of time and money, is pretty high. You could easily make an argument that those resources would be more effectively used for higher impact activities.

The money as dead children analogy might be particularly useful here, since we're comparing kids with kids.

Is it unethical to have children pre-Singularity, for the risk of them dying?

Yes, this this this this this this this. "The capacity of human minds is limited and I'll accept climbing up higher in abstraction levels at the price of forgetting how the lower-level gears turn." If I could upvote this multiple times, I would.

This is the crux of this entire approach. Learn the higher level, applied abstractions. And learn the very basic fundamentals. Forget learning how the lower-level gears turn: just learn the fundamental laws of physics. If you ever need to figure out a lower-level gear, you can just derive it from your knowledge of the fundamentals, combined with your big-picture knowledge of how that gear fits into the overall system.

On the other hand, if you don't have a solid grasp of linear algebra, your ability to do most types of machine learning is seriously impaired. You can learn techniques like e.g. matrix inversions as needed to implement the algorithms you're learning, but if you don't understand how those techniques work in their original context, they become very hard to debug or optimize. Similarly for e.g. cryptography and basic information theory.

That's probably more the exception than the rule, though; I sense that the point of most prerequisites in a traditional science curriculum is less to provide skills to build on and more to build habits of rigorous thinking.