Compared to its competition in the AGI race, MIRI was always going to be disadvantaged by both lack of resources and the need to choose an AI design that can predictably be made Friendly as opposed to optimizing mainly for capability. For this reason, I was against MIRI (or rather the Singularity Institute as it was known back then) going into AI research at all, as opposed to pursuing some other way of pushing for a positive Singularity.

In any case, what other approaches to Friendliness would you like MIRI to consider? The only other approach that I'm aware of that's somewhat developed is Paul Christiano's current approach (see for example https://medium.com/ai-control/alba-an-explicit-proposal-for-aligned-ai-17a55f60bbcf), which I understand is meant to be largely agnostic about the underlying AI technology. Personally I'm pretty skeptical but then I may be overly skeptical about everything. What are your thoughts? I don't recall seeing you having commented on them much.

Are you aware of any other ideas that MIRI should be considering?

How did you arrive at the conclusion that we're not facing big expected costs with these questions? It seems to me that for example the construction of large nuclear arsenals and lack of sufficient safeguards against nuclear war has already caused a large expected cost, and may have been based on one or more incorrect philosophical understandings (e.g., to the question of, what is the right amount of concern for distant strangers and future people). Similarly with "how much should we prioritize fast technological development?" But this is just from intuition since I don't really know how to compute expected costs when the uncertainties involved have a large moral or normative component.

My best guess is that we will get to do many-centuries-of-current-humanity worth of thinking before we really need to get any of these questions right.

Do you expect technological development to have plateaued by then (i.e., AIs will have invented essentially all technologies feasible in this universe)? If so, do you think there won't be any technologies among them that would let some group of people/AIs unilaterally alter the future of the universe according to their understanding of what is normative? (For example, intentionally or accidentally destroy civilization, or win a decisive war against the rest of the world.) Or do you think something like a world government will have been created to control the use of such technologies?

The goal / best case is that the development of AI doesn't depend on sorting out these kinds of considerations for its own sake, only insofar as the AI has to actually make critical choices that depend on these considerations.

Isn't a crucial consideration here how soon after the development of AI they will be faced with such choices? If the answer is "soon" then it seems that we should try to solve the problems ahead of time or try to delay AI. What's your estimate? And what do you think the first such choices will be?

Does your link to the first thread imply that you believe securing one's bitcoin (and realizing its unique benefits) is ultimately a futile venture, especially in the presence of an adversary of advanced intelligence?

Yes, that looks likely to be the case.

To the second link, I guess you mean to imply the monetary policy of Bitcoin is ultimately flawed due to its deflationary nature?

That's part of it. If decentralized cryptocurrency is ultimately good for the world, then Bitcoin may be bad because its flawed monetary policy prevents or delays widespread adoption of cryptocurrency. But another part is that cryptocurrency and other cypherpunk/cryptoanarchist ideas may ultimately be harmful even if they are successful in their goals. For example they tend to make it harder for governments to regulate economic activity, but we may need such regulation to reduce existential risk from AI, nanotech, and other future technologies.

If one wants to push the future in a positive direction, it seems to me that there are better things to work on than Bitcoin.

FAI designs that require high confidence solutions to many philosophical problems also do not seem very promising to me at this point. I endorse looking for alternative approaches.

I agree that act-based agents seem to require fewer high confidence solutions to philosophical problems. My main concern with act-based agents is that these designs will be in competition with fully autonomous AGIs (either alternative designs, or act-based agents that evolve into full autonomy due to inadequate care of their owners/users) to colonize the universe. The dependence on humans and lack of full autonomy in act-based agents seem likely to cause a significant weakness in at least one crucial area of this competition, such as general speed/efficiency/creativity, warfare (conventional, cyber, psychological, biological, nano, etc.), cooperation/coordination, self-improvement, and space travel. So even if these agents turn out to be "safe", I'm not optimistic that we "win" in the long run.

My own idea is to aim for FAI designs that can correct their philosophical errors, autonomously, the same way that we humans can. Ideally, we'd fully understand how humans reason about philosophical problems and how philosophy normatively ought to be done before programming or teaching that to an AI. But realistically, due to time pressure, we might have to settle for something suboptimal like teaching through examples of human philosophical reasoning. Of course there's lots of ways for this kind of AI to go wrong as well, so I also consider it to be a long shot.

Do you think that we need to understand issues like this one, and have confidence in that understanding, prior to building powerful AI systems?

Let me ask you a related question. Suppose act-based designs are as successful as you expect them to be. We still need to understand issues like the one described in Eliezer's post (or solve the meta-problem of understanding philosophical reasoning) at some point, right? When do you think that will be? In other words, how much time do you think successfully creating act-based agents buys us?

I'm not sure how much there is to learn from the field of computer security, with regard to the OP's question. It's relatively easy to cultivate an adversarial mindset and get funding for conferences, research centers, labs, intellectual firepower, etc., when adversaries exist at the present time and are causing billions of dollars of damage each year. How to do that if the analogous adversaries are not expected to exist for a decade or more, and we expect it will be too late to get started once the adversaries do exist?

I don't follow Bitcoin development very closely, basically just reading about it if a story shows up on New York Times or Wired. If you're curious as to why, see this post and this thread.

If the identity of the individual were confirmed it would perhaps, at a minimum, elevate their engineer/thinker status such that other ideas and pieces of work attributed to them may receive more attention (and maybe help) from many others who would perhaps not otherwise have happened upon them.

This is interesting and something I hadn't thought about. Now I'm more curious who Satoshi is and why he or she or they have decided to remain anonymous. Thanks! You might want to post your idea somewhere else too, like the Bitcoin reddit or forum, since probably not many people will get to read it here.

Can you say any more about alternatives you've been considering?

Not much to say, unfortunately. I tried looking at some frequentist ideas for inspiration, but didn't find anything that seemed to have much bearing on the kind of philosophical problems we're trying to solve here.