It's hard to even imagine how to make a mind - build a brain - that does what's 'right', what it 'should'. We, the humans who have to build that mind, don't know what's right a lot of the time; we change our minds about what's right, and say that we were wrong before.
And yet everything we need has to be inside our minds somewhere, in some sense. Not upon the stars is it written. What's 'right' doesn't come from outside us, as a great light from the sky. So it has to be within humans. But how do you get it out of humans and into a new mind?
Start with what's really there in human minds. Then ask what we would think, if we knew everything a stronger mind knew. Ask what we would think if we had years and years to think. Ask what we would say was right, if we knew everything inside our own minds, all the real reasons why we decide what we decide. If we could change, become more the people we wished we were - what would we think then?
Building a mind which will figure all that out, and then do it, is about as close as we can now imagine to building something that does what's 'right', starting from only what's already there in human minds and brains.
xkcd's Up-Goer Five comic gave technical specifications for the Saturn V rocket using only the 1,000 most common words in the English language.
This seemed to me and Briénne to be a really fun exercise, both for tabooing one's words and for communicating difficult concepts to laypeople. So why not make a game out of it? Pick any tough, important, or interesting argument or idea, and use this text editor to try to describe what you have in mind with extremely common words only.
This is challenging, so if you almost succeed and want to share your results, you can mark words where you had to cheat in *italics*. Bonus points if your explanation is actually useful for gaining a deeper understanding of the idea, or for teaching it, in the spirit of Gödel's Second Incompleteness Theorem Explained in Words of One Syllable.
As an example, here's my attempt to capture the five theses using only top-thousand words:
If you make a really strong computer and it is not very nice, you will not go to space today.
Other ideas to start with: agent, akrasia, Bayes' theorem, Bayesianism, CFAR, cognitive bias, consequentialism, deontology, effective altruism, Everett-style ('Many Worlds') interpretations of quantum mechanics, entropy, evolution, the Great Reductionist Thesis, halting problem, humanism, law of nature, LessWrong, logic, mathematics, the measurement problem, MIRI, Newcomb's problem, Newton's laws of motion, optimization, Pascal's wager, philosophy, preference, proof, rationality, religion, science, Shannon information, signaling, the simulation argument, singularity, sociopathy, the supernatural, superposition, time, timeless decision theory, transfinite numbers, Turing machine, utilitarianism, validity and soundness, virtue ethics, VNM-utility