[set 200 years after a positive singularity at a Storyteller's convention]

If We Win Then...

My friends, my friends, good news I say
The anniversary’s today
A challenge faced, a future won
When almost came our world undone

We thought for years, with hopeful hearts
Past every one of the false starts
We found a way to make aligned
With us, the seed of wondrous mind

They say at first our child-god grew
It learned and spread and sought anew
To build itself both vast and true
For so much work there was to do

Once it had learned enough to act
With the desired care and tact
It sent a call to all the people
On this fair Earth, both poor and regal

To let them know that it was here
And nevermore need they to fear
Not every wish was it to grant
For higher values might supplant

But it would help in many ways:
Technologies it built and raised
The smallest bots it could design
Made more and more in ways benign

And as they multiplied untold
It planned ahead, a move so bold
One planet and 6 hours of sun
Eternity it was to run

Countless probes to void disperse
Seed far reaches of universe
With thriving life, and beauty's play
Through endless night to endless day

Now back on Earth the plan continues
Of course, we shared with it our values
So it could learn from everyone
What to create, what we want done

We chose, at first, to end the worst
Diseases, War, Starvation, Thirst
And climate change and fusion bomb
And once these things it did transform

We thought upon what we hold dear
And settled our most ancient fear
No more would any lives be stolen
Nor minds themselves forever broken

Now back to those far speeding probes
What should we make be their payloads?
Well, we are still considering
What to send them; that is our thing. 

The sacred task of many aeons
What kinds of joy will fill the heavens?
And now we are at story's end
So come, be us, and let's ascend

A fair portion of the most visible bits are John downstream of I think something like https://www.lesswrong.com/posts/qGMonyLRXFRnCWSj6/generalized-coming-out-of-the-closet. From what I gather from a distance the bay has a pretty intense BDSM scene, which I'm sure is great for many people, but it's something with sharp edges that don't look like they're being sufficiently respected or understood by all participants.

Almost every woman I have known well enough that they would have told me about it has showed psychological harm from previous unwanted sexual advances or, very often, worse. This is regularly the largest trauma they have, and some of the stories are pretty shocking and consistent in a way that lets you trace it the way minds in general are shaped around this.

One of the core cultural adaptations to the fact that the male sex drive can be expressed destructively is normalising consent as a required feature of sexuality. @johnswentworth's arranging language here seems to reduce the potency of that bright line in a way which looks to me likely to increase the rate of serious harm and generate legitimate feelings of unsafety in a large number of women.

Additionally, normalising nonconsent as a way to get sex tilts desperate or inexperienced men towards terrible, life ruining, mistakes when the person they advance on doesn't have this specific kink AND the unstated consent to not need verbal consent with this person at this time.

May your growth mindset be strong enough to both increase your circle of moral concern to the widest you would reflectively endorse, and your capabilities to meet the challenge of influencing that circle well.

I think of Decisive Strategic Advantage as the key differentiator, but not sure how best to make that into a short handle.

I have not, as a don't have stock to donate. My guess it it at least skips the DAF step, so it's simpler, though it might not actually be fully straightforward. Curious to hear from someone who tries it.

The actual logistics of donating stock are a pain.

If they set up an every.org page it looks straightforward? e.g. select stock on this

I like the phrase "myopic consequentialism" for this, and it often has bad consequences because bounded agents need to cultivate virtues (distilled patterns that work well across many situations, even when you don't have the compute or information on exactly why it's good in many of those) to do well rather than trying to brute-force search in a large universe.

Plausibly the largest update to the foundations of my epistemology I've had since reading the sequences. Neat.

Feels like it captures and grounds out something critical about how to reason in a complex and uncertain domain without resorting to deference. Spot checking lots of parts of a world model and finding them consistent is genuine though not strong evidence of the overall conclusion, because they could have been inconsistent.

It's neat how this bootstraps from nothing, even if for high confidence you want other forms of evidence too.

Excellent post, plausibly the most rigorous explanation of the core reasons to expect doom, but really really needs a more memetic handle. Actually suggest the authors go back and pick one even now, perhaps "Catastrophic Misalignment is the Default", and make the current title a subtitle.

I don't know what the analogue is here. Better scaffolding? More capable LLM?

Fine tuning on examples of doing the thing you want, or making sure your new trick gets into datasets? Even more high effort than the compiler version for the amount of improves it?