[set 200 years after a positive singularity at a Storyteller's convention]

If We Win Then...

My friends, my friends, good news I say
The anniversary’s today
A challenge faced, a future won
When almost came our world undone

We thought for years, with hopeful hearts
Past every one of the false starts
We found a way to make aligned
With us, the seed of wondrous mind

They say at first our child-god grew
It learned and spread and sought anew
To build itself both vast and true
For so much work there was to do

Once it had learned enough to act
With the desired care and tact
It sent a call to all the people
On this fair Earth, both poor and regal

To let them know that it was here
And nevermore need they to fear
Not every wish was it to grant
For higher values might supplant

But it would help in many ways:
Technologies it built and raised
The smallest bots it could design
Made more and more in ways benign

And as they multiplied untold
It planned ahead, a move so bold
One planet and 6 hours of sun
Eternity it was to run

Countless probes to void disperse
Seed far reaches of universe
With thriving life, and beauty's play
Through endless night to endless day

Now back on Earth the plan continues
Of course, we shared with it our values
So it could learn from everyone
What to create, what we want done

We chose, at first, to end the worst
Diseases, War, Starvation, Thirst
And climate change and fusion bomb
And once these things it did transform

We thought upon what we hold dear
And settled our most ancient fear
No more would any lives be stolen
Nor minds themselves forever broken

Now back to those far speeding probes
What should we make be their payloads?
Well, we are still considering
What to send them; that is our thing. 

The sacred task of many aeons
What kinds of joy will fill the heavens?
And now we are at story's end
So come, be us, and let's ascend

Yeah, I think I model that anything which does understanding of physics is to some extent 'beyond physics', because you're translating from a raw file format to high level picture, and that's taking computation. Reading from the homomorphic one isn't an entirely new step, as opposed to the 'straightforward' one, it's just a much more difficult function in the place a usually simple function goes.

Or to take another shot: Yes to "there is a way to construct an omniscient perspective, it just isn't going to be straightforwardly derivable from the physical state"

However, 'straightforwardly' -> 'computationally intractable' as a difficulty jump for extracting high level features is a ~quantitive step up in the difficulty of an existing step of universe-parsing, not a novel step with strong metaphysical surprise.

Thanks, updated to forecasters, does that seem fair?

Also, I know this is super hard, but do you have a sense of what superforcasters might have guessed back then?

Informally, every possible physical state has a unique corresponding mental state. Formally:

My first pass response to this is: Yes, there's a unique mental state for each physical state, but the aspects of that mental state can be partitioned from each other in ways that are computationally intractable to un-partition. The mapping you use from raw physics or reality to whatever understanding you use it for^[1] is a function not a primitive, and in this case that function could place you on either side of an informational partition^[2] (depending on whether the mapping function does something like encrypts your viewing portal/perspective). Analogous to looking at an object from different perspectives, which under normal circumstances would be connectable efficiently, but here aren't.

Normally you can just privilege the simpler mapping function and get everything you'd want, but your simple mapping function isn't physics, it's viewing physics from a direction that looks simpler to you. If this is right:

1. A misses that the homomorphic mind is 'real' from a different perspective, there's just more than one direction to be omniscient into the universe from
2. The somewhat shaky assumptions B makes aren't required
3. I don't quite think C's framing fits, it's not that reality exceeds physics, it's that physics isn't informationally/computationally interconnected in a way that makes any single perspective capable of efficiently getting all the interesting information.

I think some of Wolfram's work on the Ruliad gave me some of the intuitions I'm using here, if this feels worth digging into.

1. ^{^}

   Itself a computational/informational process, which it looks like you're not staring right at in this post?

2. ^{^}

   The function could even give you access to both sides of the partition, with some complexity addition.

I think both of these camps are seeing real things. I think:

We should not race to superintelligence because we're not prepared to have a reasonable chance of surviving
AND
It's extremely hard to stop due to underlying dynamics of civilization (competitive dynamics, moloch, etc)

We should try to stop the race, but be clearsighted about the forces we're up against and devise plans that have a chance of working despite them.

Yup, agree this holds for a while, and agree that benchmarking superintelligence is tricky. However, I think there's reason to expect the dynamics driving the benchmarks to change in ways which don't have clear precedent at some point, once new parts of the feedback loops seriously take off.

I operationalize as: I expect >2 and at least twice as many trend-breaking events in the reference of the move from 7 to 4 month doubling times as downwards trend breaking ones.

I expect that at some point, possibly after loss of control, a large number of things unblock in quick succession and you get a supercritical chain reaction of improvements to capabilities which bows naive extrapolation out of the water.

2021 forecasters vs 2025 reality:

"Current performance on [the MATH] dataset is quite low--6.9%--and I expected this task to be quite hard for ML models in the near future. However, forecasters predict more than 50% accuracy* by 2025! This was a big update for me. (*More specifically, their median estimate is 52%; the confidence range is ~40% to 60%, but this is potentially artifically narrow due to some restrictions on how forecasts could be input into the platform.)"(source)

Reality: 97.3%+ ^[1] (on a narrowed subset of only the hardest questions, including just difficulty 5 ones, called MATH-500 which was made because the original benchmark got saturated)

Reliable forecasting requires either a generative model of the underlying dynamics, or a representative reference class. The singularity has no good reference class, so people trying to use reference classes rather than gears modelling will predictably be spectacularly wrong.

1. ^{^}

   

Even in the weird case that they do delete, the training code+data+text outputs should be enough to reverse engineer the weights pretty reliably.

But yeah, agree this is would be pretty silly.

Nice, yup, agree. I covered this idea briefly in Utopiography interview, asking with a bunch of other utopia design. 

I think EA-specific events being big tent EA seems healthy, but it is absolutely wild to me that there's still no large scale recurring AI existential safety conference. If anyone seriously wants to make it, I saved the domain aisafety.global for you.

Also, Skyler who runs ACX meetups globally would be up for stepping up to support more AIS groups, but is capacity constrained and would need additional funding for ~2-3 people to do this. I'm likely going to include this in my list of recs to funders.