Aha. Ugh, what an unfortunate sequence of events.
Any explanations for why Nick Bostrom has been absent, arguably notably, in recent public alignment conversations (particularly since chatgpt)?
He's not on this list (yet other FHI members, like Toby Ord, are). He wasn't on the FLI open letter, too, but I could understand why he might've avoided endorsing that letter given its much wider scope.
Almost certainly related to that email controversy from a few months ago. My sense is people have told him (or he has himself decided) to take a step back from public engagement.
I think I disagree with this, but it's not a totally crazy call, IMO.
Most of the argument can be boiled down to a simple syllogism: the superior intelligence is always in control; as soon as AI is more intelligent than we are, we are no longer in control.
Seems right to me. And it's a helpful distillation.
...When we think about Western empires or alien invasions, what makes one side superior is not raw intelligence, but the results of that intelligence compounded over time, in the form of science, technology, infrastructure, and wealth. Similarly, an unaided human is no match for most animals. AI, no matter how inte
Thanks Duncan!
There's a somewhat obscure but fairly-compelling-to-me model of psychology which states that people are only happy/okay to the extent that they have some sort of plan, and also expect that plan to succeed.
What's the name of this model; or, can you point to the fuller version of it? Seems right and would see it fleshed out.
hi Matt! on the coordination crux, you say
The first AGIs we construct will be born into a culture already capable of coordinating, and sharing knowledge, making the potential power difference between AGI and humans relatively much smaller than between humans and other animals, at least at first.
but wouldn’t an AGI be able to coordinate and do knowledge sharing with humans because
a) it can impersonate being a human online and communicate with them via text and speech and
b) it‘ll realize such coordination is vital to accomplish it‘s goals a...
You can get many of the benefits of having one country through mechanisms like free trade agreements, open borders, shared currency zones etc.
This is key in my opinion.
Duplicates - digital copies as opposed to genetic clones - might not require new training (unless a whole/partial restart/retraining was being done).
Wouldn't new training be strongly adaptive -- if not strictly required -- if the duplicate's environment is substantively different from the environment of its parent?
When combined with self-modification, there could be 'evolution' without 'deaths' of 'individuals' - just continual ship of Theseus processes. (Perhaps stuff like merging as well, which is more complicated...
I'll check it out -- thanks Zachary!
Isn't Zvi's post an attempt to explain those observations?