>then it just needs to find one poor schmuck to accept deliveries and help it put together its doomsday weapon.
Yes, but do I take it for granted that an AI will be able to manipulate the human into creating a virus that will kill literally everyone on Earth, or at least a sufficient number to allow the AI to enact some secondary plans to take over the world? Without being detected? Not with anywhere near 100% probability. I just think these sorts of arguments should be subject to Drake equation-style reasonings that will dilute the likelihood of doom under most circumstances.
This isn't an argument for being complacent. But it does allow us to push back against the idea that "we only have one shot at this."
I outlined my expectations, not a "plan".
>You lack imagination, its painfully easy, also cost + required IQ has been dropping steadily every year.
Conversely, it's possible that doomers are suffering from an overabundance of imagination here. To be a bit blunt, I don't take it for granted that an arbitrarily smart AI would be able to manipulate a human into developing a supervirus or nanomachines in a risk-free fashion.
The fast takeoff doom scenarios seem like they should be subject to Drake equation-style analyses to determine P(doom). Even if we develo...
>The likely result of humanity facing down an opposed superhuman intelligence is a total loss. Valid metaphors include “a 10-year-old trying to play chess against Stockfish 15”, “the 11th century trying to fight the 21st century,” and “Australopithecus trying to fight Homo sapiens“.
But obviously these metaphors are not very apt, since humanity kinda has a massive incumbent advantage that would need to be overcome. Rome Sweet Rome is a fun story not because 21st century soldiers and Roman legionnaires are intrinsically equals but because the technologica...
I just want to be clear I understand your "plan".
We are going to build a powerful self-improving system, and then let it try end humanity with some p(doom)<1 (hopefully) and then do that iteratively?
My gut reaction to a plan like that looks like this "Eff you. You want to play Russian roulette, fine sure do that on your own. But leave me and everyone else out of it"
AI will be able to invent highly-potent weapons very quickly and without risk of detection, but it seems at least pretty plausible that...... this is just too difficult
You lack imagination, i...
>As it turns out, the only thing that matters was scale.
I mean, in some sense yes. But AlphaGo wasn't trained by finding a transcript of every Go game that had ever been played, but instead was trained via self-play RL. But attempts to create general game-playing agents via similar methods haven't worked out very well, in my understanding. I don't assume that if we just threw 10x or 100x data at them that this would change...
>The architecture that can play 100 games and does extremely well at game 101 the first try gets way more points than one that ...
>The finished system should be able to extend shoggoth tentacles into a given computer, identify what that computer is doing and make it do it better or differently.
Sure. GPT-X will probably help optimize a lot of software. But I don't think having more resource efficiency should be assumed to lead to recursive self-improvement beyond where we'd be at given a "perfect" use of current software tools. Will GPT-X be able to break out of those current set of tools, only having been trained to complete text and not to actually optimize systems? I don't take this for granted, and my view is that LLMs are unlikely to devise radically new software architectures on their own.
Sure, this is useful. To your other posts, I don't think we're really disagreeing about what AGI is - I think we'd agree that if you took a model with GPT4-like capabilities and hooked it up to a chess API to reinforce it you would end up with a GPT4 model that's very good at playing chess, not something that has strongly-improved its general underlying world model and thus would also be able to say improve its LSAT score. And this is what I'm imaging most self-play training would accomplish... but I'm open to being wrong. To your point about having a "ben...
I'm asking specifically about the assertion that "RL style self play" could be used to iterate to AGI. I don't see what sort of game could lead to this outcome. You can't have this sort of self-play with "solve this math problem" as far as I can tell, and even if you could I don't see why it would promote AGI as opposed to something that can solve a narrow class of math problems.
Obviously LLMs have amazing generalist capabilities. But as far as I can tell you can't iterate on the next version of these models by hooking them up to some sort of API that prov...
RL isn't magic though. It works in the Go case because we can simulate Go games quickly and easily score the results and then pit adversarial AIs against eachother in order to iteratively learn.
I don't think this sort of process lends itself to the sort of tasks that we can only see an AGI accomplishing. You can't train it to say write a better version of Winds of Winter than GRRM could because you don't have a good algorithm to score each iteration.
So what I'm really trying to ask is what specific sort of open ended problems do we see being particularly conducive to fostering AGI, as opposed to a local maximizer that's highly specialized towards the particular problem?
>First problem, A lot of future gains may come from RL style self play (IE:let the AI play around solving open ended problems)
How do people see this working? I understand the value of pointing to AI dominance in Chess/Go as illustrating how we should expect AI to recursively exceed humans at tasks, but I can't see how RL would be similarly applied to "open-ended problems" to promote similar explosive learning. What kind of open problems with a clear and instantly-discernable reward function would promote AGI growth, rather than a more-narrow type of growth geared towards solving the particular problem well?
I agree that there seems to be a lot of handwaving about the nanotech argument, but I can't say that I agree here:
>But for the sake of argument, let's say that the AGI does manage to create a nanotech factory, retain control, and still remain undetected by the humans.
>It doesn't stay undetected long enough to bootstrap and mass produce human replacement infrastructure.
It seems like the idea is that the AI would create nanomachines that it could host itself on while starting to grey goo enough of the Earth to overtake humanity. While human... (read more)