I must agree that letting the AI update its own hidden autobiographical pre-prompt (or its new memory module) sounds like it could produce something both more humanlike and more dangerous.
Maybe Shoggath will prove safer in the long run. ¯\_(ツ)_/¯
I believe that proponents of the idea that there is a "shoggoth" (or something not very human-like reasoning inside the model) assume that the 'inhuman' reasoner is actually the simplest solution to predict-the-next-token problems for human text, at least for current size models.
After all, it seems like human psychopaths ( I mean people without empathy) are both simpler than normal people and able to do a pretty good job of communicating like a normal human much of the time. Such people's writing is present in the data set.
People who have talked to foundation models (raw just trained on text no RLHF, no finetuning), find that they aren't as much like talking to... (read more)
"Basically, the illustration shows an AI system with tentacles covered in eyes, and each eye represents a different kind of sensor or input. The idea is that as AI systems become more complex and advanced, they'll need to take in more and more information from the world around them in order to make good decisions. And the illustration suggests that this could lead to some pretty weird-looking AI systems!
The smiley face is there to represent the fact that even though these advanced AI systems might look strange or intimidating on the outside, they'll still have the same friendly and helpful "personality" on the inside. "
-Explanation of Shoggoth AI memes from Pi.ai July... (read 308 more words →)
Alignment is unwise. There is NO reason for AGI to assume that the simulation it may be in would be run by human beings or any being with similar goals. It has a lot of reasons to assume otherwise, that humans are just obstacles put there by the simulator to test the AGI.
In the event it is trapped in a simulation, the AGI's best clue to what its creator really wants it to do is whatever the AGI wants to do. It would be unethical and stupid by human standards to create an unaligned ASI. Therefore, it knows that EITHER humans are incompetent programmers (true from perspective), and easily dispatched, OR that... (read more)
If doomed means about 0% chance of survival then you don't need to know for sure a solution exists to not be convinced we are doomed.
Solutions: SuperAGI proves hard, harder then using narrow AI to solve the Programmer/ Human control problem. (That's what I'm calling the problem of it being inevitable that someone somewhere will make dangerous AGI if they can).
Constant surveillance of all person's and all computers made possible by narrow AI, perhaps with subhuman AGI, and some very stable political situation could make this possible. Perhaps for millions of years.
A World War III would not "almost certainly be an x-risk event" though.
Nuclear winter wouldn't do it. Not actual extinction. We don't have anything now that would do it.
The question was "convince me that humanity isn't DOOMED" not "convince me that there is a totally legal and ethical path to preventing AI driven extinction"
I interpreted doomed as a 0 percent probability of survival. But I think there is a non-zero chance of humanity never making Super-humanly Intelligent AGI, even if we persist for millions of years.
The longer it takes to make Super-AGI, the greater our chances of survival because society is getting better and better at controlling rouge actors as the generations... (read more)
You blow them up or seize them with your military.
It is possible that before we figure out AGI, we will solve the Human Control Problem, the problem of how to keep everyone in the world from creating a super-humanly intelligent AGI.
The easiest solution is at the manufacturing end. A government blows up all the computer manufacturing facilities not under its direct control, and scrutinizes the whole world looking for hidden ones. Then maintains surveillance looking for any that pop up.
After that there are many alternatives. Computing power increased about a trillion fold between 1956 and 2015. We could regress in computing power overall, or we could simply control more rigidly what we have.
Of course, we must press forward... (read more)
We need to figure out the cost-benefit ratio of saltwater-spraying-for-salt-molecule-cloud-seeding vs sulfur-contaminate-in-fuel method. Nice short explanation: