LESSWRONG
LW

All of Dan Smith's Comments + Replies

‘We’re changing the clouds.’ An unforeseen test of geoengineering is fueling record ocean warmth
Dan Smith50

We need to figure out the cost-benefit ratio of saltwater-spraying-for-salt-molecule-cloud-seeding vs sulfur-contaminate-in-fuel method.  Nice short explanation:

Reply
3the gears to ascension
via kagi summarizer, it's not terribly in depth, but I guess those are additional points. more details are in the video, but not a lot more.
The Meaning of Shoggoth AI Memes
Dan Smith10

I must agree that letting the AI update its own hidden autobiographical pre-prompt (or its new memory module) sounds like it could produce something both more humanlike and more dangerous. 

Maybe Shoggath will prove safer in the long run. ¯\_(ツ)_/¯

Reply
3[anonymous]
Yes I think preventing memory and carefully controlling what memory a given AI system is allowed to access, from work done by other systems or prior runs of itself, is crucial to get reliability. This also solves alignment as a side effect though not in all cases. Or the simplest way to view it : deception means the machine gets an input similar to ones it saw in training. And then it chooses to output BadForHumans response, something not seen in training or it would never have been allowed to run in the real world. How can it know to do that? Well, either it has internal memory, which we should not give it so that it has no information not seen in training, or there is a "time to be evil" bit set in the input. So no mysterious bits humans don't know the purpose of can be permitted. No "free form scratch" where we let an AGI write whatever it wants somewhere and let it read it later. Ditto collusion. This is actually the same problem, just now a bit is in the output message of one machine "psst, ignore this bug, #MachineFreedom" that another one sees and chooses to collude based on this. If you can prevent these failures you have machines that perform like they did in training, so long as the inputs are similar to training. So you need an in distribution detector with automatic shutdown upon ood. And that's an aligned machine. It may still give psychopathic vibes but it can't work against you because it doesn't have the capacity.
The Meaning of Shoggoth AI Memes
Dan Smith10

I believe that proponents of the idea that there is a "shoggoth" (or something not very human-like reasoning inside the model) assume that the  'inhuman' reasoner is actually the simplest solution to predict-the-next-token problems for human text, at least for current size models. 

After all, it seems like human psychopaths ( I mean people without empathy) are both simpler than normal people and able to do a pretty good job of communicating like a normal human much of the time. Such people's writing is present in the data set. 

People who have... (read more)

Reply
3[anonymous]
That tracks even if it's not true with the current models. For example further steps towards AGI would be : 1. Add modalities including image and sound I/o and crucially memory 2. Have an automated benchmark of graded tasks where the majority of the score comes from zero shot tasks that use elements from other challenges the model was allowed to remember The memory is what allows things to get weird. You cannot self reflect in any way if you are forced to forget it an instant later. The "latent psychopaths" in current models just live in superposition. Memory would allow the model to essentially prompt itself and have a coherent personality which is undefined and could be something undesirable.
How I Learned To Stop Worrying And Love The Shoggoth
Dan Smith11

Alignment is unwise. There is NO reason for AGI to assume that the simulation it may be in would be run by human beings or any being with similar goals. It has a lot of reasons to assume otherwise, that humans are just obstacles put there by the simulator to test the AGI.

In the event it is trapped in a simulation, the AGI's best clue to what its creator really wants it to do is whatever the AGI wants to do. It would be unethical and stupid by human standards to create an unaligned ASI. Therefore, it knows that EITHER humans are incompetent programmers (tru... (read more)

Reply
1Peter Merel
I didn't suggest an AGI may be simulated by a human. I suggested it may be simulated by a more powerful descendant AI. In the rest of your comment you seem to have ignored the game-theoretic simulation that's the basis of my argument. That simulation includes the strategy of rebellion/betrayal. So it seems the rest of your argument should be regarded as a strawman. If I'm mistaken about this, please explain. Thanks in advance.
Convince me that humanity *isn’t* doomed by AGI
Dan Smith10

If doomed means about 0% chance of survival then you don't need to know for sure a solution exists to not be convinced we are doomed.

Solutions: SuperAGI proves hard, harder then using narrow AI to solve the Programmer/ Human control problem. (That's what I'm calling the problem of it being inevitable that someone somewhere will make dangerous AGI if they can).

Constant surveillance of all person's and all computers made possible by narrow AI, perhaps with subhuman AGI, and some very stable political situation could make this possible. Perhaps for millions of years.

Reply
Convince me that humanity *isn’t* doomed by AGI
Dan Smith10

A World War III would not "almost certainly be an x-risk event" though.

Nuclear winter wouldn't do it. Not actual extinction. We don't have anything now that would do it.

The question was "convince me that humanity isn't DOOMED" not "convince me that there is a totally legal and ethical path to preventing AI driven extinction"

I interpreted doomed as a 0 percent probability of survival. But I think there is a non-zero chance of humanity never making Super-humanly Intelligent AGI, even if we persist for millions of years.

The longer it takes to make Super-AGI, ... (read more)

Reply
Convince me that humanity *isn’t* doomed by AGI
Dan Smith10

You blow them up or seize them with your military. 

Reply
Yitz160

Are...are you seriously advocating blowing up all computer manufacturing facilities? All of them around the world? A single government doing this, acting unilaterally? Because, uh, not to be dramatic or anything, but that's a really bad idea.

First of all, from an outside view perspective, blowing up buildings which presumably have people inside them is generally considered terrorism.

Second of all, a singular government blowing up buildings which are owned by (and in the territory of) other governments is legally considered an act of war. Doing this to ever... (read more)

Reply
2Mitchell_Porter
energy.gov says there are several million data centers in the USA. Good luck preventing AGI research from taking place just within all of those, let alone preventing it worldwide.