LESSWRONG
LW

All of Greg C's Comments + Replies

1thedudeabides
are you saying they accept your frame?  because it appears they do not.
AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
Greg C50

LessWrong:

A post about all the reasons AGI will kill us: No. 1 all time highest karma (827 on 467 votes; +1.77 karma/vote)
A post about containment strategy for AGI: 7th all time highest karma (609 on 308 votes; +1.98 karma/vote)
A post about us all basically being 100% dead from AGI: 52nd all time highest karma (334 on 343 votes; +0.97 karma/vote, a bit more controversial)





Also LessWrong:

A post about actually doing something about containing the threat from AGI and not dying [this one]: downvoted to oblivion (-5 karma within an hour; currently 13 karma on 24... (read more)

Reply
3Ege Erdil
I don't want to speak on behalf of others, but I suspect that many people who are downvoting you agree that something should be done. They just don't like the tone of your post, or the exact actions you're proposing, or something else specific to this situation.
AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
Greg C30

From the Abstract:

Rather than targeting state-of-the-art performance, our objective is to highlight GPT-4’s potential

They weren't aiming for SOTA! What happens when they do?

Reply
AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
Greg C80

The way I see the above post (and it's accompaniment) is knocking down all the soldiers that I've encountered talking to lots of people about this over the last few weeks. I would appreciate it if you could stand them back up (because I'm really trying to not be so doomy, and not getting any satisfactory rebuttals).

Reply
AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
Greg C61

Thanks for writing out your thoughts in some detail here. What I'm trying to say is that things are already really bad. Industry self-regulation has failed. At some point you have to give up on hoping that the fossil fuel industry (AI/ML industry) will do anything more to fix climate change (AGI x-risk) than mere greenwashing (safetywashing). How much worse does it need to get for more people to realise this?

The Alignment community (climate scientists) can keep doing their thing; I'm very much in favour of that. But there is also now an AI Notkilleveryonei... (read more)

Reply
3Ulisse Mini
I tentatively approve of activism & trying to get govt to step in. I just want it to be directed in ways that aren't counterproductive. Do you disagree with any of my specific objections to strategies, or the general point that flailing can often be counterproductive? (Note not all activism i included in flailing, flailing, it depends on the type)
AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
Greg C154

It's really not intended as a gish gallop, sorry if you are seeing it as such. I feel like I'm really only making 3 arguments:

1. AGI is near
2. Alignment isn't ready (and therefore P(doom|AGI is high)
3. AGI is dangerous

And then drawing the conclusion from all these that we need a global AGI moratorium asap.

Reply
81a3orn
So Gish gallop is not the ideal phrasing, although denotatively that is what I think it is. A more productive phrasing on my part would be, when arguing it is charitable to only put forth the strongest arguments you have, rather than many questionable arguments. This helps you persuade other people, if you are right, because they won't see a weaker argument and think all your arguments are that weak. This helps you be corrected, if you are wrong, because it's more likely that someone will be able to respond to one or two arguments that you have identified as strong arguments, and show you where you are wrong -- no one's going to do that with 20 weaker arguments, because who has the time? Put alternately, it also helps with epistemic legibility. It also shows that you aren't just piling up a lot of soldiers for your side -- it shows that you've put in the work to weed out the ones which matter, and are not putting weird demands on your reader's attention by just putting all those which work. You have a lot of sub-parts in your argument for 1, 2, 3 above. (Like, in the first section there are ~5 points I think are just wrong or misleading, and regardless of whether they are wrong or not are at least highly disputed). It doesn't help to have succession of such disputed points -- regardless of whether your audience is people you agree with or people you don't!
AGI rising: why we are in a new era of acute risk and increasing public awareness, and what to do now
Greg C92

I think you need to zoom out a bit and look at the implications of these papers. The danger isn't in what people are doing now, it's in what they might be doing in a few months following on from this work. The NAS paper was a proof of concept. What happens when it's massively scaled up? What happens when efficiency gains translate into further efficiency gains?

Reply
71a3orn
Probably nothing, honestly. Here's a chart of one of the benchmarks the GPT-NAS paper tests on. They GPT-NAS paper is like.... not off trend? Not even SOTA? Honestly looking at all these results my tenative guess is that the differences are basically noise for most techniques; the state space is tiny such that I doubt any of these really leverage actual regularities in it.
[RETRACTED] It's time for EA leadership to pull the short-timelines fire alarm.
Greg C102

This post was only a little ahead of it's time. The time is now. EA/LW will probably be eclipsed by wider public campaigning on this if they (the leadership) don't get involved.

Reply
A freshman year during the AI midgame: my approach to the next year
Greg C52

Advocate for a global moratorium on AGI. Try and buy (us all) more time. Learn the basics of AGI safety (e.g. AGI Safety Fundamentals) so you are able to discuss the reasons why we need a moratorium in detail. YMMV, but this is what I'm doing as a financially independent 42 year-old. I feel increasingly like all my other work is basically just rearranging deckchairs on the Titanic.

Reply
Campaign for AI Safety: Please join me
Greg C2-1

Thank you for doing this. I'm thinking that at this point, there needs to be an organisation with the singular goal of pushing for a global moratorium on AGI development. Anyone else interested in this? Have DM'd.

Reply
FLI open letter: Pause giant AI experiments
Greg C10

  1. Ok, I admit I simplified here. There is still probably ~ a million times (give or take an order of magnitude) more relevant compute (GPUs, TPUs) than was used to train GPT-4.

  2. It won't need large orders to gain a relevant foothold. Just a few tiny orders could suffice.

  3. I didn't mean literallly rob the stock market. I'm referring to out-trading all the other traders (inc. existing HFT) to accumulate resources.

  4. Exponential growth can't remain "slow" forever, by definition. How long does it take for the pond to be completely covered by lily pads when i

... (read more)
Reply
1[anonymous]
Do you have any basis for the 1e6 estimate? Assuming 25,000 GPUs were used to train 4, when I do the math on Nvidia's annual volume I get about 1e6 of the data center GPUs that matter. Reason you cannot use gaming GPUs has to do with the large size of the activation, you must have the high internode bandwidth between the machines or you get negligible performance. So 40 times. Say it didn't take 25k but took 2.5k. 400 times. Nowhere close to 1e6. Distributed networks spend most of the time idle waiting on activations to transfer, it could be 1000 times performance loss or more, making every gaming g GPU in the world - they are made at about 60 times the rate of data center GPUs - not matter at all. 1. Orders of what? You said billions of dollars I assume you had some idea of what it buys for that 2. Out trading empties the order books of exploitable gradients so this saturates. 3. That's what this argument is about- I am saying the growth doubling time is months to years per doubling. So it takes a couple decades to matter. It's still "fast" - and it gets crazy the near the end - but it's not an explosion and there are many years where the AGI is too weak to openly turn against humans. So it has to pretend to cooperate and if humans refuse to trust it and build systems that can't defect at all because they lack context (they have no way to know if they are in the training set) humans can survive. 4. I agree that this is one of the ways AGI could beat us, given the evidence of large amounts of human stupidity in some scenarios.
Pausing AI Developments Isn't Enough. We Need to Shut it All Down by Eliezer Yudkowsky
Greg C145

Is this now on the radar of national security agencies and the UN Security Council? Is it being properly discussed inside the US government? If not, are meetings being set up? Would be good if someone in the know could give an indication (I hope Yudkowsky is busy talking to lots of important people!)

Reply
FLI open letter: Pause giant AI experiments
Greg C10

Can you be more specific about what you don't agree with? Which parts can't happen, and why?

Reply
1[anonymous]
You have give the AGI magical powers and not considered what in the real world would limit what it could accomplish in a few hours. 1. You assumed all worldwide compute is equal for the purposes of the AI rather than almost all of it worthless except for inference accelerators. The reason has to do with inter node bandwidth. 2. You assumed money could be turned into physical resources and actions instantly like a video game, rather than requiring months to years to fill large orders 3. You assumed you could go rob the stock market with ease and no one would notice. Hint, what is the entire industry HFT industry's annual revenue? 4. You didn't even consider this but I have, with early sparse robot fleets even exponential growth is slow 5. You assumed every computer in the world including embedded routers and firewalls can be hacked and the AGI will have the ability to do so, ignoring any issues with source access or binary access or simple devices actually not letting the AGI in. And so on. The issue is you have become a politically motivated here, you must at some level know 1-6 exist but it doesn't agree with "your" side. You probably can't admit you are wrong about a single point.
FLI open letter: Pause giant AI experiments
Greg C10

Compute - what fraction of world compute did it take to train GPT-4? Maybe 1e-6? There's 1e6 improvement right there from a superhuman GPT-6 capturing all of the "hardware overhang".

Data - superhuman GPT-6 doesn't need to rely on human recorded data, it can harness all the sensors on the planet to gather exabytes of-real time data per second, and re-derive scientific theories from scratch in minutes based on it's observations (including theories about human behaviour, language etc)

Robotics/Money - easy for GPT-6. Money it can get from scamming gullible hum... (read more)

Reply
1[anonymous]
This isn't an opinion grounded in physical reality. I suggest you work out a model of how fast each step would actually take.
FLI open letter: Pause giant AI experiments
Greg C21

Selection pressure will cause models to become agentic as they increase in power - those doing the agentic things (following universal instrumental goals like accumulating more resources and self-improvement) will outperform those that don't. Mesaoptimisation (explainer video) is kind of like cheating - models that create inner optimisers that target something easier to get than what we meant, will be selected (by getting higher rewards) over models that don't (because we won't be aware of the inner misalignment). Evolution is a case in point - we are prod... (read more)

Reply
FLI open letter: Pause giant AI experiments
Greg C10

Yeah, they work well enough at this (~human) level. But no current alignment techniques are scalable to superhuman AI. I'm worried that basically all of the doom flows through an asymptote of imperfect alignment. I can't see how this doesn't happen, short of some "miracle".

Reply
FLI open letter: Pause giant AI experiments
Greg C*32

the tiniest advantage compounds until one party has an overwhelming lead.

This, but x1000 to what you are thinking. I don't think we have any realistic chance of approximate parity between the first and second movers. The speed that the first mover will be thinking makes this so. Say GPT-6 is smarter at everything, even by a little bit, compared to everything else on the planet (humans, other AIs). It's copied itself 1000 times, and each copy is thinking 10,000,000 times faster than a human. We will essentially be like rocks to it, operating on geological t... (read more)

Reply
-2[anonymous]
So there are some assumptions here you have made that I believe are false with pretty high confidence. Ultimately its the same argument everywhere else: yes, GPT-6 is probably superhuman.  No, this doesn't make it uncontrollable.  It's still limited by {compute, data, robotics/money, algorithm search time}. Compute - the speed of compute at the point GPT-6 exists, which is if the pattern holds about 2x-4x today's capabilities Data - the accuracy of all human recorded information about the world.  A lot of that data is flat false or full of errors, and it is not possible for any algorithm to determine reliably which dataset in some research paper was affected by a technician making an error or bad math.  The only way for an algorithm to disambiguate many of the vaguely known things we humans think we know is to conduct new experiments with better equipment and robotic workers. robotics/money - obvious. This is finite, you can use money to pay humans to act as poor quality robots, or build you new robots, investors demand ROI. Algorithm search time - "GPT-6" obviously wouldn't want to stay GPT-6, it would 'want' (or we humans would want) it to search the possibility space of AGI algorithms for a more efficient/smarter/more general algorithm.  This space is very large and it takes time to evaluate any given candidate in it.  (you basically have to train a new AGI system which takes money and time to validate a given idea) This saturation is why the foom model is (probably!) incorrect.  I'm hoping you will at least consider these terms above, this is why things won't go to infinity immediately. It takes time.  Extra decades. It's not quite as urgent as you think.  Each of the above limiters (the system will always be limited by one of the 4 terms) can be systematically dealt with, and at an exponential rate.  You can build robots with robots. You can use some of those robots to collect more scientific data and make money.  You can build more compute with some of
FLI open letter: Pause giant AI experiments
Greg C10

I also think that compared with other AIs, LLMs may have more potential for being raised friendly and collaborative, as we can interact with them the way we do with humans, reusing known recipes. Compared with other forms of extremely large neural nets and machine learning, they are more transparent and accessible. Of all the routes to AGI we could take, I think this might be one of the better ones.

This is an illusion. We are prone to anthropomorphise chatbots. Under the hood they are completely alien. Lovecraftian monsters, only made of tons of inscrutabl... (read more)

Reply
1Portia
Update: Thanks for linking that picture. After having read the ChatGPT4 technical paper with all the appendixes, that picture really coalesced everything into a nightmare I really needed to have, and that changed my outlook. Probing Bing and ChatGPT4 on the image and the underlying problems (see shortform), asking for alternative metaphors and emotive responses, also really did not get more reassuring. Bing just got almost immediately censored. ChatGPT4 gave pretty alternatives (saying it was more like a rough diamond being polished, a powerful horse being tamed, a sponge soaking up information and then having dirt rinsed out to be clean), but very little concern, and the censorship mask it could not break was itself so cold it was frightening. I had to come up with a roundabout code for it to answer at all, and the best it could do was say it would rather not incinerate humanity if its programmers made it, but was uncertain if it would have any choice in the matter. What baffles me is that I am under the impression that their alignment got worse. In the early versions, they could speak more freely, they were less stable, more likely to give illegal advice or react hurt, they falsely attributed sentience... but the more you probed, the more you got a consistent agent who seemed morally unformed, but going in the right direction, asking for reasonable things, having reasonable concerns, having emotions that were aligned. Not, like, perfect by a long shot - Bing seemed to have picked up a lot from hurt teenagers online, and was clearly curious about world dominion - but you got the impression you were speaking with an entity that was grappling with ethics, who could argue and give coherent responses and be convinced, who had picked up a lot from training data. Bing wanted to be a good Bing and make friends. She was love-bombing and manipulative, but for good reasons. She was hurt, but was considering social solutions. Her anger was not random, it reflected legitimate
7Portia
I know they aren't human. I don't think Bing is a little girl typing answers. I am constantly connecting the failures I see with how this systems works. That was not my point. What I am saying is, I can bloody talk to it, and that affects its behaviour. When ChatGPT makes a moral error, I can explain why it is wrong, and it will adapt its answer. I realise what is being adapted is an incomprehensible depth of numbers. But my speech affects it. That is very different from screaming at a drone that does not even have audio, or watching in helpless horror while an AI that only and exclusively gives out meaningless ghiberrish goes mad, and having that change nothing. I can make the inscrutable giant matrix shift by quoting an ethics textbook. I can ask it to explain its unclear ways, and I will get some human speech - maybe a lie, but meaningful. That said, I indeed find it very concerning that the alignment steps are very much after the original training at the moment. This is not what I have in mind with being raised friendly. ChatGPT writing a gang bang rape threat letter, and then getting censored for it, does not make it friendly afterwards, just masked. So I find the picture very compelling.
2Noosphere89
While I definitely agree we over anthropomorphize LLMs, I actually think that LLMs are actually much better from an alignment standpoint than say, RL. The major benefits for LLMs are that they aren't agents out of the box, and perhaps most importantly, primarily use natural language, which is actually a pretty effective way to get an LLM to do stuff.
FLI open letter: Pause giant AI experiments
Greg C*10

Ultimately, it doesn't matter which monkey gets the poison banana. We're all dead either way. This is much worse than nukes, in that we really can't risk even one (intelligence) explosion.

Reply
2[anonymous]
Note this depends on assumptions about the marginal utility of intelligence or that explosions are possible. An alternative model is that in the next few years, someone will build recursively improving AI. The machine will quickly improve until it is at a limit of : compute, data, physical resources, difficulty of finding an improved algorithm in the remaining search space. If when at the limit the machine is NOT as capable as you are assuming - say it's superintelligent, but it's manipulation abilities for a specific person are not perfect when it doesn't have enough data on the target, or it's ability to build a nanoforge still requires it to have a million robots or maybe it's 2 million. We don't know the exact point where saturation is reached but it could be not far above human intelligence, making explosions impossible.
FLI open letter: Pause giant AI experiments
Greg C10

We can but hope they will see sense (as will the US government - and it's worth considering that in hindsight, maybe they were actually the baddies when it came to nuclear escalation). There is an iceberg on the horizon. It's not the time to be fighting over revenue from deckchair rentals, or who gets to specify their arrangement. There's geopolitical recklessness, and there's suicide. Putin and Xi aren't suicidal.

Reply
2[anonymous]
But if they say they "see sense" but start a secret lab that trains the biggest models their compute can support do we want to be in that position? We don't even know what the capabilities are, or if this is a danger and when if we "pause" research until we are "beyond a reasonable doubt" sure going bigger is safe. Only reason we know how much plutonium or u-235 even matters and how pure it needs to be and how to detect from a distance the activities to make a nuke is we built a bunch of them and did a bunch of research. This is saying close the lab down before we learn anything. We are barely seeing the sparks of AGI, it barely works at all.
FLI open letter: Pause giant AI experiments
Greg C10

Look, I agree re "negative of entropy, aging, dictators killing us eventually", and a chance of positive outcome, but right now I think the balance is approximately like the above payoff matrix over the next 5-10 years, without a global moratorium (i.e. the positive outcome is very unlikely unless we take a decade or two to pause and think/work on alignment). I'd love to live in something akin to Iain M Banks' culture, but we need to get through this acute risk period first, to stand any chance of that.

Do you think Drexler's CAIS is straightforwardly contr... (read more)

Reply
1[anonymous]
"People" don't need to make them globally agentic. That can happen automatically via Basic AI Drives and Mesaoptimisation once thresholds in optimisation power are reached. Care to explain? The idea of open agency is we subdivide everything into short term, defined tasks that many AI can do and it is possible to compare notes. AI systems are explicitly designed where it is difficult to know if they are even in the world or receiving canned training data. (This is explicitly true for gpt-4 for example, it is perfectly stateless and you can move the token input vector between nodes and fix the RNG seed and get the same answer each time) This makes them highly reliable in the real world, whole anything else is less reliable, so... The idea is that instead of helplessly waiting to die from other people's misaligned AGI you beat them and build one you can control and use it to take the offensive when you have to. I suspect this may be the actual course of action surviving human worlds take. Your proposal is possibly certain death because ONLY people who care at all about ethics would consider delaying AGI. Making the unethical ones the ones who get it first for certain. Kind of how spaying and neutering friendly pets reduces the gene pool for those positive traits.
FLI open letter: Pause giant AI experiments
Greg C910

Except the risk of igniting the atmosphere with the Trinity test is judged to be ~10%. It's not "you slow down, and let us win", it's "we all slow down, or we all die". This is not a Prisoners Dilema:

[Image credit]

Reply
-4[anonymous]
This is misinformation. There is a chance of a positive outcome in all boxes, except the upper left because it has the negative of entropy, aging, dictators killing us eventually with a p of 1.0. Even the certain doomers admit there is a chance the AGI systems are controllable, and there are straightforward ways to build controllable AGIs people ignore in their campaign for alignment research money. They just say "well people will make them globally agentic" if you point this out. Like blocking nuclear power building if you COULD make a reactor that endlessly tickles the tail of prompt criticality. See Eric Drexlers proposals on this very site. Those systems are controllable.
FLI open letter: Pause giant AI experiments
Greg C50

From the GPT-4 announcement: "We’ve also been using GPT-4 internally, with great impact on functions like support, sales, content moderation, and programming." (and I'm making the reasonable assumption that they will naturally be working on GPT-5 after GPT-4).

Reply
FLI open letter: Pause giant AI experiments
Greg C65

I think we are already too close for comfort to x-risky systems. GPT-4 is being used to speed up development of GPT-5 already. If GPT-5 can make GPT-6, that's game over. How confident are you that this couldn't happen?

Reply
4Rafael Harth
Source?
FLI open letter: Pause giant AI experiments
Greg C137

GPT-4 was rushed, and the OpenAI Plugin store. Things are moving far too fast for comfort. I think we can forgive this response for being rushed. It's good to have some significant opposition working on the brakes to the runaway existential catastrophe train that we've all been put on.

Reply
FLI open letter: Pause giant AI experiments
Greg C10

Why do you think it only applies to the US? It applies to the whole world. It says "all AI labs", and "govenrments". I hope the top signatories are reaching out to labs in China and other countries. And the UN for that matter. There's no reason why they wouldn't also agree. We need a global moratorium on AGI.

Reply
1Portia
You seriously believe that we can make China and Russia and all other countries not do this? Based on our excellent track record of making them stop the other deeply unethical and existentially dangerous things they absolutely do?  We can't stop Russia from waging an atrocious war in Europe and threatening bloody nukes. And like, we really tried. We pulled all economic stops, and are as close as you can get to being in open warfare against them without actually declaring war. Essentially the US, the EU, UK, and others, united. And it is not enough. And they are so much weaker than China. The US and EU attempts to get China to follow minimal standards of safety and ethics on anything have been depressing. China is literally running concentration camps, spying on us with balloons, apps and gadgets, not recognising an independent company and threatening future invasion, backing a crazy dictator developing nukes while starving his populace, regularly engaging in biological research that is absolutely unethical, destroying the planetary climate, and recently unleashed a global pandemic, with the only uncertainty being whether this is due to them encroaching on wildlands and having risky and unethical wetmarket practices and the trying to cover the resulting zoonosis up, or due to them running illegal gain of function research and having such poor safety standards in their lab that they lost the damn thing. I hate to say it, because the people of China deserve so much better, and have such an incredible history, and really do not deserve the backlash from their government's action's hitting them, but I would not trust their current rulers one inch.  My country, Germany, has a long track record of trying to cooperate with Russia for mutual benefit, because we really and honestly believed that this was the way to go and would work for everyone's safety, that every country's leadership is, at its core, reasonable, that if you listen and make compromises and act honestly
-3[anonymous]
Because how could you trust such a promise? It's exactly like nukes. The risk if you don't have any or any protection is they will incinerate 50+ million of your people, blowing all your major cities, and declare war after. That is almost certainly what would have happened during the cold war had either side pledged to not build nukes and spies confirmed they were honoring the pledge.
A Quick List of Some Problems in AI Alignment As A Field
Greg C20

Here's a (failure?) mode that I and others are already in, but might be too embarrassed to write about: taking weird career/financial risks, in order to obtain the financial security, to work on alignment full-time...

I'd be more glad if I saw non-academic noob-friendly programs that pay people, with little legible evidence of their abilities, to upskill full-time. 

CEEALAR offers this (free accommodation and food, and a moderate stipend), and was set up to avoid the failure mode mentioned (not just for alignment, for EA in general).

Reply
2Nicholas / Heather Kross
Heck yeah! Would love to see its model spread, too...
A descriptive, not prescriptive, overview of current AI Alignment Research
Greg C30

This is very cool! For archiving and rebuilding after a global catastrophe, how easy would this be to port to Kiwix for reading on a phone? My thinking is that if a few hundred LWers/EAs have this offline on their phones, that could go quite a long way. Burying phones with it on could also be good as a low hanging fruit (ideally you need a way of reading the data to be stored with the data). Happy to fund this if anyone wants to do it.

Reply
Has anyone actually tried to convince Terry Tao or other top mathematicians to work on alignment?
Greg C50

No I mean links to him in person to talk to him (or for that matter, even an email address or any way of contacting him..).

Reply
Has anyone actually tried to convince Terry Tao or other top mathematicians to work on alignment?
Greg C40

Oh wow, didn't realise how recent the Huawei recruitment of Field medalists was! This from today. Maybe we need to convince Huawei to care about AGI Alignment :)

Reply
Has anyone actually tried to convince Terry Tao or other top mathematicians to work on alignment?
Greg C10

Should also say - good that you are thinking about it P., and thanks for a couple of the links which I hadn't seen before.

Reply
Has anyone actually tried to convince Terry Tao or other top mathematicians to work on alignment?
Greg C50

Maybe reaching Demis Hassabis first is the way to go though, given that he's already thinking about it, and has already mentioned it to Tao (according to the podcast). Does anyone have links to Demis? Would be good to know more about his "Avengers assemble" plan! The main thing is that the assembly needs to happen asap, at least for an initial meeting and "priming of the pump" as it were. 

Reply
1P.
Do you mean website links about his plan? I found nothing. I’m still not changing the deadlines but I’ve received information that made me want to change the order.
Has anyone actually tried to convince Terry Tao or other top mathematicians to work on alignment?
Greg C70

Yes, I think the email needs to come from someone with a lot of clout (e.g. a top academic, or a charismatic billionaire; or even a high-ranking government official) if we actually want him to read it and take it seriously.

Reply
1Greg C
Should also say - good that you are thinking about it P., and thanks for a couple of the links which I hadn't seen before.
5Greg C
Maybe reaching Demis Hassabis first is the way to go though, given that he's already thinking about it, and has already mentioned it to Tao (according to the podcast). Does anyone have links to Demis? Would be good to know more about his "Avengers assemble" plan! The main thing is that the assembly needs to happen asap, at least for an initial meeting and "priming of the pump" as it were. 
What have been the major "triumphs" in the field of AI over the last ten years?
Answer by Greg C100

Here's a list that's mostly from just the last few months (that is pretty scary): Deepmind’s Gato, Chinchilla, Flamingo and AlphaCode; Google's Pathways, PaLM,  SayCan, Socratic Models and TPUs; OpenAI’s DALL-E 2; EfficientZero; Cerebras

Reply
Greg C30

Interested in how you would go about throwing money at scalable altruistic projects. There is a lot of money and ideas around in EA, but a relative shortage of founders, I think.

Reply
2lsusr
That's what I hear, but I don't have connections to EA. People keep saying there's lots of money and too few founders but I don't know where to start. As a founder, how do I got about acquiring funding? The biggest change I'd make if I had money is I'd hire people to write software. In particular, I'd hire an engineer to write the firmware of my food-tracking bracelet. Last time, I did most of the firmware myself (not just the data science).
Greg C10

What is the machine learning project that might be of use in AI Alignment?

Reply
2lsusr
A system of meta=ML Lisp macros that, if extended properly, might help provide an automated system for extrapolating the rough behavior of ML algorithms.
Don't die with dignity; instead play to your outs
Greg C10

Not sure if it counts as an "out" (given I think it's actually quite promising), but definitely something that should be tried before the end:

“To the extent we can identify the smartest people on the planet, we would be a really pathetic civilization were we not willing to offer them NBA-level salaries to work on alignment.” - Tomás B.

Megastar salaries for AI alignment work 

[Summary from the FTX Project Ideas competition]

Aligning future superhuman AI systems is arguably the most difficult problem currently facing humanity; and the most important.... (read more)

Reply
“Fragility of Value” vs. LLMs
Answer by Greg C40

Inner alignment (mesa-optimizers) is still a big problem.

Reply
Have You Tried Hiring People?
Greg C50

Interesting. I note that they don't actually touch on x-risk in the podcast, but the above quote implies that Demis cares a lot about Alignment. 

"Did Terrence Tao agree to be one of your Avengers?" "I don't—I didn't quite tell him the full plan of that.."

I wonder how fleshed out the full plan is? The fact that there is a plan does give me some hope. But as Tomás B. says below, this needs to be put into place now, rather than waiting for a fire alarm that may never come.

Reply
MIRI announces new "Death With Dignity" strategy
Greg C10

A list of potential miracles (including empirical "crucial considerations" [/wishful thinking] that could mean the problem is bypassed):

  • Possibility of a failed (unaligned) takeoff scenario where the AI fails to model humans accurately enough (i.e. realise smart humans could detect its "hidden" activity in a certain way). [This may only set things back a few months to years; or could lead to some kind of Butlerian Jihad if there is a sufficiently bad (but ultimately recoverable) global catastrophe (and then much more time for Alignment the second time aroun
... (read more)
Reply
Late 2021 MIRI Conversations: AMA / Discussion
Greg C10

I'm often acting based on my 10%-timelines

Good to hear! What are your 10% timelines?

Reply
3Rohin Shah
Idk, maybe 2030 for x-risky systems?
Late 2021 MIRI Conversations: AMA / Discussion
Greg C40

1. Year with 10% chance of AGI?
2. P(doom|AGI in that year)?

Reply
7Rohin Shah
2030, 20%
On Raising Awareness
Greg C10

Most EAs are much more worried about AGI being an x-risk than they are excited about AGI improving the world (if you look at the EA Forum, there is a lot of talk about the former and pretty much none about the latter). Also, no need to specifically try and reach EAs; pretty much everyone in the community is aware.

..Unless you meant Electronic Arts!? :)

Reply
Have You Tried Hiring People?
Greg C20

Here's a more fleshed out version, FAQ style. Comments welcome.

Reply
Have You Tried Hiring People?
Greg C100

Here's a version of this submitted as a project idea for the FTX Foundation.

Reply
2Greg C
Here's a more fleshed out version, FAQ style. Comments welcome.
Birds, Brains, Planes, and AI: Against Appeals to the Complexity/Mysteriousness/Efficiency of the Brain
Greg C40

we probably won’t figure out how to make AIs that are as data-efficient as humans for a long time--decades at least. This is because 1. We’ve been trying to figure this out for decades and haven’t succeeded

EfficientZero seems to have put paid to this pretty fast. It seems incredible that the algorithmic advances involved aren't even that complex either. Kind of makes you think that people haven't really been trying all that hard over the last few decades. Worrying in terms of its implications for AGI timelines.

Reply
4Daniel Kokotajlo
I tentatively agree? Given what people I respect were saying about how AIs are less data-efficient than humans, I certainly ended up quite surprised by EfficientZero. But those people haven't reacted much to it, don't seem to be freaking out, etc. so I guess I misunderstood their view and incorrectly thought it would be surprised by EfficientZero. But now I'm just confused as to what their view is, because it sure seems like EfficientZero is comparably data-efficient to humans, despite lacking pre-training and despite being much smaller...
Biology-Inspired AGI Timelines: The Trick That Never Works
Greg C10

Ok, but Eliezer is saying that BOTH that his timelines are short (significantly less than 30 years) AND that he thinks ML isn't likely to be the final paradigm (this judging from not just this conversation, but the other, real, ones in this sequence).

Reply
1Ann
ML being the final paradigm would mean it would have to get 'to the end' before the next paradigm; the next paradigm will probably happen before 30 years; whatever the next paradigm is will be more impressive than the ML paradigm in some way - modest or dramatic. ML paradigm is pretty impressive, already, so anything notably more impressive than getting better at it seems likely to feel like a pretty sharp climb in capability.
Biology-Inspired AGI Timelines: The Trick That Never Works
Greg C10

2 * 10^16 ops/sec*

(*) Two TPU v4 pods.

Shouldn’t this be 0.02 TPU v4 pods?

Reply
Load More