All of lorepieri's Comments + Replies

For an argument against the sim hypothesis see https://lorenzopieri.com/sim_hypothesis/  or the full article https://philpapers.org/rec/PIETSA-6  (The Simplicity Assumption and Some Implications of the Simulation Argument for our Civilization). 

In a nutshell:

0- Suppose by absurd that we are in a simulation. 
1- We are equally likely to be in one of the many simulations.
2- The vast majority of simulations are simple [see paper to understand why this is reasonable].
3- Therefore, we are very likely to be in a simple simulation.
4- Therefore,... (read more)

Hi Clement, I do not have much to add to the previous critiques, I also think that what needs to be simulated is just a consistent enough simulation, so the concept of CI doesn't seem to rule it out.  

You may be interested in a related approach ruling out the sim argument based on computational requirements, as simple simulations should be more likely than complex one, but we are pretty complex. See "The Simplicity Assumption and Some Implications of the Simulation Argument for our Civilization" (https://philarchive.org/rec/PIETSA-6)

Cheers!

Yes voter, if you can read this: why? It would be great to get an explanation (anon).

Damn, we did not last even 24hrs...  

Thanks for the alternative poll.  One would think that with rules 2 and 5 out of the way it should be harder to say Yes. 

How confident are you that someone is going to press it? If it's pressed: what's the frequency of someone pressing it? What can learn from it? Does any of the rules 2-5 play a crucial role in the decision to press it? 

(we are still alive so far!)

4Ustice
I was more or less going to say the same thing. No, I wouldn’t press the button except in the most extremely bad scenarios I can imagine. As for how confident I am in that, I’m pretty tempted to say certain. Whether it is due to nihilistic glee, curiosity, clumsiness, or sheer stupidity, that button is going to be pressed. Now, there are scenarios that I can imagine that delay things for a human-significant amount of time. Factors that I can think of right now that would expand the timeline: * ease of access to a doom button * cost to access a doom button * time of access for a doom button * intentionality verification requirements to press a doom button * societal and cultural significance of the doom button * scale of knowledge of * doom buttons * how to access doom buttons * how to operate doom buttons I do live in Florida, so my estimates may be atypical. I would estimate that for about 100 000 people the chance of someone pressing a doom button sitting right in front of them with full instructions on any given day would be around 1:100 for odds. A roll of a d100 sounds about right there. So that’s 1.00e-8 per person, per day. Using 8 billion as the world population comparing, the magnitudes of 1.00e-8 and 8.00e9, the population is going to swamp the odds of a doom button being pressed in a short time.
2Seth Herd
Out of 8 billion people? Very near 100%. Of your handful of respondents? I have no idea, nor why I should care. I'm asking you what we can learn from it, because you clearly have an idea. If you want to wait for the poll to fill before saying more, that's fine.

This is a pretty counter-intuitive point indeed, but up to a certain threshold this seems to me the approach that minimise risks, by avoiding large capability jumps and improving the "immune system" of society. 

1RogerDearnaley
At current open source model's risk levels, I completely agree. Obviously it's hard to know (from outside OpenAI) how bad open-sourced unfiltered GTP-4 would be, but my impression is that that also isn't capable of being seriously dangerous, so I suspect the same might be true there, and I agree that adapting to it may "help society's immune system" (after rather a lot of spearhishing emails, public opinion-manipulating propaganda, and similar scams). [And I don't see propaganda as a small problem: IMO the rise of Fascism that led to the Second World War was partly caused by it taking society a while to adjust to the propaganda capabilities of radio and film (those old propaganda films that look so hokey to us now used to actually work), and the recent polarization of US politics and things like QAnon and InfoWars I think have a lot to do with the same for social media.] So my "something more dangerous than atomic energy" remark above is anticipatory, for what I expect from future models such as GPT-5/6 if they were open sourced unfiltered/unaligned. So I basically see two possibilities: 1. At some point (my current guess would be somewhere around the GPT-5 level), we're going to need to stop open-sourcing these, or else if we don't unacceptable amounts of damage will be done, unless 2. We figure out a way of aligning models that is "baked in" during the pretraining stage or some other way, and that cannot then be easily fine-tuned out again using 1% or less of the amount of compute needed to pretrain the model in the first place. Filtering dangerous information out of the pretraining set might be an example of a candidate, some form of distillation process of an aligned model that actually managed to drop unaligned capabilities might be another.

Thanks for the insightful comment. Ultimately the different attitude is about the perceived existential risk posed by the technology and the risks coming by acting on accelerating AI vs not acting. 

And yes I was expecting not to find much agreement here, but that's what makes it interesting :) 

A somewhat similar statistical reasoning can be done to argue that the abundance of optional complexity (things could have been similar but simpler) is evidence against the simulation hyphotesis.

See https://philpapers.org/rec/PIETSA-6  (The Simplicity Assumption and Some Implications of the Simulation Argument for our Civilization)

This is based on the general principle of computational resources being finite for any arbitrary civilisations (assuming infinities are not physical) and therefore minimised when possible by the simulators. In particular one... (read more)

Let’s start with one of those insights that are as obvious as they are easy to forget: if you want to master something, you should study the highest achievements of your field.

Even if we assume this, it does not follow that we should try to recreate the subjective conditions that led to (perceived) "success".  The environment is always changing (tech, knowledge base, tools), so many learnings will not apply.  Moreover, biographies tend to create a narrative after the fact, emphasizing the message the writer want to convey. 

I prefer the strategy to master the basics from previous works and then figure out yourself how to innovate and improve the state of the art.

Using the Universal Distribution in the context of the simulation argument makes a lot of sense if we think that the base reality has no intelligent simulators, as it fits with our expectations that a randomly generated simulator is very likely to be coincise. But for human (or any agent-simulators) generated simulations, a more natural prior is how easy is the simulation to be run (Simplicity Assumption), since agent-simulators face concrete tradeoffs in using computational resources, while they have no pressing tradeoffs on the length of the program.&nbs... (read more)

This is also known as Simplicity Assumption: "If we randomly select the simulation of a civilization in the space of all possible simulations of that civilization that have ever been run, the likelihood of picking a given simulation is inversely correlated to the computational complexity of the simulation."

In a nutshell, the amount  of computation needed to perform simulations matters (if resources are somewhat finite in base reality, which is fair to imagine), and over the long  term simple simulations will dominate the space of sims.

See here fo... (read more)

Regarding (D), it has been elaborated more in this paper (The Simplicity Assumption and Some Implications of the Simulation Argument for our Civilization).

I would suggest to remove "I dont think you are calibrated properly about the ideas that are most commonly shared in the LW community. " and present your argument, without speaking for the whole community. 

Very interesting division, thanks for your comment. 

Paraphrasing what you said,  in the informational domain we are very close to post scarcity already (minimal effort to distribute high level education and news globally), while in the material and human attention domain we likely still need advancements in robotics and AI to scale.

You mean the edit functionality of Gitlab? 

Thanks for the gitbook tip, I will look into it.

Yes, the code is open source: https://gitlab.com/postscarcity/map

Interesting paradox. 

As other commented, I see multiple flaws:

  1. We believe to seem to know that there is a reality that exists. I doubt we can conceive reality, but only a vague understanding of it. Moreover we have no experience of "not existing", so it's hard to argue that we have a strong grasp on deeply understanding that there is a reality that exists.
  2. Biggest issue is here imho  (this is a very common misunderstanding): math is just a tool which we use to describe our universe, it is not (unless you take some approach like the mathematical uni
... (read more)

Not conclusive, but still worth doing in my view due to the relative easiness. Create the spreadsheet, make it public and let's see how it goes.

I would add the actual year in which you think it will happen.

Yea, what I meant is that the slides of Full Stack Deep Learning course materials provide a decent outline of all of the significant architectures worth learning.

I would personally not go to that low level of abstraction (e.g. implementing NNs in a new language) unless you really feel your understanding is shaky.  Try building an actual side project, e.g. an object classifier for cars, and problems will arise naturally.

1Kenny
Wonderful – I'll keep that in mind when I get around to reviewing/skimming that outline. Thanks for sharing it. I have a particularly idiosyncratic set of reasons for the particular kind of 'yak shaving' I'm thinking of, but your advice, i.e. to NOT do any yak shaving, is noted and appreciated.

I fear that measuring modifications it's like measuring a moving target. I suspect it will be very hard to consider all the modifications, and many AIs may blend each other under large modifications.  Also it's not clear how hard some modifications will be without actually carrying out those modifications.

Why not fixing a target, and measuring the inputs needed (e.g. flops, memory, time) to achieve goals? 

I'm working on this topic too, I will PM you.  

Also feel free to reach out if topic is of interest.

1JasonBrown
Yes, it's still unclear how to measure modification magnitude in general (or if that's even possible to do in a principled way) but for modifications which are limited to text, you could use the entropy of the text and to me that seems like a fairly reasonable and somewhat fundamental measure (according to information theory). Thank you for the references in your other comment, I'll make sure to give them a read!

Other useful references:

-On the Measure of Intelligence https://arxiv.org/abs/1911.01547 

-S. Legg and M. Hutter, A collection of definitions of intelligence, Frontiers in Artificial Intelligence and applications, 157 (2007), 

-S. Legg and M. Hutter, Universal intelligence: A definition of machine intelligence, Minds and Machines, 17 (2007), pp. 391-444.  https://arxiv.org/pdf/0712.3329.pdf 

-P. Wang, On Defining Artificial Intelligence, Journal of Artificial General Intelligence, 10 (2019), pp. 1-37.

-J. Hernández-Orallo, The measure of al... (read more)

This is the most likely scenario, with AGI getting heavily regulated, similarly to nuclear. It doesn't get much publicity because it's "boring". 

The 1 million prize problem should be "clearly define the AI alignement problem". I'm not even joking, actually understanding the problem and enstablising that there is a problem in the first place may give us hints to the solution.

1Esben Kran
Yes, defining the challenge also seems to get us 90%< there already. A Millennium prize has a possibility of being too vague compared to the others.

In research there are a lot of publications, but few stand the test of time. I would suggest to you to look at the architectures which brought significant changes and ideas, those are still very relevant as they:

- often form the building block of current solutions

- they help you build intuition on how architectures can be improved

- it is often assumed in the field that you know about them

- they are often still useful, especially when having low resources

You should not need to look at more than 1-2 architectures per year in each field (computer vision, NLP,... (read more)

1Kenny
Thanks for the reply! This seems helpful and, I think, matches what I expected might be a good heuristic. I'm not sure I know how to identify "the architectures which brought significant changes and ideas" – beyond what I've already been doing, i.e. following some 'feeds' and 'skimming headlines' with an occasional full read of posts like this. What would you think about mostly focusing on SOTA and then, as needed, and potentially recursively, learning about the 'prior art' on which the current SOTA is built/based? Or does the "Full Stack Deep Learning" course materials provide a (good-enough) outline of all of the significant architectures worth learning about? A side project I briefly started a little over a year ago, but have since mostly abandoned, was to re-implement the examples/demos from the Machine Learning course I took. I found the practical aspect to be very helpful – it was also my primary goal for taking the course; getting some 'practice'. Any suggestions about that for this 'follow-up survey'? For my side project, I was going to re-implement the basic models covered by that first course in a new environment/programming-language, but maybe that's too much 'yak shaving' for a broad survey.

ARC is a nice attempt. I also participated in the original challenge on Kaggle. The issue is that the test can be gamed (as anyone on Kaggle did) brute forcing over solution strategies. 

An open-ended or interactive version of ARC may solve this issue.

I'm working on these lines to create an easy to understand numeric evaluation scale for AGIs. The dream would be something like: "Gato is AGI level 3.5, while the average human is 8.7." I believe the scale should factor in that no single static test can be a reliable test of intelligence (any test can be gamed and overfitted).

A good reference on the subject is "The Measure of All Minds"  by Orallo. 

Happy to share a draft, send me a DM if interested.

2Davidmanheim
I think that building compound metrics here is just another way to provide something for people to Goodhart - but I've written much more about the pros and cons of different approaches elsewhere, so I won't repeat myself here.

When you say "switching" it reminds me of the "big switch" approach of https://en.wikipedia.org/wiki/General_Problem_Solver.

Regarding to how they do it, I believe the relevant passage to be:

Because distinct tasks within a domain can share identical embodiments, observation formats and action specifications, the model sometimes needs further context to disambiguate tasks. Rather than providing e.g. one-hot task identifiers, we instead take inspiration from (Brown et al., 2020; Sanh et al., 2022; Wei et al., 2021) and use prompt conditioning.

I guess it shoul... (read more)

Fair analysis, I agree with the conclusions. The main contribution seems to be a proof that transformers can handle many tasks at the same time. 

Not sure if you sorted the tests in order of relevance, but I also consider the "held-out" test as being the more revealing. Besides finetuning, it would be interesting to test the zero-shot capabilities.

3Oliver Sourbut
I didn't methodically order the experiment ideas, but they are meant to be roughly presented in order of some combination of concreteness/tractability and importance. What do you think of my speculation about the tagging/switching/routing internal mechanism?

A single network is solving 600 different tasks spanning different areas. 100+ of the tasks are solved at 100% human performance. Let that sink in. 

While not a breaktrough in arbitrary scalable generality, the fact that so many tasks can be fitted into one architecture is surprising and novel. For many real life applications, being good in 100-1000 tasks makes an AI general enough to be deployed as an error tollerant robot, say in a warehouse. 

The main point imho is that this architecture may be enough to be scaled (10-1000x parameters) in few years to a useful proto-AGI product.

2alexlyzhov
But the mere fact that one network may be useful for many tasks at once has been extensively investigated since 1990s.

What's stopping the companies from hiring a new researcher? People are queueing for tech jobs.

5sovran
Researchers aren't exactly fungible; replacing a skilled researcher with a new hire would still slow down progress. given how many people want to help, but have no idea how to help, this is a gap in the market worth filling. 
2sovran
Not everyone concerned about safety is looking to leave. The concerned have three options: stay and try to steer towards safety, continue moving on the current trajectory, or just leave. Helping some of those who’ve changed their mind about capabilities gain actually get out is only a net negative if those people staying in the field would’ve changed the trajectory of the field. I simply don’t think that everyone should try help by staying and trying to change. There is absolutely room for people to help by just leaving, and reducing the amount of work going into capabilities gain. Different paths will make sense for different people. There’s space to support both those researchers who’re trying to steer towards safety, and those who just want out. I’ve seen a lot of work towards the former, but almost none towards the latter. I want to speak to concerned researchers so that I can begin to better understand which individual researchers should indeed just leave, and which should stay. I really don’t think the problem is overdetermined.

If by "sort of general, flexible learning ability that would let them tackle entirely new domains" we include adding new tokenised vectors in the training set, then this fit the definition. Of course this is "cheating" since the system is not learning purely by itself, but for the purpose of building a product or getting the tasks done this does not really matter. 

And it's not unconcievable to imagine self-supervised tokens generation to get more skills and perhaps a K-means algorithm to make sure that the new embeddings do not interfere with previous knowledge. It's a dumb way of getting smarter, but apparently it works thanks to scale effects!

I would agree with "proto-AGI". I might soon write a blog on this, but ideally we could define a continuous value to track how close we are to AGI, which is increasing if:

-the tasks to solve are very different from each other

-the tasks are complex

-how well a task have been solved

-few experience (or info) is fed to the system

-experience is not directly related to the task

-experience is very raw

-computation is done in few steps

Then adding new tasks and changing the environment.

I have always been cautios, but I would say yes this time. 

With the caveat that it learns new tasks only from supervised data, and not reusing previous experience.

5Yuli_Ban
So perhaps a "proto-AGI" is a better term to use for it. Not quite the full thing just yet, but shows clear generality across a wide number of domains. If it can spread out further and become much larger, as well as have recursivity (which might require an entirely different architecture), it could become what we've all been waiting for.

The fact that adding new tasks doesn't diminuish performance on previous tasks is highly non trivial!

It may be that there is a lot of room in the embedding space to store them. The wild thing is that nothing (apart few hardware iterations) stop us to increase the embedding space if really needed.

Possibly the first truly AGI paper. 

Even though it is just exploiting the fact that all the narrow problems can be solved as sequence problems via tokenisation, it's remarkable that the tasks do not interferee distructively between each other. My gut feeling is that this is due the very high dimensional space of the embedding vectors.

It leaves ample room for grow.

My main point is that there is not enough evidence for a strong claim like doom-soon. In absence of hard data anybody is free to cook up argument pro or against doom-soon. 

You may not like my suggestion, but I would strongly advise to get deeper into the field and understand it better yourself, before taking important decisions.

In terms of paradigms, you may have a look at why building AI-software development is hard (easy to get to 80% accurate, hellish to get to 99%),  AI-winters and hype cycles (disconnect between claims-expectations and reality), the development of dangerous technologies (nuclear, biotech) and how stability has been achieved.

Don't look at opinions, look for data and facts.  Speculations, opinions or beliefs cannot be the basis on which you take decisions or update your knowledge. It's better to know few things, but with high confidence. 

Ask yourself, which hard data points are there in favour of doom-soon? 

Facts and data are of limited use without a paradigm to conceptualize them. If you have some you think are particularly illuminative though by all means share them here.

Geniuses or talented researchers are not that impactful as much as the right policy.  Contribute creating the right conditions (work environment, education, cross contamination, funding, etc.) to make good research flourish.  At the same time if fundamentals are not covered (healthcare, housing, etc.) people are not able to focus on much more than suvival. So pretty much anything that makes the whole system works better helps.

As an example, there are plenty of smart individuals in poor counties which are not able to express their potential. 

1Alex K. Chen (parrot)
Yeah this is from the top-down (and doing things from the top-down requires jumping through hoops). People at Foresight care less about these hoops than people elsewhere(+super-open to ppl from weird backgrounds), and Foresight is WAY higher-impact than most organizations. Bottom-up: there are SOME people I know who hang out with AI people and who understand them inside from the bottom-up, and this is SOMETHING that can help

Thanks.  Yes, pretty much in line with the authors. Btw, I would super happy to be wrong and see advancement in those areas, especially the robotic one.

 Thanks for the offer, but I'm not interested in betting money. 

A close call, but I would lean still on no. Engineering the prompt is where humans leverage all their common sense and vast (w.r.t.. the AI) knowledge. 

The bottom line is: nobody has a strong argument in support of the inevitability of the doom scenario (If you have it, just reply to this with a clear and self contained argument.). 

From what I'm reading in the comments and in other papers/articles, it's a mixture of beliefs, estrapolations from known facts, reliance on what "experts" said, cherry picking. Add the fact that bad/pessimistic news travel and spread faster than boring good news.

A sober analysis enstablish that super-AGI can be dangerous (indeed there are no theorems forbidding this either... (read more)

6Leo P.
Because unchecked convergent instrumental goals for AGI are already in contrast with humanity goals. As soon as you realize humanity may have reasons to want to shut down/restrain an AGI (through whatever means), this gives ground to the AGI to wipe humanity.

The downvotes are excessive, the post is provoking,  but interesting.

I think you will not even need to "push the fat man". The development on an AGI will be slow and gradual (as any other major technology) and there will be incidents along the way (e.g. an AGI chatbot harassing someone). Those incidents will periodically mandate new regulations, so that measurements to tackle real AGI related dangers will be enacted, similarly to what happens in the nuclear energy sector.  They will not be perfect, but there will be regulations.

The tricky part is... (read more)

Matthew, Tamay: Refreshing post, with actual hard data and benchmarks. Thanks for that.

My predictions:

  • A model/ensemble of models achieves >80% on all tasks in the MMLU benchmark

No in 2026, no in 2030. Mainly due to the fact that we don't have much structured data and incentives to solve some of the categories. A powerful unsupervised AI would be needed to clear those categories, or more time.

  • A credible estimate reveals that an AI lab deployed EITHER >10^30 FLOPs OR hardware that would cost $1bn if purchased through competitive cloud computing ve
... (read more)
2FeepingCreature
How much would your view shift if there was a model that could "engineer its own prompt", even during training?
3Matthew Barnett
The criteria adjusts for inflation.
2Nathan Helm-Burger
Nice specific breakdown! Sounds like you side with the authors overall. Want to also make the 3:1 bet with me?

This is a possible AGI scenario, but it's not clear why it should be particularly likely. For instance the AGI may reason that going aggressive will also be the fastest route to be terminated. Or the AGI may consider that keeping humans alive is good, since they were responsable for the AGI creation in the first place. 

What you describe is the paper-clip maximiser scenario, which is arguably the most extreme end of the spectrum of super-AGI behaviours.

2johnlawrenceaspden
  Absolutely! It may want to go aggressive, but reason that its best plan is to play nice until it can get into a position of strength. So, in a sense, all rational agents are paperclip maximisers. Even the hoped-for 'friendly AI' is trying to get the most it can of what it wants,  its just that what it wants is also what we want. The striking thing about a paperclipper in particular is the simplicity of what it wants. But even an agent that has complex desires is in some sense trying to get the best score it can, as surely as it can.

This would not be a conclusive test, but definitely a cool one and may spark a lot of research. Perhaps we could get started with something NLP based, opening up more and more knowledge access to the AI in the form of training data. Probably still not feasible as of 2022 in term of raw compute required.

Load More