LESSWRONG
LW

Comment Permalink

I appreciate your posting this here, and I do agree that any information from AlphaGo Zero is limited in our ability to apply it to forecasting things like AGI.

That said, this whole article is very defensive, coming up with ways in which the evidence might not apply, not coming up with ways in which it isn't evidence.

I don't think Eliezer's article was a knock-down argument, and I don't think anyone including him believes that. But I do think the situation is some weak evidence in favor for his position over yours.

I also think it's stronger evidence than you seem to think according to the framework you lay down here!

For example, a previous feature of AI for playing games like Chess or Go was to capture information about the structure of the game via some complex combination. However in AlphaGo Zero, very little specific information about Go is required. The change in architecture actually subsumes some amount of the combination of tools needed.

Again I don't think this is a knockdown argument or very strong or compelling evidence--but it looks as though you are treating it as essentially zero evidence which seems unjustified to me.

4RobinHanson7y

As I said, I'm treating it as the difference of learning N simple general tools to learning N+1 such tools. Do you think it stronger evidence than that, or do you think I'm not acknowledging how big that is?

4magfrump7y

I think it is evidence that 'simple general tools' can be different from one another along multiple dimensions. This is a specific instance of complex details being removed to improve performance, where using the central correct tool was the ONLY moving part. I am interpreting your disagreement here to mean that you disagree that any single simple tool will be powerful enough in practice, and not in theory. I hope you agree that if someone acquired all magic powers ever written about in fiction with no drawbacks they would be at an enormous advantage over the rest of the world combined. If that was the simple tool, it would be big enough. Then if the question is "how big of an advantage can a single simple tool give," and the observation is, "this single simple tool gives a bigger advantage on a wider range of tasks than we have seen with previous tools," then I would be more concerned with bigger, faster moving simple tools in the future having different types or scales of impact.

RobinHanson7y30

I disagree with the claim that "this single simple tool gives a bigger advantage on a wider range of tasks than we have seen with previous tools."

See in context

37 What Evidence Is AlphaGo Zero Re AGI Complexity?

by RobinHanson

22nd Oct 2017

3 min read

37

Eliezer Yudkowsky write a post on Facebook on on Oct 17, where I replied at the time. Yesterday he reposted that here (link), minus my responses. So I’ve composed the following response to put here:

I have agreed that an AI-based economy could grow faster than does our economy today. The issue is how fast the abilities of one AI system might plausibly grow, relative to the abilities of the entire rest of the world at that time, across a range of tasks roughly as broad as the world economy. Could one small system really “foom” to beat the whole rest of the world?

As many have noted, while AI has often made impressive and rapid progress in specific narrow domains, it is much less clear how fast we are progressing toward human level AGI systems with scopes of expertise as broad as those of the world economy. Averaged over all domains, progress has been slow. And at past rates of progress, I have estimated that it might take centuries.

Over the history of computer science, we have developed many general tools with simple architectures and built from other general tools, tools that allow super human performance on many specific tasks scattered across a wide range of problem domains. For example, we have superhuman ways to sort lists, and linear regression allows superhuman prediction from simple general tools like matrix inversion.

Yet the existence of a limited number of such tools has so far been far from sufficient to enable anything remotely close to human level AGI. Alpha Go Zero is (or is built from) a new tool in this family, and its developers deserve our praise and gratitude. And we can expect more such tools to be found in the future. But I am skeptical that it is the last such tool we will need, or even remotely close to the last such tool.

For specific simple tools with simple architectures, architecture can matter a lot. But our robust experience with software has been that even when we have access to many simple and powerful tools, we solve most problems via complex combinations of simple tools. Combinations so complex, in fact, that our main issue is usually managing the complexity, rather than including the right few tools. In those complex systems, architecture matters a lot less than does lots of complex detail. That is what I meant by suggesting that architecture isn’t the key to AGI.

You might claim that once we have enough good simple tools, complexity will no longer be required. With enough simple tools (and some data to crunch), a few simple and relatively obvious combinations of those tools will be sufficient to perform most all tasks in the world economy at a human level. And thus the first team to find the last simple general tool needed might “foom” via having an enormous advantage over the entire rest of the world put together. At least if that one last tool were powerful enough. I disagree with this claim, but I agree that neither view can be easily and clearly proven wrong.

Even so, I don’t see how finding one more simple general tool can be much evidence one way or another. I never meant to imply that we had found all the simple general tools we would ever find. I instead suggest that simple general tools just won’t be enough, and thus finding the “last” tool required also won’t let its team foom.

The best evidence regarding the need for complexity in strong broad systems is the actual complexity observed in such systems. The human brain is arguably such a system, and when we have artificial systems of this sort they will also offer more evidence. Until then one might try to collect evidence about the distribution of complexity across our strongest broadest systems, even when such systems are far below the AGI level. But pointing out that one particular capable system happens to use mainly one simple tool, well that by itself can’t offer much evidence one way or another.

DeepMindAI Takeoff

Curated

37

Mentioned in

137Yudkowsky vs Hanson on FOOM: Whose Predictions Were Better?

New Comment

44 comments, sorted by

top scoring

Click to highlight new comments since: Today at 3:28 AM

[-]magfrump7y270

I appreciate your posting this here, and I do agree that any information from AlphaGo Zero is limited in our ability to apply it to forecasting things like AGI.

That said, this whole article is very defensive, coming up with ways in which the evidence might not apply, not coming up with ways in which it isn't evidence.

I also think it's stronger evidence than you seem to think according to the framework you lay down here!

Again I don't think this is a knockdown argument or very strong or compelling evidence--but it looks as though you are treating it as essentially zero evidence which seems unjustified to me.

[-]RobinHanson7y40

[-]magfrump7y40

I think it is evidence that 'simple general tools' can be different from one another along multiple dimensions.

...we solve most problems via complex combinations of simple tools. Combinations so complex, in fact, that our main issue is usually managing the complexity, rather than including the right few tools.

This is a specific instance of complex details being removed to improve performance, where using the central correct tool was the ONLY moving part.

And thus the first team to find the last simple general tool needed might “foom” via having an enormous advantage over the entire rest of the world put together. At least if that one last tool were powerful enough. I disagree with this claim, but I agree that neither view can be easily and clearly proven wrong.

I am interpreting your disagreement here to mean that you disagree that any single simple tool will be powerful enough in practice, and not in theory. I hope you agree that if someone acquired all magic powers ever written about in fiction with no drawbacks they would be at an enormous advantage over the rest of the world combined. If that was the simple tool, it would be big enough.

Then if the question is "how big of an advantage can a single simple tool give," and the observation is, "this single simple tool gives a bigger advantage on a wider range of tasks than we have seen with previous tools," then I would be more concerned with bigger, faster moving simple tools in the future having different types or scales of impact.

[-]RobinHanson7y30

I disagree with the claim that "this single simple tool gives a bigger advantage on a wider range of tasks than we have seen with previous tools."

[-]Nadbor Drozd7y150

I feel like this and many other arguments for AI-skepticism are implicitly assuming AGI that is amazingly dumb and then proving that there is no need to worry about this dumb superintelligence.

Remember the old "AI will never beat humans at every task because there isn't one architecture that is optimal at every task. An AI optimised to play chess won't be great at trading stocks (or whatever) and vice versa"? Well, I'm capable of running a different program on my computer depending on the task at hand. If your AGI can't do the same as a random idiot with a PC, it's not really AGI.

I am emphatically not saying that Robin Hanson has ever made this particular blunder but I think he's making a more subtle one in the same vein.

Sure, if you think of AGI as a collection of image recognisers and go engines etc. then there is no ironclad argument for FOOM. But the moment (and probably sooner) that it becomes capable of actual general problem solving on par with it's creators (i.e. actual AGI) and turns its powers to recursive self-improvement - how can that result in anything but FOOM? Doesn't matter if further improvements require more complexity or less complexity or a different kind of complexity or whatever. If human researchers can do it then AGI can do it faster and better because it scales better, doesn't sleep, doesn't eat and doesn't waste time arguing with people on facebook.

This must have been said a million times already. Is this not obvious? What am I missing?

[-]Rob Bensinger7y120

Sure, if you think of AGI as a collection of image recognisers and go engines etc. then there is no ironclad argument for FOOM. But the moment (and probably sooner) that it becomes capable of actual general problem solving on par with it's creators (i.e. actual AGI) and turns its powers to recursive self-improvement - how can that result in anything but FOOM?

The main thing that would predict slower takeoff is if early AGI systems turn out to be extremely computationally expensive. The MIRI people I've talked to about this are actually skeptical because they think we're already in hardware overhang mode.

This isn't obvious from a simple comparison of the world's available computing hardware to the apparent requirements of human brains, though; it requires going into questions like, "Given that the human brain wasn't built to e.g. convert compute into solving problems in biochemistry, how many of the categories of things the brain is doing do you need if you are just trying to convert compute into solving problems in biochemistry?"

If the first AGI systems are slow and expensive to run, then it could take time to use them to make major hardware or software improvements, particularly if the first AGI systems are complicated kludges in the fashion of human brains. (Though this is a really likely world-gets-destroyed scenario, because you probably can't align something if you don't understand its core algorithms, how it decomposes problems into subproblems, what kinds of cognitive work different parts of the system are doing and how this work contribute to the desired outcome, etc.)

If human researchers can do it then AGI can do it faster and better because it scales better, doesn't sleep, doesn't eat and doesn't waste time arguing with people on facebook.

Agreed. This is a really good way of stating "humans brains aren't efficiently converting compute into research"; those little distractions, confirmation biases, losses of motivation, coffee breaks, rationalizations, etc. add up fast.

[-]jprwg7y20

The main thing that would predict slower takeoff is if early AGI systems turn out to be extremely computationally expensive.

Surely that's only under the assumption that Eliezer's conception of AGI (simple general optimisation algorithm) is right, and Robin's (very many separate modules comprising a big intricate system) is wrong? Is it just that you think that assumption is pretty certain to be right? Or, are you saying that even under the Hansonian model of AI, we'd still get a FOOM anyway?

[-]Rob Bensinger7y80

I wouldn't say that the first AGI systems are likely to be "simple." I'd say they're likely to be much more complex than typical narrow systems today (though shooting for relative simplicity is a good idea for safety/robustness reasons).

Humans didn't evolve separate specialized modules for doing theoretical physics, chemistry, computer science, etc.; indeed, we didn't undergo selection for any of those capacities at all, they just naturally fell out of a different set of capacities we were being selected for. So if the separate-modules proposal is that we're likely to figure out how to achieve par-human chemistry without being able to achieve par-human mechanical engineering at more or less the same time, then yeah, I feel confident that's not how things will shake out.

I think that "general" reasoning in real-world environments (glossed, e.g., as "human-comparable modeling of the features of too-complex-to-fully-simulate systems that are relevant for finding plans for changing the too-complex-to-simulate system in predictable ways") is likely to be complicated and to require combining many different insights and techniques. (Though maybe not to the extent Robin's thinking?) But I also think it's likely to be a discrete research target that doesn't look like "a par-human surgeon, combined with a par-human chemist, combined with a par-human programmer, ..." You just get all the capabilities at once, and on the path to hitting that threshold you might not get many useful precursor or spin-off technologies.

[-]jprwg7y50

Humans didn't evolve separate specialized modules for doing theoretical physics, chemistry, computer science, etc.; indeed, we didn't undergo selection for any of those capacities at all, they just naturally fell out of a different set of capacities we were being selected for.

Yes, a model of brain modularity in which the modules are fully independent end-to-end mechanisms for doing tasks we never faced in the evolutionary environment is pretty clearly wrong. I don't think anyone would argue otherwise. The plausible version of the modularity model claims the modules or subsystems are specialised for performing relatively narrow subtasks, with a real-world task making use of many modules in concert - like how complex software systems today work.

As an analogy, consider a toolbox. It contains many different tools, and you could reasonably describe it as 'modular'. But this doesn't at all imply that it contains a separate tool for each DIY task: a wardrobe-builder, a chest-of-drawers-builder, and so on. Rather, each tool performs a certain narrow subtask; whole high-level DIY tasks are completed by applying a variety of different tools to different parts of the problem; and of course each tool can be used in solving many different high-level tasks. Generality is achieved by your toolset offering broad enough coverage to enable you to tackle most problems, not by having a single universal thing-doer.

... I also think [HLAI is] likely to be a discrete research target ... You just get all the capabilities at once, and on the path to hitting that threshold you might not get many useful precursor or spin-off technologies.

What's your basis for this view? For example, do you have some strong reason to believe the human brain similarly achieves generality via a single universal mechanism, rather than via the combination of many somewhat-specialised subsystems?

[-]scarcegreengrass7y10

I'm updating because I think you outline a very useful concept here. Narrow algorithms can be made much more general given a good 'algorithm switcher'. A canny switcher/coordinator program can be given a task and decide which of several narrow programs to apply to it. This is analogous to the IBM Watson system that competed in Jeopardy and to the human you describe using a PC to switch between applications. I often forget about this technique during discussions about narrow machine learning software.

[-]Raiden7y140

Robin, or anyone who agrees with Robin:

What evidence can you imagine would convince you that AGI would go FOOM?

[-]jprwg7y170

While I find Robin's model more convincing than Eliezer's, I'm still pretty uncertain.

That said, two pieces of evidence that would push me somewhat strongly towards the Yudkowskian view:

A fairly confident scientific consensus that the human brain is actually simple and homogeneous after all. This could perhaps be the full blank-slate version of Predictive Processing as Scott Alexander discussed recently, or something along similar lines.

Long-run data showing AI systems gradually increasing in capability without any increase in complexity. The AGZ example here might be part of an overall trend in that direction, but as a single data point it really doesn't say much.

[-]RobinHanson7y190

This seems to me a reasonable statement of the kind of evidence that would be most relevant.

[-]magfrump7y60

My sense is that AGZ is a high profile example of how fast the trend of neural nets (which mathematically have existed in essentially modern form since the 60s) can make progress. The same techniques have had a huge impact throughout AI research and I think counting this as a single data point in that sense is substantially undercounting the evidence. For example, image recognition benchmarks have used the same technology, as have Atari playing AI.

[-]jprwg7y60

That could represent one step in a general trend of subsuming many detailed systems into fewer simpler systems. Or, it could represent a technology being newly viable, and the simplest applications of it being explored first.

For the former to be the case, this simplification process would need to keep happening at higher and higher abstraction levels. We'd explore a few variations on an AI architecture, then get a new insight that eclipses all these variations, taking the part we were tweaking and turning it into just another parameter for the system to learn by itself. Then we'd try some variations of this new simpler architecture, until we discover an insight that eclipses all these variations, etc. In this way, our AI systems would become increasingly general without any increase in complexity.

Without this kind of continuing trend, I'd expect increasing capability in NN-based software will have to be achieved in the same way as in regular old software: integrating more subsystems, covering more edge cases, generally increasing complexity and detail.

[-]magfrump7y60

I think there are some strong points supporting the latter possibility, like the lack of similarly high profile success in unsupervised learning and the use of massive amounts of hardware and data that were unavailable in the past.

That said, I think someone five years ago might have said "well, we've had success with supervised learning but less with unsupervised and reinforcement learning." (I'm not certain about this though)

I guess in my model AGZ is more like a third or fourth data point than a first data point--still not conclusive and with plenty of space to fizzle out but starting to make me feel like it's actually part of a pattern.

[-]whpearson7y100

What evidence would convince you that AGI won't go FOOM?

[-]Razvan Ciuca7y50

If Deep Learning people suddenly starting working hard on models with dynamic architectures who self-modify (i.e. a network outputs its own weight and architecture update for the next time-step) and they *don't* see large improvements in task performance, I would take that as evidence against AGI going FOOM.

[-]the gears to ascension7y40

(for what it's worth, the current state of things has me believing that foom is likely to be much smaller than yudkowsky worries, but also nonzero. I don't expect fully general, fully recursive self improvement to be a large boost over more coherent metalearning techniques we'd need to deploy to even get AGI in the first place.)

[-]Ilses7y10

How do you draw a line between weight updates and architecture updates?

[-]whpearson7y90

I'm currently unsure of the speed of takeoff. Things that would convince me it was fast.

1) Research that showed that the ability to paradigm shift was a general skill, and not just mainly right place/right time (this is probably hard to get).

2) Research that showed that the variation in human task ability for economically important tasks is mainly due to differences in learning from trial and error situations and less to do with tapping into the general human culture built up over time.

3) Research that showed that computers were significantly more information efficient than humans for finding patterns in research. I am unsure of the amount needed here though.

4) Research that showed that the speed of human thought is a significant bottle neck in important research. That is it takes 90% of the time.

I'm trying to think of more here

[-]tristanm7y140

Disagreements here are largely going to revolve around how this observation and similar ones are interpreted. This kind of evidence must push us in some direction. We all agree that what we saw was surprising - a difficult task was solved by a system with no prior knowledge or specific information to this task baked in. Surprise implies a model update. The question seems to be which model.

The debate referenced above is about the likelihood of AGI "FOOM". The Hansonian position seems to be that a FOOM is unlikely because obtaining generality across many different domains at once is unlikely. Is AlphaGo evidence for or against this position?

There is definitely room for more than one interpretation. On the one hand, AG0 did not require any human games to learn from. It was trained via a variety of methods that were not specific to Go itself. It used neural net components that were proven to work well on very different domains such as Atari. This is evidence that the components and techniques used to create a narrow AI system can also be used on a wide variety of domains.

On the other hand, it's not clear whether the "AI system" itself should be considered as only the trained neural network, or the entire apparatus including using MCTS to simulate self play in order to generate supervised training data. The network by itself plays one game, the apparatus learns to play games. You could choose to see this observation instead as "humans tinkered for years to create a narrow system that only plays Go." AG0, once trained, cannot go train on an entirely different game and then know how to play both at a superhuman level (as far as I know, anyway. There are some results that suggest it's possible for some models to learn different tasks in sequence without forgetting). So one hypothesis to update in favor of is "there is a tool that allows a system to learn to do one task, this tool can be applied to many different tasks, but only one task at a time."

But would future, more general, AI systems do something similar to human researchers, in order to train narrow AI subcomponents used for more specific tasks? Could another AI do the "tinkering" that humans do, trained via similar methods? Perhaps not with AG0's training method specifically. But maybe there are other similar, general training algorithms that could do it, and we want to know if this one method that proves to be more general than expected suggests the existence of even more general methods.

It's hard to see how this observation can be evidence against this, but there are also no good ways to determine how strongly it is for it, either. So I don't see how this can favor Hanson's position at all, but how much it favors EY's is open to debate.

[-]scarcegreengrass7y10

The techniques you outline for incorporating narrow agents into more general systems have already been demoed, I'm pretty sure. A coordinator can apply multiple narrow algorithms to a task and select the most effective one, a la IBM Watson. And I've seen at least one paper that uses a RNN to cultivate a custom RNN with the appropriate parameters for a new situation.

[-]Ben Pace7y100

Promoted to Featured for engaging substantively with a previous Featured post, about an important topic (Foom).

[-]John_Maxwell7y60

For those of us without a lot of background on AlphaGo Zero, does anyone care to summarize how general the tool used to create it is likely to be?

[-]tristanm7y100

The only things that are required, I believe, is that the full state of the game can be fed into the network as input, and that the action space is small enough to be represented by network output and is discrete, which allows MCTS to be used. If you can transform an arbitrary problem into this formulation then in theory the same methods can be used.

[-]Razvan Ciuca7y90

As far as I can see you can use the same techniques to learn to play any perfect information zero-sum game

[-]moridinamael7y30

Is there any reason why the same techniques couldn't be applied to imperfect information non-zero-sum games?

[-]entirelyuseless27y30

I agree with this, although it might not work for some theoretically possible games that humans would not actually play.

Life in the real world, however, is not a perfect information zero-sum game, or even an approximation of one. So there is no reason to suppose that the techniques use will generalize to a fooming AI.

[-]the gears to ascension7y110

Here are some examples of recent work that uses these same tools to make other critical components of a more general AI:

https://coxlab.github.io/prednet/

https://arxiv.org/abs/1707.06203

https://deepmind.com/blog/differentiable-neural-computers/

[-]habryka7y50

(Edit note: Made the links into actual links, let me know if you do not want me to fix/improve small things like this in future comments of yours)

[-]lukstafi7y10

The general tool: residual networks variant of convolutional NNs, MCTS-like variable-depth tree search. Prerequisites: input can be presented as K layers of N-D data (where N=1,2,3... not too large), the action space is discrete. If the actions are not discrete, an additional small module would be needed to quantize the action space based on the neural network's action priors.

[-]moridinamael7y50

Honest question - is there some specific technical sense in which you are using "complex"? Colloquially, complex just means a thing consisting of many parts. Any neural network is "a thing consisting of many parts", and I can generally add arbitrarily many "parts" by changing the number of layers or neurons-per-layer or whatever at initialization time.

I don't think this is what you mean, though. You mean something like architectural complexity, though I think the word "architectural" is a weasel word here that lets you avoid explaining what exactly is missing from, e.g., AlphaGo Zero. I think by "complex" you mean something like "a thing consisting of many distinct sub-things, with the critical functional details spread across many levels of components and their combinations". Or perhaps "the design instructions for the thing cannot be efficiently compressed". This is the sense in which the brain is more "complex" than the kidneys.

(Although, the design instructions for the brain can be efficiently compressed, and indeed brains are made from surprisingly simple instructions. A developed, adult brain can't be efficiently compressed, true, but that's not a fair comparison. A blank-slate initialized AlphaGo network, and that same network after training on ten million games of Go, are not the same artifact.)

Other words aside from "complex" and "architecture" that I think you could afford to taboo for the sake of clarity are "simple" and "general". Is the idea of a neural network "simple"? Is a convnet "general"? Is MCTS a "simple, general" algorithm or a "complex, narrow" one? These are bad questions because all those words must be defined relative to some standard that is not provided. What problem are you trying to solve, what class of techniques are you regarding as relevant for comparison? A convnet is definitely "complex and narrow" compared to linear regression, as a mathematical technique. AlphaGo Zero is highly "complex and narrow" relative to a vanilla convnet.

If your answer to "how complex and how specific a technique do you think we're missing?" is always "more complex and more specific than whatever Deepmind just implemented", then we should definitely stop using those words.

[-]jprwg7y40

Perhaps "size of compiled program" would be one way to make a crude complexity estimate. But I definitely would like to be able to better define this metric.

In any case, I don't think the concept of software complexity is meaningless or especially nebulous. A program with a great many different bespoke modules, which all interact in myriad ways, and are in turn full of details and special-cases and so on, is complex. A program that's just a basic fundamental core algorithm with a bit of implementation detail is simple.

I do agree that the term "complexity" is often used in unhelpful ways; a common example is the claim that the brain must be astronomically complex purely on the basis of it having so many trillions of connections. Well, a bucket of water has about 6x10^26 hydrogen bonds, but who cares? This is clearly not a remotely useful model of complexity.

I do think learned complexity makes the problem of defining complexity in general harder, since that training data can't count for nothing. Otherwise, you could claim the interpreter is the program, and the program you feed into it is really the training data. So clearly, the simpler and more readily available the training data, the less complexity it adds. And the cheapest simplest training data of all would be that generated from self-play.

>Although, the design instructions for the brain can be efficiently compressed, and indeed brains are made from surprisingly simple instructions.

Can you elaborate on this? If this is based on the size of the functional genome, can you assure me that the prenatal environment, or simply biochemistry in general, offers no significant additional capabilitiy here?

I'm reminded of gwern's hypothetical interpreter that takes a list of integers and returns corresponding frames of Pirates of the Caribbean (for which I can't now find a cite... I don't think I imagined this?). Clearly the possibility of such an interpreter does not demonstrate that the numbers 0 through 204480 are generically all that's needed to encode Pirates of the Caribbean in full.

[-]moridinamael7y30

I think that's a good way of framing it. Imagine it's the far future, long after AI is a completely solved problem. Just for fun, somebody writes the smallest possible fully general seed AI in binary code. How big is that program? I'm going to guess it's not bigger than 1 GB. The human genome is ~770 MB. Yes, it runs on "chemistry", but that laws of physics underpinning chemistry/physics actually don't take that many bytes to specify. Certainly not hundreds of megabytes.

Maybe a clearer question would be, how many bytes do you need to beam to aliens, in order for them to grow a human? The details of the structure of the embryonic cell, the uterus, the umbilical cord, the mother's body, etc., are mostly already encoded in the genome, because a genome contains the instructions for copying itself via reproduction. Maybe you end up sending a few hundred more megabytes of instructions as metadata for unpacking and running the genome, but not more than that.

Still, though, genomes are bloated. I'll bet you can build an intelligence on much less than 770 MB. 98.8% of the genome definitely has nothing to do with the secret sauce of having a powerful general intelligence. We know this because we share that much of our genome with chimps. Yes, you need a body to have a brain, so there's a boring sense in which you need the whole genome to build a brain, but this argument doesn't apply to AIs, which don't need to rely on ancient legacy biology.

[-]PDV7y50

I would take this more seriously if I'd seen any evidence that Robin's position had updated at all since the first Yudkowsky-Hanson FOOM debate. Which, despite seeing many discussions, between them and otherwise, I have not.

As it is, updating on this post would be double-counting.

[-]RobinHanson7y40

Consider the example of whether a big terror attack indicates that there has been an increase in the average rate or harm of terror attacks. You could easily say "You can't possibly claim that big terror attack yesterday is no evidence; and if it is evidence it is surely in the direction of the ave rate/harm having increased." Technically correct, but then every other day without such a big attack is also evidence for a slow decrease in the rate/harm of attacks. Even if the rate/harm didn't change, every once in a while you should expect a big attack. This in the sense in which I'd say that finding one more big tool isn't much evidence that big tools will matter more in the future. Sure the day when you find such a tool is itself weak evidence in that direction. But the whole recent history of that day and all the days before it may be an evidential wash.

[-]Davidmanheim7y30

I think it's reasonable to question how relevant this achievement is to our estimates of the speed of GAI, and I appreciate the specification of what evidence Robin thinks would be useful for estimating this speed - I just don't see its relevance.
"The best evidence regarding the need for complexity in strong broad systems is the actual complexity observed in such systems."
There is an underlying question of how complex a system needs to be to exhibit GAI; a single example of evolved intelligence sets an upper limit, but is very weak evidence about the minimum. So my question for Robin is; what evidence can we look for about such a minimum?

(My suggestion: The success of "Narrow AI" at tasks like translation and writing seem like clear but weak evidence that many products of a moderately capable mind is achievable by relatively low complexity AI with large datasets. Robin: do you agree? If not, what short term successes or failures do you predict on the basis of your assumption that NAI successes aren't leading to better GAI?)

[-]Kevin Van Horn7y20

Arguing about the mostly likely outcome is missing the point: when the stakes are as high as survival of the human race, even a 1% probability of an adverse outcome is very worrisome. So my question to Robin Hanson is this: are you 99% certain that the FOOM scenario is wrong?

[-]Rob Bensinger7y30

I disagree with this line of argument. It's true that moderately lower-probability scenarios deserve extra attention if they're higher-stakes, but if the full range of scenarios in which early AGI systems realize large capability gains totaled to only 1% probability, then they would deserve little attention. Practically speaking, in real life, there are just too many other urgent medium- and high-probability scenarios to worry about for us to have the resources to focus on all the 1%-probability futurist scenarios.

[-]Kevin Van Horn7y40

If there are more than a few independent short-term extinction scenarios (from any cause) with a probability higher than 1%, then we are in trouble -- their combined probability would add up to a significant probability of doom.

As far as resources go, even if we threw 100 times the current budget of MIRI at the problem, that would be $175 million, which is

- 0.005% of the U.S. federal budget,

- 54 cents per person living in the U.S., or

- 2 cents per human being.

[-]Dan Clemmensen7y00

Let's assume you are correct, or at least "less wrong". Then there will eventually exsist a set of simple tools {T} that can be combined to solve many AGI-level problems. A given problem P will require a particular "complex" solver S(P). But then we achieve AGI by creating a system S(S(P)) that can produce S(P) given P and the rules for use of the elements in {T}. It seems to me that S(S(P)) may be amenable to an AlphaGo Zero approach.

[-]Dan Clemmensen7y-10

It seems to me that systems of the AlphaGo Zero type should be able to derive the rules for subatomic particles from the rules of QED, the rules of nuclear physics from the rules of subatomic particles, and the rules of physical chemistry from the rules of nuclear physics. I'm not sure why anyone would do this, except that if if it fails, it might tell us something about missing rules, or possible find new rules. For instance, the first step might identify ro exclude some new subatomic particles, or identify or definitively exclude new particles at the quark level. The leap from physcal chemistry to the whole of biochemistry is more of a challenge: I suspect the some human guidance to select intermediate goals would simplify things (protein folding has been mentioned.)

[-]whpearson7y130

I'm not sure why you expect this... Go is easily simulatable. We find it hard to simulate simple quantum systems like atoms well. Let alone aggregate materials.

Moderation Log