= 27d567bc2d48d9f40e9ccc20ea370b15)
timtyler comments on Tallinn-Evans $125,000 Singularity Challenge - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (369)
...and what is "FOOM"? Or are 1 and 2 supposed to serve as a definition?
Either way, this is looking pretty ridiculous :-(
I was going to give a formal definition¹ but then I noticed you said either way. Assume that 1 and 2 are the definition of FOOM: that is a possible event, and that it is the end of everything. I challenge you to substantiate your claim of "ridiculous", as formally as you can.
Do note that I will be unimpressed with "anything defined by 1 and 2 is ridiculous". Asteroid strikes and rapid climate change are two non-ridiculous concepts that satisfy the definition given by 1 and 2.
¹. And here it is: FOOM is the concept that self-improvement is cumulative and additive and possibly fast. Let X be an agent's intelligence, and let X + f(X) = X^ be the function describing that agent's ability to improve its intelligence (where f(X) is the improvement generated by an intelligence of X, and X^ is the intelligence of the agent post-improvement). If X^ > X, and X^ + f(X^) evaluates to X^^, and X^^ > X^, the agent is said to be a recursively self-improving agent. If X + f(X) evaluates in a short period of time, the agent is said to be a FOOMing agent.
Ridiculousness is in the eye of the beholder. Probably the biggest red flag was that there was no mention of what was supposedly going to be annihilated - and yes, it does make a difference.
The supposedly formal definition tells me very little - because "short" is not defined - and because f(X) is not a specified function. Saying that it evaluates to something positive is not sufficient to be useful or meaningful.
Fast enough that none of the other intelligences in Earth can copy its strengths or produce countermeasures sufficient to stand a chance in opposing it.
Yes - though it is worth noting that if Google wins, we may have passed that point without knowing it back in 1998 sometime.
Foom is finding the slider bar in the config menu labeled 'intelligence' and moving it all the way to the right.
I don't know what it is - but I am pretty sure that is not it.
If you check with: http://lesswrong.com/lw/wf/hard_takeoff/
...you will see there's a whole bunch of vague, hand-waving material about how fast that happens.
If you're willing to reject every definition presented to you, you can keep asking the question as long as you want. I believe this is typically called 'trolling'.
What is your definition of foom?
Fooming has been pretty clearly described. Fooming amounts to an entity drastically increasing both its intelligence and ability to manipulate reality around it in a very short time, possibly a few hours or weeks, by successively improving its hardware and/or software.
Uh huh. Where, please?
Possibly a few hours or weeks?!? [emphasis added]
Is is a few hours? Or a few weeks? Or something else entirely? ...and how much is "drastically".
Vague definitions are not worth critics bothering attacking.
In an attempt to answer my own question, this one is probably the closest I have seen from Yudkowsky.
It apparently specifies less than a year - though seems "rather vague" about the proposed starting and finishing capabilities.
Example locations where this has been defined include Mass Driver's post here where he defined it slightly differently as "to quickly, recursively self-improve so as to influence our world with arbitrarily large strength and subtlety". I think he meant indefinitely large there, but the essential idea is the same. I note that you posted comments in that thread, so presumably you've seen that before, and you explicitly discussed fooming. Did you only recently decide that it wasn't sufficiently well-defined? If so, what caused that decision?
Well, I've seen different timelines used by people in different contexts. Note that this isn't just a function of definitions, but also when one exactly has an AI start doing this. An AI that shows up later, when we have faster machines and more nanotech, can possibly go foom faster than an AI that shows up earlier when we have fewer technologies to work with. But for what it is worth, I doubt anyone would call it going foom if the process took more than a few months. If you absolutely insist on an outside estimate for purposes of discussion, 6 weeks should probably be a decent estimate.
It isn't clear to me what you are finding too vague about the definition. Is it just the timeline or is it another aspect?
This might be a movie threat notion-- if so, I'm sure I'll be told.
I assume the operational definition of FOOM is that the AI is moving faster than human ability to stop it.
As theoretically human-controlled systems become more automated, it becomes easier for an AI to affect them. This would mean that any humans who could threaten an AI would find themselves distracted or worse by legal, financial, social network reputational, and possibly medical problems. Nanotech isn't required.
Yes, that seems like a movie threat notion to me, if an AI has the power to do those things to arbitrary people it likely can scale up from there so quickly to full control that it shouldn't need to bother with such steps, although it is minimally plausible that a slow growing AI might need to do that.
No, I've been aware for the issue for a loooong time.
Ok. So what caused you to use the term as if it had a specific definition when you didn't think it did? Your behavior is very confusing. You've discussed foom related issues on multiple threads. You've been here much longer than I have; I don't understand why we are only getting to this issue now.
I did raise this closely-related issue over two years ago. To quote the most relevant bit:
There may well be other instances in between - but scraping together references on the topic seems as though it would be rather tedious.
I did what, exactly?
The quote you give focuses just on the issue of time-span. It also has already been addressed in this thread. Machine intelligence in the sense it is often used is not at all the same as artificial general intelligence. This has in fact been addressed by others in this subthread. (Although it does touch on a point you've made elsewhere that we've been using machines to engage in what amounts to successive improvement which is likely relevant.)
I would have thought that your comments in the previously linked thread started by Mass Driver would be sufficient, like when you said:
And again in that thread where you said:
Although rereading your post, I am now wondering if you were careful to put "anti-foom" in quotation marks because it didn't have a clear definition. But in that case, I'm slightly confused to how you knew enough to decide that that was an anti-foom argument.
Right - so, by "anti-foom factor", I meant: factor resulting in relatively slower growth in machine intelligence. No implication that the "FOOM" term had been satisfactorily quantitatively nailed down was intended.
I do get that the term is talking about rapid growth in machine intelligence. The issue under discussion is: how fast is considered to be "rapid".
Six weeks - from when? Machine intelligence has been on the rise since the 1950s. Already it exceeds human capabilities in many domains. When is the clock supposed to start ticking? When is it supposed to stop ticking? What is supposed to have happened in the middle?
There is a common and well-known distinction between what you mean by 'machine intelligence' and what is meant by 'AGI'. Deep Blue is a chess AI. It plays chess. It can't plan a stock portfolio because it is narrow. Humans can play chess and plan stock portfolios, because they have general intelligence. Artificial general intelligence, not 'machine intelligence', is under discussion here.
Nothing is "arbitrarily large" in the real world. So, I figure that definition confines FOOM to the realms of fantasy. Since people are still discussing it, I figure they are probably talking about something else.
Tim, I have to wonder if you are reading what I wrote, given that the sentence right after the quote is "I think he meant indefinitely large there, but the essential idea is the same. " And again, if you thought earlier that foom wasn't well-defined what made you post using the term explicitly in the linked thread? If you have just now decided that it isn't well-defined then a) what do you have more carefully defined and b) what made you conclude that it wasn't narrowly defined enough?
What distinction are you trying to draw between "arbitrarily large" and "indefinitely large" that turns the concept into one which is applicable to the real world?
Maybe you can make up a definition - but what you said was "fooming has been pretty clearly described". That may be true, but it surely needs to be referenced.
What exactly am I supposed to have said in the other thread under discussion?
Lots of factors indicate that "FOOM" is poorly defined - including the disagreement surrounding it, and the vagueness of the commonly referenced sources about it.
Usually, step 1 in those kinds of discussions is to make sure that people are using the terms in the same way - and have a real disagreement - and not just a semantic one.
Recently, I participated in this exchange - where a poster here gave p(FOOM) = 0.001 - and when pressed they agreed that they did not have a clear idea of what class of events they were referring to.
Arbitrarily large means just that in the mathematical sense. Indefinitely large is a term that would be used in other contexts. In the contexts that I've seen "indefinitely" used and the way I would mean it, it means so large as to not matter as the exact value for the purpose under discussion (as in "our troops can hold the fort indefinitely").
Disagreement about something is not always a definitional issue. Indeed, when dealing with people on LW where people try to be rational as possible and have whole sequences about tabooing words and the like, one shouldn't assign a very high probability to disagreements being due to definitions. Moreover, as one of the people who assigns a low probability to foom and have talked to people here about those issues, I'm pretty sure that we aren't disagreeing on definitions. Our estimates for what the world will probably look like in 50 years disagree. That's not simply a definitional issue.
Ok. So why are you now doing step 1 years later? And moreover, how long should this step take as you've phrased it, given that we know that there's substantial disagreement in terms of predicted observations about reality in the next few years? That can't come from definitions. This is not a tree in a forest.
Yes! Empirical evidence. Unfortunately, it isn't very strong evidence. I don't know if he meant in that context that he didn't have a precise definition or just that he didn't feel that he understood things well enough to assign a probability estimate. Note that those aren't the same thing.
I don't see how the proposed word substitution is supposed to help. If FOOM means: "to quickly, recursively self-improve so as to influence our world with indefinitely large strength and subtlety", we still face the same issues - of how fast is "quickly" and how big is "indefinitely large". Those terms are uncalibrated. For the idea to be meaningful or useful, some kind of quantification is needed. Otherwise, we are into "how long is a piece of string?" territory.
I did also raise the issue two years ago. No response, IIRC. I am not too worried if FOOM is a vague term. It isn't a term I use very much. However, for the folks here - who like to throw their FOOMs around - the issue may merit some attention.
If indefinitely large is still too vague, you can replace it with ""to quickly, recursively self-improve so as to influence our world with sufficient strength and subtlety such that it can a) easily wipe out humans b) humans are not a major threat to it achieving almost any goal set and c) humans are sufficiently weak that it doesn't gain resources by bothering to bargain with us." Is that narrow enough?