= 27d567bc2d48d9f40e9ccc20ea370b15)
steven0461 comments on The Curve of Capability - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (264)
It seems like you're entirely ignoring feedback effects from more and better intelligence being better at creating more and better intelligence, as argued in Yudkowsky's side of the FOOM debate.
And hardware overhang (faster computers developed before general cognitive algorithms, first AGI taking over all the supercomputers on the Internet) and fast infrastructure (molecular nanotechnology) and many other inconvenient ideas.
Also if you strip away the talk about "imbalance" what it works out to is that there's a self-contained functioning creature, the chimpanzee, and natural selection burps into it a percentage more complexity and quadruple the computing power, and it makes a huge jump in capability. Nothing is offered to support the assertion that this is the only such jump which exists, except the bare assertion itself. Chimpanzees were not "lopsided", they were complete packages designed for an environment; it turned out there were things that could be done which created a huge increase in optimization power (calling this "symbolic processing" assumes a particular theory of mind, and I think it is mistaken) and perhaps there are yet more things like that, such as, oh, say, self-modification of code.
Interesting. Can you elaborate or link to something?
I'm not Eliezer, but will try to guess what he'd have answered. The awesome powers of your mind only feel like they're about "symbols", because symbols are available to the surface layer of your mind, while most of the real (difficult) processing is hidden. Relevant posts: Detached Lever Fallacy, Words as Mental Paintbrush Handles.
Thanks.
The posts (at least the second one) seem to point that symbolic reasoning is overstated and at least some reasoning is clearly non-symbolic (e.g. visual).
In this context the question is whether the symbolic processing (there is definitely some - math, for example) gave pre-humans the boost that allowed the huge increase in computing power, so I am not seeing the contradiction.
Speech is a kind of symbolic processing, and is probably an important capability in mankind's intellectual evolution, even if symbolic processing for the purpose of reasoning (as in syllogisms and such) is an ineffectual modern invention.
Susan Blackmore argues that what originally caused the "huge increase in optimization power" was memes - not symbolic processing - which probably started up a bit later than the human cranium's expansion did.
The genetic difference between a chimp and a human amounts to about ~40–45 million bases that are present in humans and missing from chimps. And that number is irrespective of the difference in gene expression between humans and chimps. So it's not like you're adding a tiny bit of code and get a superapish intelligence.
Nothing is offered to support the assertion that there is another such jump. If you were to assert this then another premise of yours, that an universal computing device can simulate every physical process, could be questioned based on the same principle. So here is an antiprediction, humans are on equal footing with any other intelligence who can master abstract reasoning (that does not necessarily include speed or overcoming bias).
In a public debate, it makes sense to defend both sides of an argument, because each of the debaters actually tries to convince a passive third party whose beliefs are not clearly known. But any given person that you are trying to convince doesn't have a duty to convince you that the argument you offer is incorrect. It's just not an efficient thing to do. A person should always be allowed to refute an argument on the grounds that they don't currently believe it to be true. They can be called on contradicting assertions of not believing or believing certain things, but never required to prove a belief. The latter would open a separate argument, maybe one worth engaging in, but often a distraction from the original one, especially when the new argument being separate is not explicitly acknowledged.
I agree with some of what you wrote although I'm not sure why you wrote it. Anyway, I was giving an argumentative inverse of what Yudkowsky asserted and hereby echoed his own rhetoric. Someone claimed A and in return Yudkowsky claimed that A is a bare assertion, therefore ¬A, whereupon I claimed that ¬A is a bare assertion therefore the truth-value of A is again ~unknown. This of course could have been inferred from Yudkowskys statement alone, if interpreted as a predictive inverse (antiprediction), if not for the last sentence which states, "[...] and perhaps there are yet more things like that, such as, oh, say, self-modification of code." [1] Perhaps yes, perhaps not. Given that his comment already scored 16 when I replied, I believed that highlighting that it offered no convincing evidence for or against A would be justified by one sentence alone. Here we may disagree, but note that my comment included more information than that particular sentence alone.
You should keep track of whose beliefs you are talking about, as it's not always useful or possible to work with the actual truth of informal statements where you analyze correctness of debate. A person holding a wrong belief for wrong reasons can still be correct in rejecting an incorrect argument for incorrectness of those wrong reasons.
If A believes X, then (NOT X) is a "bare assertion", not enough to justify A changing their belief. For B, who believes (NOT X), stating "X" is also a bare assertion, not enough to justify changing the belief. There is no inferential link between refuted assertions and beliefs that were held all along. A believes X not because "(NOT X) is a bare assertion", even though A believes both that "(NOT X) is a bare assertion" (correctly) and X (of unknown truth).
That is true. Yet for a third party, one that is unaware of any additional substantiation not featured in the debate itself, a prediction and its antipredication cancel out each other. As a result no conclusion can be drawn by an uninformed bystander. This I tried to highlight without having to side with one party.
They don't cancel out each other, as they both lack convincing power, equally irrelevant. It's an error to state as arguments what you know your audience won't agree with (change their mind in response to). At the same time, explicitly rejecting an argument that failed to convince is entirely correct.
Let's assume that you contemplate the possibility of an outcome Z. Now you come across a discussion between agent A and agent B discussing the prediction that Z is true. If agent B does proclaim the argument X in favor of Z being true and you believe that X is not convincing then this still gives you new information about agent B and the likelihood of Z being true. You might now conclude that Z is slightly more likely to be true because of additional information in favor of Z and the confidence of agent B necessary to proclaim that Z is true. Agent A does however proclaim argument Y in favor of Z being false and you believe that Y is equally unconvincing than argument X in favor of Z being true. You might now conclude again that the truth-value of Z is ~unknown as each argument and the confidence of its facilitator ~outweigh each other.
Therefore no information is irrelevant if it is the only information about any given outcome in question. Your judgement might weigh less than the confidence of an agent with possible unknown additional substantiation in favor of its argument. If you are unable to judge the truth-value of an exclusive disjunction then that any given argument about it is not compelling does tell more about you than the agent that does proclaim it.
Any argument alone has to be taken into account, if only due to its logical consequence. Every argument should be incorporated into your probability estimations for that it signals a certain confidence (for that it is proclaimed at all) of the agent that is uttering it. Yet if there exists a counterargument that is inverse to the original argument you'll have to take that into account as well. This counterargument might very well outweigh the original argument. Therefore there are no arguments that lack the power to convince, however small, yet arguments can outweigh and trump each other.
ETA: Fixed the logic, thanks Vladimir_Nesov.
Z XOR ¬Z is always TRUE.
(I know what you mean, but it looks funny.)
B believes that X argues for Z, but you might well believe that X argues against Z. (You are considering a model of a public debate, while this comment was more about principles for an argument between two people.)
Also, it's strange that you are contemplating levels of belief in Z, while A and B assert it being purely true or false. How overconfident of them.
(Haven't yet got around to a complete reply rectifying the model, but will do eventually.)
What's clearly fundamental about the human/chimpanzee advantage, the thing that made us go FOOM and take over the world, is that we can, extremely efficiently, share knowledge. This is not as good as fusing all our brains into a giant brain, but it's much much better than just having a brain.
This analysis possibly suggests that "taking over the world's computing resources" is the most likely FOOM, because it is similar to the past FOOM, but that is weak evidence.
See my reply to saturn on recursive self-improvement. Potential hardware overhang, I already addressed. Nanotechnology is thus far following the curve of capability, and there is every reason to expect it will continue to do so in the future. I already explained the sense in which chimpanzees were lopsided. Self modification of code has been around for decades.
May be minorly off-topic: nothing Drexler hypothesised has, as far as I know, even been started. As I understand it, the state of things is still that we still have literally no idea how to get there from here, and what's called "nanotechnology" is material science or synthetic biology. Do you have details of what you're describing as following the curve?
Perhaps start here, with his early work on the potential of hypertext ;-)
A good source of such details is Drexler's blog, where he has written some good articles about -- and seems to consider highly relevant -- topics like protein design and DNA origami.
(cough) I'm sure Drexler has much detail on Drexler's ideas. Assume I'm familiar with the advocates. I'm speaking of third-party sources, such as from the working worlds of physics, chemistry, physical chemistry and material science for example.
As far as I know - and I have looked - there's little or nothing. No progress to nanobots, no progress to nanofactories. The curve in this case is a flat line at zero. Hence asking you specifically for detail on what you are plotting on your graph.
There has been some impressive sounding research done on simulated diamondoid tooltips for this kind of thing. (Admittedly, done by advocates.)
I suspect when these things do arrive, they will tend to have hard vacuum, cryogenic temperatures, and flat surfaces as design constraints.
Well, that's a bit like saying figuring out how to smelt iron constituted no progress to the Industrial Revolution. These things have to go a step at a time, and my point in referring to Drexler's blog was that he seems to think e.g. protein design and DNA origami do constitute real progress.
As for things you could plot on a graph, consider the exponentially increasing amount of computing power put into molecular modeling simulations, not just by nanotechnology advocates, but people who actually do e.g. protein design for living today.
Also, I'm not sure what you mean by "symbolic processing" assuming a particular theory of mind -- theories of mind differ on the importance thereof, but I'm not aware of any that dispute its existence. I'll second the request for elaboration on this.
I'll also ask, assuming I'm right, is there any weight of evidence whatsoever that would convince you of this? Or is AI go foom for you a matter of absolute, unshakable faith?
It would be better if you waited until you had made somewhat of a solid argument before you resorted to that appeal. Even Robin's "Trust me, I'm an Economist!" is more persuasive.
The Bottom Line is one of the earliest posts in Eliezer's own rationality sequences and describes approximately this objection. You'll note that he added an Addendum:
I'm resisting the temptation to say "trust me, I'm an AGI researcher" :-) Bear in mind that my bottom line was actually the pro "AI go foom" side; it's still what I would like to believe.
But my theory is clearly falsifiable. I stand by my position that it's fair to ask you and Eliezer whether your theory is falsifiable, and if so, what evidence you would agree to have falsified it.
But barely. ;)
You would not believe how little that would impress me. Well, I suppose you would - I've been talking with XiXi about Ben, after all. I wouldn't exactly say that your status incentives promote neutral reasoning on this position - or Robin on the same. It is also slightly outside of the core of your expertise, which is exactly where the judgement of experts is notoriously demonstrated to be poor.
You are trying to create AGI without friendliness and you would like to believe it will go foom? And this is supposed to make us trust your judgement with respect to AI risks?
Incidentally, 'the bottom line' accusation here was yours, not the other way around. The reference was to question its premature use as a fully general counterargument.
We are talking here about predictions of the future. Predictions. That's an important keyword that is related to falsifiability. Build a flipping AGI of approximately human level and see if whether the world as we know it ends within a year.
You just tagged teamed one general counterargument out to replace it with a new one. Unfalsifiability has a clear meaning when it comes to creating and discussing theories and it is inapplicable here to the point of utter absurdity. Predictions, for crying out loud.
No indeed, they very strongly promote belief in AI foom - that's why I bought into that belief system for a while, because if true, it would make me a potential superhero.
Nope, it's exactly in the core of my expertise. Not that I'm expecting you to believe my conclusions for that reason.
When I believed in foom, I was working on Friendly AI. Now that I no longer believe that, I've reluctantly accepted human level AI in the near future is not possible, and I'm working on smarter tool AI instead - well short of human equivalence, but hopefully, with enough persistence and luck, better than what we have today.
That is what falsifiability refers to, yes.
My theory makes the prediction that even when recursive self-improvement is used, the results will be within the curve of capability, and will not produce more than a steady exponential rate of improvement.
Are you saying your theory makes no other predictions than this?
RWallace, you made a suggestion of unfalsifiabiity, a ridiculous claim. I humored you by giving the most significant, obvious and overwhelmingly critical way to falsify (or confirm) the theory. You now presume to suggest that such a reply amounts to a claim that this is the only prediction that could be made. This is, to put it in the most polite terms I am willing, disingenuous.
-sigh-
This crap goes on year after year, decade after bloody decade. Did you know the Singularity was supposed to happen in 2000? Then in 2005. Then in 2010. Guess how many Singularitarians went "oh hey, our predictions keep failing, maybe that's evidence our theory isn't actually right after all"? If you guessed none at all, give yourself a brownie point for an inspired guess. It's like the people who congregate on top of a hill waiting for the angels or the flying saucers to take them up to heaven. They just go "well our date was wrong, but that doesn't mean it's not going to happen, of course it is, Real Soon Now." Every time we actually try to do any recursive self-improvement, it fails to do anything like what the AI foom crowd says it should do, but of course, it's never "well, maybe recursive self-improvement isn't all it's cracked up to be," it's always "your faith wasn't strong enough," oops, "you weren't using enough of it," or "that's not the right kind" or some other excuse.
That's what I have to deal with, and when I asked you for a prediction, you gave me the usual crap about oh well you'll see when the Apocalypse comes and we all die, ha ha. And that's the most polite terms I'm willing to put it in.
I've made it clear how my theory can be falsified: demonstrate recursive self-improvement doing something beyond the curve of capability. Doesn't have to be taking over the world, just sustained improvement beyond what my theory says should be possible.
If you're willing to make an actual, sensible prediction of RSI doing something, or some other event (besides the Apocalypse) coming to pass, such that if it fails to do that, you'll agree your theory has been falsified, great. If not, fine, I'll assume your faith is absolute and drop this debate.
That the Singularity concept pattern-matches doomsday cults is nothing new to anyone here. You looked further into it and declared it false, wedrifid and others looked into it and declared it possible. The discussion is now about evidence between those two points of view. Repeating that it looks like a doomsday cult is taking a step backwards, back to where we came to this discussion from.
So, I'm vaguely aware of Singularity claims for 2010. Do you have citations for people making such claims that it would happen in 2000 or 2005?
I agree that pushing something farther and farther into the future is a potential warning sign.
The pattern you are completing here has very little relevance to the actual content of the conversation. The is no prediction here about the date of a possible singularity and, for that matter, no mention of how probable it is. When, or if, someone such as yourself creates a human level general intelligent agent and releases it then that will go a long way towards demonstrating that one of the theories is false.
You have iterated through a series of argument attempts here, abandoning each only to move to another equally flawed. The current would appear to be 'straw man'... and not a particularly credible straw man at that. (EDIT: Actually, no you have actually kept the 'unfalsifiable' thing here, somehow.)
Your debating methods are not up to the standards that are found to be effective and well received on lesswrong.
If you think that most Singularities will be Unfreindly, the Anthropic Shadow means that their absense from our time-line isn't very strong evidence against their being likely in the future: no matter what proportion of the multiverse sees the light cone paperclipped in 2005, all the observers in 2010 will be in universes that weren't ravaged.
You are not an expert on recursive self improvement, as it relates to AGI or the phenomenon in general.
In fairness, I'm not sure anyone is really an expert on this (although this doesn't detract from your point at all.)
You are right, and I would certainly not expect anyone to have such expertise for me to take their thoughts seriously. I am simply wary of Economists (Robin) or AGI creator hopefuls claiming that their expertise should be deferred to (only relevant here as a hypothetical pseudo-claim). Professions will naturally try to claim more territory than would be objectively appropriate. This isn't because the professionals are actively deceptive but rather because it is the natural outcome of tribal instincts. Lets face it - intellectual disciplines and fields of expertise are mostly about pissing on trees with but with better hygiene.
Yes, but why would the antipredictions of AGI researcher not outweigh yours as they are directly inverse? Further, if your predictions are not falsifiable then they are by definition true and cannot be refuted. Therefore it is not unreasonable to ask for what would prematurely disqualify your predictions so as to be able to argue based on diverging opinions here. Otherwise, as I said above, we'll have two inverse predictions outweigh each other, and not the discussion about risk estimations we should be having.
The claim being countered was falsifiability. Your reply here is beyond irrelevant to the comment you quote.
rwallace said it all in his comment that has been downvoted. Since I'm unable to find anything wrong with his comment and don't understand yours at all, which has for unknown reasons be upvoted, there's no way for me to counter what you say besides by what I've already said.
Here's a wild guess of what I believe to be the positions. rwallace asks you what information would make you update or abandon your predictions. You in turn seem to believe that predictions are just that, the utterance of that might be possible, unquestionable and not subject to any empirical criticism.
I believe I'm at least smarter than the general public, although I haven't read a lot of Less Wrong yet. Further I'm always willing to announce that I have been wrong and to change my mind. This should at least make you question your communication skills regarding outsiders, a little bit.
Theories are collections or proofs and a hypothesis is a prediction or collection of predictions and must be falsifiable or proven to become a collection of proofs that is a theory. It is not absurd at all to challenge predictions based on their refutability, as any prediction that isn't falsifiable will be eternal and therefore useless.
The wikipedia article on falsifiablility would be a good place to start if you wish to understand what is wrong with way falsification has been used (or misused) here. With falsifiability understood, seeing the problem should be straightforward.