= 27d567bc2d48d9f40e9ccc20ea370b15)
WrongBot comments on Existential Risk and Public Relations - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (613)
How influential are his publications if they could not convince Ben Goertzel (SIAI/AGI researcher), someone who has read Yudkowsky's publications and all of the LW sequences? You could argue that he and other people don't have the smarts to grasp Yudkowsky's arguments, but who does? Either Yudkowsky is so smart that some academics are unable to appreciate his work or there is another problem. How are we, we who are far below his level, supposed to evaluate if we should believe what Yudkowsky says if we are neither smart enough to do so nor able to subject his work to empirical criticism?
The problem here is that telling someone that Yudkowsky spent a lot of time thinking and writing about something is not a qualification. Further it does not guarantee that he would acknowledge and welcome the contributions of others who disagree.
Ben Goertzel believes in psychic phenomenon (see here for details), so his failure to be convinced by Eliezer is not strong evidence against the correctness of Eliezer's stance.
For what it's worth, Eliezer has been influential/persuasive enough to get the SIAI created and funded despite having absolutely no academic qualifications. He's also responsible for coining "Seed AI".
Indeed, I was just trying to figure out how someone with money or power who wants to know what is the right thing to do but who does not have the smarts should do. Someone like a politician or billionaire who would either like to support some AGI research or the SIAI. How are they going to decide what to do if all AGI experts tell them that there is no risk from AGI research and that the SIAI is a cult when at the same time the SIAI tells them the AGI experts are intellectual impotent and the SIAI is the only hope for humanity to survive the AI revolution. What should someone who does not have the expertise or smarts to estimate those claims, but who nevertheless has to decide how to use his power, should do? I believe this is not an unrealistic scenario as many rich or powerful people want to do the right thing, yet do not have the smarts to see why they should trust Yudkowsky instead of hundreds of experts.
Interesting, when did he come up with the concept of "Seed AI". Because it is mentioned in Karl Schroeder's Ventus (Tor Books, 2000.) ISBN 978-0312871970.
Didn't find the phrase "Seed AI" there. One plot element is a "resurrection seed", which is created by an existing, mature evil AI to grow itself back together in case it's main manifestation is destroyed. A Seed AI is a different concept, it's something the pre-AI engineers put together that grows into a superhuman AI by rewriting itself more and more powerful. A Seed AI is specifically a method to get to AGI from not having one, not just an AI that grows from a seed-like thing. I don't remember recursive self-improvement being mentioned with the seed in Ventus.
A precursor concept where the initial AI bootstraps itself by merely learning things, not necessarily by rewriting it's own architecture, goes all the way back to Alan Turing's 1950 paper on machine intelligence.
Here is a quote from Ventus:
[...]
...and here's a quote from I.J. Good, from 1965:
He didn't coin the term "Seed AI" either.
Yes, but I believe it is a bit weird for a Wikipedia article to state that someone is the originator of the Seed AI theory when he just coined the term. I wasn't disputing anything, just trying to figure out if it is actually the case that Yudkowsky came up with the concept in the first place.
Not the concept - the term.
"Seed AI theory" probably refers to something or another in here - which did indeed originate with Yu'El.
Presumably http://en.wikipedia.org/wiki/Seed_AI should be considered to be largely SIAI marketing material.
It is further explained that the Winds were designed to evolve on their own so they are not mere puppets of human intentions but possess their own intrinsic architecture.
In other places in the book it is explained how humans did not create their AI Gods but that they evolved themselves from seeds designed by humans.
I don't think the failure of someone to be convinced of some position is ever strong evidence against that position. But this argument here is genuinely terrible. I disagree with person x about y, therefore person x is wrong about z? Do we even have to go into why this is fallacious?
Ever is a strong word. If a competent expert in a field who has a known tendency to err slightly on the side of too much openness to the cutting edge fails to be convinced by a new finding within his field that says an awful lot.
That is simply not the form of the argument you quote. "Ben Goertzel believes in psychic phenomenon" can not be represented as "I disagree with person x ".
I'm being generous and giving the original comment credit for an implicit premise. As stated the argument is "Person x believes y, therefore person x is wrong about z." this is so obviously wrong it makes my head hurt. WrongBot's point is that someone has to have a poor reasoning capacity to believe in psy. But since he didn't provide any evidence to that effect it reduces to 'I disagree with Goertzel about psy'.
Fair point re: "ever".
I generally don't try to provide evidence for every single thing I say, and I am especially lax about things that I consider to be incredibly obvious.
But I'm annoyed enough to lay out a very brief summary of why belief in PSI is ludicrous:
No one has to give evidence for everything they say but when things that you thought were obviously wrong begin to get defended by physics-literate reductionist materialists that seems like a good time to lower your confidence.
Well to begin with, Goertzel's paper claims to be such a mechanism. Have you read it? I don't know if it works or not. Seems unwise to assume it doesn't though.
Publication bias, motivated cognition and effect size are all concerns and were my previous explanation. I found this meta-analysis upset that view for me.
Oh man! I left out the most important objection!
If PSI exploits weird physics in a complicated manner and produces such tiny effects, where the hell did the mechanism come from? PSI would obviously be a very useful adaptation, so why don't we see it in other species? Why aren't the effects stronger, since there's such a strong evolutionary pressure in favor of them?
Goertzel's paper also includes psychokinesis as a PSI phenomenon supported by strong evidence. I would love to see the study he's talking about for that one. Or a video.
All of this is also discussed in Outside the Gates. I can try to dig up what he said this weekend.
The experiments aren't macroscopic. The results involve statistical deviations from expected normal distributions of say, white noise generators when participants try to will the results in different directions. I don't think these results are nearly as compelling as other things, see Jahn and Dunne 2005 for example. They had some methodological issues and the one attempt that was made at replication, while positive, wasn't significant at anywhere near the level of the original.
If you're actually interested you should consider checking out the book. It is a quick, inexpensive read. Put it this way: I'm not some troll who showed up here to argue about parapsychology. Six months ago I was arguing your position here with someone else and they convinced me to check out the book. I then updated significantly in the direction favoring psi (not enough to say it exists more likely than not, though). Everything you've said is exactly what I was saying before. It turns out that there are sound responses to a lot of the obvious objections, making the issue not nearly as clear cut as I thought.
It would be wrong if it were a logical deduction instead of an inference. That is, if WrongBot actually wrote 'therefore' or otherwise signaled absolute deductive certainty then he would be mistaken. As is he presents it as evidence, which it in fact is.
There is a clear implied premise 'psychic phenomenon are well known to be bullshit'. Not all baseline premises must be supported in an argument. Instead, the argument should be considered stronger or weaker depending on how reliable the premises are. I don't think WrongBot loses too much credibility in this case by dismissing psychic phenomenon.
It isn't even evidence until you include a premise about the likelihood of y, which we agree is the implied premise.
I think I'm just restating the exchange I had with komponisto on this point. Goertzel's position isn't that of someone who is doesn't know any physics or Enlightenment-style rationality. It is clearly a contrarian position which should be treated rather differently since we can assume he is familiar with the reasons why psychic phenomena are 'well known to be bullshit'. It is a fully generalizable tactic which can be used against all and any contrarian thinkers. Try "Robin Hanson thinks we should cut health care spending 50%, therefore he is less likely to be right about fertility rate."
This is obviously going to be the case when trying to convince an individual of something. The beliefs (crackpot or otherwise) of the target audience are always going to be relevant to persuasively. As a comment directed in part to the wider lesswrong audience the assumed premises will be different.
If I were a reader who thought Robin's position on health care was as implausible as belief in magic and thought that making claims about the fertility was similar to AI strategy then I would take this seriously. As it stands the analogy is completely irrelevant.
The extent to which it is fallacious depends rather strongly on what y and z (and even x) are, it seems to me.
Any argument of this nature needs to include some explanation of why someone's ability to think about y is linked to their ability to think about z. But even with that (which wasn't included in the comment) you can only conclude that y and z imply each other. You can't just conclude z.
In other words, you have to show Goertzel is wrong about psychic phenomenon before you can show that his belief in it is indicative of reasoning flaws elsewhere.
I don't disagree in principle, but psychic phenomena are pretty much fundamentally ruled out by current physics. So a person's belief in them raises serious doubts about that person's understanding of science at the very least, if not their general rationality level.
I got the impression from Damien Broderick's book that a lot of PSI researchers do understand physics and aren't postulating that PSI phenomena use the sort of physical interactions gravity or radio waves use. There's a story that Einstein was interested in PSI research, but declared it nonsense when the claimed results showed PSI effects that weren't subject to the inverse square law, so this isn't a new idea.
Damien Broderick's attitude in his book is basically that there's a bunch of anomalous observations and neither a satisfactory explanation or, in his opinion, a refutation for them exists. Goertzel's attitude is to come up with a highly speculative physical theory that could explain that kind of phenomena, and which would take a bit more than "would need extra particles" to show as nonsense.
"Not understanding basic physics" doesn't really seem to cut it in either case. "It's been looked into by lots of people, a few of them very smart, for 80 years, and nothing conclusive has come out of it, so most likely there isn't anything in it, and if you still want to have a go, you better start with something the smart people in 1970s didn't have" is basically the one I've got.
I'm not holding my breath over the recent Bem results, since he seems to be doing pretty much the same stuff that was done in the 70s and always ended up failing one way or the other, but I'm still waiting for someone more physics-literate to have a go at Goertzel's pilot wave paper.
"Not understanding basic physics" sounds like a harsh quasi-social criticism, like "failing at high-school material". But that's not exactly what's meant here. Rather, what's meant is more like "not being aware of how strong the evidence against psi from 20th-century physics research is".
The Bayesian point here is that if a model M assigns a low probability to hypothesis H, then evidence in favor of M is evidence against H [EDIT: technically, this is not necessarily true, but it usually is in practice, and becomes more likely as P(H|M) approaches 0]. Hence each high-precision experiment that confirms quantum field theory counts the same as zillions of negative psi studies.
Evidence distinguishes between not for individual models. There may be models that are consistent with the experiments that confirm quantum field theory but also give rise to explanations for anomalous cognition.
By the Bayesian definition of evidence, "evidence for" a hypothesis (including a "model", which is just a name for a complex conjunction of hypotheses) simply means an observation more likely to occur if the hypothesis is true than if it is false.
Carroll claims that current data implies the probability of such models being correct is near zero. So I'd like to invoke Aumann here and ask what your explanation for the disagreement is. Where is Carroll's (and others') mistake?
This isn't someone with tarot cards talking about using crystal energy to talk to your dead grand parent. To condemn someone for holding a similar position to the uneducated is to rule out contrarian thought before any debate occurs. Humans are still confused enough about the world that there is room for change in our current understanding of physics. There are some pretty compelling results in parapsychology, much or all of which may be due to publication bias, methodological issues or fraud. But that isn't obviously the case, waving our hands and throwing out these words isn't an explanation of the results. I'm going to try and make a post on this subject a priority now.
Did you read the linked post by Sean Carroll? Parapsychologists aren't condemned for holding a similar position to the uneducated; they're condemned for holding a position blatantly inconsistent with quantum field theory on the strength of evidence much, much weaker than the evidence for quantum field theory. Citing a century's worth of experimentally confirmed physical knowledge is far from hand-waving.
Again, this is explicitly addressed by Carroll. Physicists are not confused in the relevant regimes here. Strong evidence that certain highly precise models are correct has been obtained, and this constrains where we can reasonably expect future changes in our current understanding of physics.
Now, I'm not a physicist, so if I'm actually wrong about any of this, I'm willing to be corrected. But, as the saying goes, there is a time to confess ignorance, and a time to relinquish ignorance.
We're don't know what the relevant regimes are here. Obviously human brains aren't producing force fields that are bending spoons.
We have some experimental results. No one has any idea what they mean except it looks like something weird is happening. People are reacting to images they haven't seen yet and we don't have any good explanation for these results. Maybe it is fraud (with what motivation?), maybe there are methodological problems (but often no one can find any), maybe there is just publication bias (but it would have to be really high to explain the results in the precognition meta-analysis).
On the other hand, maybe our physics isn't complete enough to explain what is going on. Maybe a complete understanding of consciousness would explain it. Maybe we're in a simulation and our creators have added ad hoc rules that violate the laws of physics. Physics certainly rules out some explanations but Carroll certainly hasn't shown that all but error/fraud/bias have been ruled out.
Btw, using spoon bending as the example and invoking Uri Geller is either ignorant or disingenuous of him (and I almost always love Sean Carroll). Parapsychologists more or less all recognize Geller as a fraud and an embarrassment and only the kookiest would claim that humans can bend spoons with their minds. Real parapsychological experiments are nothing like that.
I suspect it will be difficult to communicate why fraud, method error and publication bias are difficult explanations for me to accept if you aren't familiar with the results of the field. I recommend Outside the Gates of Science if you haven't read it yet.
It will actually be easy to communicate exactly what explanation there is for the events. Bem has effectively been getting a group of students to flip a bunch of coins for the last eight years. He has had them do it perfectly methodologically soundly. Only now has he had a group that - through pure, random chance - happened to flip 53% heads and 47% tails. The number of students, the number of coins, the number of flips, all are large enough that this is an unlikely event - but he's spent eight years trying to make it happen, and so happen it eventually has. Good for him!
The only problem with all of this is that the journals that we take to be sources of knowledge have this rule: anything more unlikely than x, must have some other explanation other than pure chance. This is true at first blush, but when somebody spends years trying to make pure chance spit out the result he wants, this rule fails badly. That is all that's going on here.
If someone is unable to examine the available evidence and come to a sane conclusion on a particular topic, this makes it less likely that they are able to examine the available evidence and to sane conclusions on other topics.
I don't take Goertzel seriously for the same reason I don't take young earth creationists seriously. It's not that I disagree with him, it's that his beliefs have almost no connection to reality.
(If it makes you feel better, I have read some of Goertzel's writing on AGI, and it's stuffed full of magical thinking.)
I'd be interested to hear more about that.
From Ten Years to a Positive Singularity:
and
From The Singularity Institute's Scary Idea (And Why I Don't Buy It):
From Chance and Consciousness:
And pretty much all of On the Algebraic Structure of Consciousness and Evolutionary Quantum Computation.
This is all just from fifteen minutes of looking around his website. I'm amazed anyone takes him seriously.
Oh...
wow.
I think that paper alone proves your point quite nicely.
I mostly disagree with Ben, but I don't think judging him based on that paper is fair. It's pretty bad, but it was also written in 1996. Fourteen years is a lot of time to improve as a thinker.
I had that thought too, and I was thinking of retracting or amending my comment to that effect, but looking at some of his later publications in the same journal(?) suggests that he hasn't leveled up much since then.
"The Futility Of Emergence" really annoys me. It's a perfectly useful word. It's a statement about the map rather than about the territory, but it's a useful one. Whereas magic means "unknowable unknowns", emergent means "known unknowns" - the stuff that we know follows, we just don't know how.
e.g. Chemistry is an emergent property of the Schrodinger equation, but calculating anything useful from that is barely in our grasp. So we just go with the abstraction we know, and they're separate sciences. But we do know we have that work to do.
Just linking to that essay every time someone you're disagreeing with says "emergent" is difficult to distinguish from applause lights.
Saying the word "emergent" adds nothing. You're right that it's not as bad as calling something magic and declaring that it's inherently unknowable, but it also offers zero explanatory power. To reword your example:
There is absolutely no difference in meaning when you take the word "emergent" out. That's why it isn't useful, which Eliezer was pointing out.
Nitpick: I don't think that is exactly what EY was pointing out. Take a look at the comments and the general response of "Huh? Who makes that mistake?" It seems EY was complaining about the tendency of AGI researchers to use "emergence" as if it were an explanation, not ordinary use of the word that doesn't pretend it is one but just, say, points out that the behavior is surprising given what it's composed of, or that your current methods aren't powerful enough to predict the consequences. He didn't seem to have realized that particular mistake was mostly localized to AGI people.
It seems more likely that when the cited people said "intelligence is an emergent phenomenon", they were misunderstood as proposing that as a satisfactory explanation of the phenomenon.
Nitpick accepted.
I'm not entirely sure this is correct. I wouldn't call the trajectories of planets and galaxies "properties" of Relativity, but I would call it emergent behavior due to Relativity. It's a stylistic and grammatical choice, like when to use "which" and when to use "that." They may seem the same to the uninitiated, but there's a difference and the initiated can tell when you're doing it wrong.
So, I agree with David Gerard that trying to eradicate the use of the word is misplaced. It'd be like saying "the word 'which' is obsolete, we're only going to use 'that' and look down on anyone still using 'which'." You lose far more by such a policy than you gain.
IIRC, that post was adequately dismantled in its comments.
From what I've seen, the people who comment here who have read Broderick's book have come away, if not convinced psy describes some real physical phenomena, convinced that the case isn't at all open and shut the way young earth creationism is. When an issue is such that smart, sane people can disagree then you have to actually resolve the object level disagreement before you can use someone's beliefs on the issue in a general argument about their rationality. You can't just assume it as you do here.
Yes, here WrongBot is safe to assume basic physics.
Edit for the sake of technical completeness: And biology.
Goertzel's paper on the subject is about extending the de Broglie Bohm pilot wave theory in a way that accounts for psi while being totally consistent with all known physics. Maybe it is nonsense, I haven't read it. But you can't assume it is.
I disagree. I do not need to (and should not) discard my priors when evaluating claims.
It would be an error in reasoning on my part if I did not account for the low prior (to reading it) probability of a psyonics theory being sane when evaluating the proponents other claims. For emphasis: not lowering my confidence in Goertzel's other beliefs because he is a proponent of psi without me having read his paper would be an outright mistake.
I also note that you defending Goertzel on the psi point is evidence against Goertzel's beliefs regarding AI. Extremely weak evidence.
Huh?
I mean what is written in the straightforward English sense. I mention it to emphasize that all evidence counts.