= 27d567bc2d48d9f40e9ccc20ea370b15)
PhilGoetz comments on Reason as memetic immune disorder - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (166)
Great post, thanks, upvoted.
So most any value-core will go evil if allowed to unfold to its logical conclusions. This sounds correct to me, and also it sounds just like the motivation for FAI. Now your argument that humans solve this problem by balanced deterrence among value-cores (as opposed to weighing them together in one utility function) sounds to me like a novel intuition applicable to FAI. We have some researchers on the topic here, maybe they could speak up?
An interesting observation! An objection to it is that this approach would require your AI to have inconsistent beliefs.
Personally, I believe that fast AI systems with inconsistencies, heuristics, and habits will beat verifiably-correct logic systems in most applications; and will achieve general AI long before any pure-logic systems. (This is one reason why I'm skeptical that coming up with the right decision logic is a workable approach to FAI. I wish that Eliezer had been at Ben Goertzel's last AGI conference, just to see what he would have said to Selmer Bringsjord's presentation claiming that the only safe AI would be a logic system using a consistent logic, so that we could verify that certain undesirable statements were false in that system. The AI practitioners present found the idea not just laughable, but insulting. I said that he was telling us to turn the clock back to 1960 and try again the things that we spent decades failing at. Richard Loosemore gave a long, rude, and devastating reply to Bringsjord, who remained blissfully ignorant of the drubbing he'd just received.)
That fellow Bringsjord seems to me an obvious kook, e.g. he claims to have proven that P=NP.
He claims to have an argument that P=NP. He's a philosopher, so "argument" != proof. Although approaching P=NP as a philosophical argument does strike me as kooky.
Better proof of kookhood is that he was at AGI mainly to present his work on hypercomputing, which he claimed was a computational system with more power than a Turing machine. One element of his argument was that proofs using hyperset logic (which he said is an entire field of logic nowadays; I wouldn't know) use a notation that can not even theoretically be represented by a Turing machine. These proofs were published in two-dimensional journal articles, in black-and-white print. I did not notice any fractal fonts in the proofs.
If it's this argument, it's wrong. It is based on the claim that soap films solve the Steiner problem, which they don't. I tried this myself for four pins; here is a report of six-pin soap-film configurations. The soap film, obviously, only finds a local minimum, not a global one. But finding a local minimum is computationally easy.
Elsewhere, in a paper that detracts from the credibility of the journal it appears in, he argues that people can perform hypercomputation, on the grounds that we can imagine people performing hypercomputation. (Yes, I read all 24 pages, and that's what it comes down to.)
Judging by Google, the only wide use of the word "hyperset" in mathematics is in non-well-founded set theory. If that is what he was talking about, it's equiconsistent with the usual sort of set theory and has no more significance for AI than the choice of programming language (which, in my view, has no significance for AI).
What is it with AI? Does it attract the insane, or does it drive them insane? ETA: Or attract the people that it can drive insane?
Oh... This is sad work (Bringsjord). His argument for hypercomputation by people seems remarkably similar to Alvin Plantinga's Modal Ontological Argument for God.
I am also suspect of much of what Penrose has to say about Computationalism, although I am not yet sufficiently knowledgeable to be able to directly confront his work in any meaningful way (I am working to rectify that problem. I seem to have a knack for formal logic, and I am hoping that when I get to upper division logic classes that I will be able to more directly confront arguments like Penrose's and Bringsjord's)
I came across a wikipedia article on hypercomputing a while back, http://en.wikipedia.org/wiki/Hypercomputation , the whole theory doesn't seem at all well supported to me.
It is a field with an imaginary object of study.
It would be nice though, if outsiders could show some respect by demonstrating, as is probably demonstrable but difficult, that its object of study is incoherent, not just imaginary.
I'm not really sure it makes sense to talk about mathematical objects as being imaginary but not incoherent.
Respectful outsiders?
Is that a reference to the inner sanctum of the Hypercomputation sect? ;-)
I'd be very surprised if this Universe was super-Turing, but you think it's actually incoherent? I can definitely conceive of a hypercomputational cellular automata, what is it about the idea of our Universe being hypercomputational that seems incoherent to you?
I think that it is very common for things that we casually think we can definitely conceive of to actually be incoherent. I also think that almost everyone else underestimates how common it is.
I think I'm correcting for that. Do you agree that the halting oracle function itself is well-defined? If so, what seems inconceivable about a cellular automaton whose rules depend on the output of that oracle? OK, you have to stretch the definition of a cellular automaton to allow it, perhaps by allowing cells to have unbounded state, but the result is a wholly defined and therefore surely in-principle-conceivable Universe which is super-Turing. No?
It's not incoherent. There could be such a thing as Hypercomputation.
However, nobody has found any evidence that it exists so far - and maybe they never will.
Hypercomputation enthusiasts claim that its existence doesn't matter too much - and that it's a valuable concept regardless of whether it exists or not. Maybe.
I don't disagree (i.e., I don't see any positive reason to doubt the coherence of hypercomputation – though Michael sounds like he has one), but remember not to confuse subjective conceivability and actual coherence.
And, now I see why I am skeptical of hypercomputation. It seems to all necessitate some form of computation over an infinite number of steps. This would require some severe bending of the rules or constraints of physics, wouldn't it?
timtyler's comment below mine seems to be appropriate:
Doesn't Newtonian gravity require computation over an infinite number of steps?
Hah! I just came across your comment, Phil :-) I was "Rude"?
Hey, you were sitting next to me, and egging me on by saying "No it isn't" quietly to yourself every time Bringsjord tried to assert his (nonsensical) claim.
But anyway. I'd claim that I was not rude, really. Bringsjord kept interrupting my attempts to ask my question with loud, almost shouted comments like "If you really think that, I feel sorry for you: you really need to go back and try to get a grasp of elementary logic before you ask me questions like this!!"
So I got a little .... testy. :-) :-)
I really wish someone had recorded that exchange.
An AI doesn't have to have a purely logical structure (let alone a stupid one, e.g. structureless predicates for tables and chairs) in order to be able to logically prove important things about it. It seems to me that criticism of formally proving FAI by analogy to failed logical AI equivocates between these things.
Could be correct or wildly incorrect, depending on exactly what he meant by it. Of course you have to delete "the only", but I'd be pretty doubtful of any humans trying to do recursive self-modification in a way that didn't involve logical proof of correctness to start with.
One of the big problems is that he was trying to talk about the logical correctness of human-level symbolic statements about the world. Even if the logic is correct, there is no correct, consistent mapping from the analog world, to symbolic descriptions, and back. A mapping that's close enough to work 99.99% of the time isn't good enough when you're talking about proof.
Companies are the self-improving systems of today - e.g. see Google.
They don't hack the human brain much - but they don't need to. Brains are not perfect - but they can have their inputs preprocessed, their outputs post-processed, and they can be replaced entirely by computers - via the well-known process of automation.
Do the folk at Google proceed without logical proofs? Of course they do! Only the slowest and most tentative programmer tries to prove the correctness of their programs before they deploy them. Instead most programmers extensively employ testing methodologies. Testing is the mantra of modern programmers. Test, test, test! That way they get their products to the market before the sun explodes.
As Eliezer has already showed, "test, test, test"ing AIs that aren't provably Friendly (their recursive self-modification leads to Friendly results) can have disastrous consequences.
I'd rather wait until the sun explodes rather than deploying an unFriendly AI by accident.
The consequences of failing to adopt rapid development technologies when it comes to the development of intelligent machines should be pretty obvious - the effect is to pass the baton to another team with a different development philosophy.
Waiting until the sun explodes is not one of the realistic options.
The box experiments seem irrelevant to the case of testing machine intelligence. When testing prototypes in a harness, you would use powerful restraints - not human gatekeepers.
What powerful restraints would you suggest that would not require human judgment or human-designed decision algorithms to remove?
Turn it off, encase it in nanofabricated diamond, and bury it in a deep pit. Destroy the experimental records, retaining only enough information to help future, wiser generations to one day take up again the challenge of building a Friendly AI. Scatter the knowledge in fragments, hidden in durable artifacts, scatter even the knowledge of how to find the knowledge likewise, and arrange a secret brotherhood to pass down through the centuries the ultimate keys to the Book That Does Not Permit Itself To Be Read.
Tens of thousands of years later, when civilisation has (alas) fallen and risen several times over, a collect-all-the-plot-coupons fantasy novel takes place.
Will beat equals be developed first or be more capable than.
Selmer doesn't understand LOTS of things that Eliezer understood at age 12, he's superficially similar, but it's a very superficial similarity.
Steve Omhundro has given several talks that talk about the consequences of a purely logical or rationally exact AI system.
His talk at the Sing. Summit 2007 The Nature of Self-Improving AI discussed what would happen if such an Agent were to have the wrong rules constraining its behavior. I saw a purely logical system as being one such possible agent type to which he referred.