= 27d567bc2d48d9f40e9ccc20ea370b15)
fortyeridania comments on Google may be trying to take over the world - Less Wrong Discussion
You are viewing a comment permalink. View the original post to see all comments and the full post content.
= 783df68a0f980790206b9ea87794c5b6)
Loading…= b715d02e85f7b3b4ae65ca3d3e450868)
Subscribe to RSS Feed
Powered by Reddit
= f037147d6e6c911a85753b9abdedda8d)
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (133)
Can you cite some evidence for this?
Um, surely if you take (a) people with a track record of successful achievement in an area (b) people without a track record of success but who think they know a lot about the area, the presumption that (a) is more likely to know what they're talking about should be the default presumption. It may of course not work out that way, but that would surely be the way to bet.
Yes, I agree, but that is only part of the story, right?
What if autodidacts, in their untutored excitability, are excessively concerned about a real risk? Or if a real risk has nearly all autodidacts significantly worried, but only 20% of actual experts significantly worried? Wouldn't that falsify /u/private_messaging's assertion? And what's so implausible about that scenario? Shouldn't we expect autodidacts' concerns to be out of step with real risks?
If autodidacts are excessively concerned, then why would it be worth for experts to listen to them?
It may not be. I was not taking issue with the claim "Experts need not listen to autodidacts." I was taking issue with the claim "Given a real risk, experts are more likely to be concerned than autodidacts are."
I would assume that experts are likely to be concerned to an extent more appropriate to the severity of the risk than autodidacts are.
There can be exceptions, of course, but when non-experts make widely more extreme claims than experts do on some issue, especially a strongly emotively charged issue (e.g. the End of the World), unless they can present really compelling evidence and arguments, Dunning–Kruger effect seems to be the most likely explanation.
That is exactly what I would assume too. Autodidacts' risk estimates should be worse than experts'. It does not follow that autodidacts' risk estimates should be milder than experts', though. The latter claim is what I meant to contest.
"Autodidacts" was in quotes for a reason.
Let's talk about some woo that you're not interested in. E.g. health risks of thymerosal and vaccines in general. Who's more likely to notice it, some self proclaimed "autodidacts", or normal biochemistry experts? Who noticed the possibility of a nuke, back-then conspiracy theorists or scientists? Was Semmelweis some weird outsider, or was he a regular medical doctor with medical training? And so on and so forth.
Right now, experts are concerned with things like nuclear war, run-away methane releases, epidemics, and so on, while various self proclaimed existential risk people (mostly philosophers) seem to be to greater or lesser extent neglecting said risks in favor of movie plot dangers such as runaway self improving AI or perhaps totalitarian world government. (Of course if you listen to said x-risk folks, they're going to tell you that it's because the real experts are wrong.)
All are good and relevant examples, and they all support the claim in question. Thanks!
But your second paragraph supports the opposite claim. (Again, the claim in question is: Experts are more likely to be concerned over risks than autodidacts are.) In the second paragraph, you give a couple "movie plot" risks, and note that autodidacts are more concerned about them than experts are. Those would therefore be cases of autodidacts being more concerned about risks than experts, right?
If the claim were "Experts have more realistic risk estimates than autodidacts do," then I would readily agree. But you seem to have claimed that autodidacts' risk estimates aren't just wrong--they are biased downward. Is that indeed what you meant to claim, or have I misunderstood you?
What I said was that "autodidacts" (note the scare quotes) are more likely to fail to notice some genuine risk, than the experts are. E.g. if there's some one specific medication that poses risk for a reason X, those anti vaxers are extremely unlikely to spot that, due to the lack of necessary knowledge and skills.
By "autodidacts" in scare quotes I mean interested and somewhat erudite laymen who may have read a lot of books but clearly did very few exercises from university textbooks (edit: or any other feedback providing exercises at all).
I understand the scare quotes.
I agree that autodidacts "are more likely to fail to notice some genuine risk, than experts are."
But autodidacts are also more likely to exaggerate other genuine risks than experts are, are they not?
If (3) is true, then doesn't that undermine the claim "Experts are more likely to be concerned over risks than autodidacts are"?
What I said was:
Besides, being more concerned is not the same as being more likely to be concerned. Just as being prone to panic doesn't automatically make you better at hearing danger.
To clarify, I have nothing anything against self educated persons. Some do great things. The "autodidacts" was specifically in quotes.
What is implausible, is this whole narrative where you have a risk obvious enough that people without any relevant training can see it (by the way of that paperclipping argument), yet the relevant experts are ignoring it. Especially when the idea of an intelligence turning against it's creator is incredibly common in fiction, to the point that nobody has to form that idea on their own.
In general, current AGI architectures work via reinforcement learning: reward and punishment. Relevant experts are worried about what will happen when an AGI with the value-architecture of a pet dog finds that it can steal all the biscuits from the kitchen counter without having to do any tricks.
They are less worried about their current creations FOOMing into god-level superintelligences, because current AI architectures are not FOOMable, and it seems quite unlikely that you can create a self-improving ultraintelligence by accident. Except when that's exactly what they plan for them to do (ie: Shane Legg).
Juergen Schmidhuber gave an interview on this very website where he basically said that he expects his Goedel Machines to undergo a hard takeoff at some point, with right and wrong being decided retrospectively by the victors of the resulting Artilect War. He may have been trolling, but it's a bit hard to tell.
I'd need to have links and to read it by myself.
With regards to reinforcement learning, one thing to note is that the learning process is in general not the same thing as the intelligence that is being built by the learning process. E.g. if you were to evolve some ecosystem of programs by using "rewards" and "punishments", the resulting code ends up with distinct goals (just as humans are capable of inventing and using birth control). Not understanding this, local genuises of the AI risk been going on about "omg he's so stupid it's going to convert the solar system to smiley faces" with regards to at least one actual AI researcher.
Here is his interview. It's very, very hard to tell if he's got his tongue firmly in cheek (he refers to minds of human-level intelligence and our problems as being "small"), or if he's enjoying an opportunity to troll the hell out of some organization with a low opinion of his work.
With respect to genetic algorithms, you are correct. With respect to something like neural networks (real world stuff) or AIXI (pure theory), you are incorrect. This is actually why machine-learning experts differentiate between evolutionary algorithms ("use an evolutionary process to create an agent that scores well on X") versus direct learning approaches ("the agent learns to score well on X").
What, really? I mean, while I do get worried about things like Google trying to take over the world, that's because they're ideological Singulatarians. They know the danger line is there, and intend to step over it. I do not believe that most competent Really Broad Machine Learning (let's use that nickname for AGI) researchers are deliberately, suicidally evil, but then again, I don't believe you can accidentally make a dangerous-level AGI (ie: a program that acts as a VNM-rational agent in pursuit of an inhumane goal).
Accidental and evolved programs are usually just plain not rational agents, and therefore pose rather more limited dangers (crashing your car, as opposed to killing everyone everywhere).
Well, the neural network in my head doesn't seem to want to maximize the reward signal itself, but instead is more interested in maximizing values imprinted into it by the reward signal (which it can do even by hijacking the reward signal or even by administering "punishments"). Really, reward signal is not utility, period. Teach the person to be good, and they'll keep themselves good by punishing/rewarding themselves.
I don't think it's worth worrying about the brute force iteration over all possible programs. Once you stop iterating over the whole solution space in the learning method itself, the learning method faces the problem that it can not actually ensure that the structures constructed by the learning method don't have separate goals (nor is it desirable to ensure such, as you would want to be able to teach values to an agent using the reward signal).
Firstly, I was talking about artificial neural networks, which do indeed function as reinforcement learners, by construction and mathematical proof.
Secondly, human beings often function as value learners ("learn what is good via reinforcement, but prefer a value system you're very sure about over a reward that seems to contradict the learned values") rather than reinforcement learners. Value learners, in fact, are the topic of a machine ethics paper from 2011, by Daniel Dewey.
Sorry, could you explain this better? It doesn't match up with how the field of machine learning usually works.
Yes, any given hypothesis a learner has about a target function is only correct to within some probability of error. But that probability can be very small.
With the smiley faces, I am referring to disagreement with Hibbard, summarized e.g. here on wikipedia
You're speaking as if value learners were not a subtype of reinforcement learners.
For a sufficiently advanced AI, i.e. one that learns to try different counter-factual actions on a world model, it is essential to build a model of the reward, which is to be computed on the counter-factual actions. It's this model of the reward that is specifying which action gets chosen.
Looks like presuming a super-intelligence from the start.