Aeonios comments on Reply to Holden on The Singularity Institute - Less Wrong

46 Post author: lukeprog 10 July 2012 11:20PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Article Navigation

Comments (213)

Comments (213)

Sort By: Best

Show:

You are viewing a single comment's thread.

Comment author: Aeonios comment score below threshold [+] (15 children)

Comment author: Aeonios 19 July 2012 03:41:14AM -3 points [-]

There are several reasons why I agree with the "Pascal's Mugging" comment:

Intelligence Explosion: There are several reasons why an intelligence explosion is highly unlikely. First, upgrading computer fabrication equipment requires on the order of 5-15 billion dollars. Second, intelligence is not measured in gigaflops or petaflops, and mere improvement of fabrication technology is insufficient to increase intelligence. Finally, the requisite variety that drives innovation and creation will be extremely difficult to produce in AIs of a limited quantity. Succeeding in engineering or science requires copious amounts of failure, and AIs are not immune to this either.

2.Computing Overhang: The very claim of "computing overhang" shows total ignorance of actual AI, and of the incredible complexity of human intelligence. The human brain is made up of numerous small regions which both "run programs" inside of themselves and communicate via synchronous signals with the rest of the brain in concert (in neural, and not transistor form). A human level AI would be the same, and could not simply be run on, say, your average web server, no matter how decked out it is. An AI that could run on "extra" hardware would probably be too primitive to reproduce itself on purpose, and if it did it would be a minor nuisance at worst.

The idea that AIs can be "programmed" is mostly nonsense. Very simple AIs can be "programmed", sure, but neural networks require training by experience, just like humans. An AI with human level intelligence or greater would need to be taught like a child, and any "friendliness" that came of it would be the result of its "instincts" (I'm guessing we wouldn't want AIs with aggression) and of its experience. Additionally, as mentioned above, the need for variety in intelligence to produce real progress means that copying them will not be as economical as it might seem, not to mention not nearly as simple as you make it out to be.
The timescales you present are absurd. Humans barely have an understanding of human psychology, and they do terrible at it with the knowledge they do have. We may have teraflops desktop computers in 20 years, but that does not imply that they will magically sprout intelligence! Technically, even with today's technology you could produce a program much more sophisticated than shrdlu was, and receive orders of magnitude better performance than the original did, but it is the complexity of programming something that learns that prevents it from occurring commonly. It will likely be a hundred maybe two hundred years before we have a sophisticated enough understanding of human intelligence to reproduce it in any meaningful way. We have only taken the bare first steps into the field thus far, and development has been much slower than for the rest of the computing industry.

In short, human stupidity that is occurring right now is a much greater threat to our future as a species than is any hypothetical superintelligent AI that might finally appear a hundred years or more in the future. If human civilization is even to maintain its integrity long enough to produce such a thing ever, then widespread ignorance of economics, spirituality/psychology, and general lack of sensitivity to culture and art must be dealt with first and foremost.

Comment author: DaFranker 19 July 2012 04:17:22PM * 2 points [-]

Nice try. You've almost succeeded at summarizing practically all the relevant arguments against the SI initiative that have already been refuted. Notice the last part there that says "have already been refuted".

Each of the assertions you make are ones that members of the SI have already adressed and refuted. I'd take the time to decompose your post into a list of assertions and give you links to the particular articles and posts where those arguments were taken down, but I believe this would be an unwise use of my time.

It would, at any rate, be much simpler to tell you to at least read the articles on the Facing the Singularity site, which are a good vulgarized introduction to the topic. In particular, the point of timescale overestimates is clearly adressed there, as is that of the "complexity" of human intelligence.

I'd like to also indicate that you are falsely overcomplexifying the activity of the human brain. There are no such things as "numerous small regions" that "run programs" or "communicate". These are interpretations of patterns within the natural events, which are simply, first and foremost, a huge collection of neurons sending signals to other neurons, each with its own unique set of links to particular other neurons and a domain of nearby neurons to which it could potentially link itself. This is no different from the old core sequence article here on LessWrong where Eliezer talks about how reality doesn't actually follow the rules of aerodynamics to move air around a plane - it's merely interactions of countless tiny [bits of something] on a grand scale, with each tiny [bit of something] doing its own thing, and nowhere along the entire process do the formulae we use for aerodynamics get "solved" to decide where one of the [bits of something] must go.

Anyway, I'll cut myself short here - I doubt any more deserves to be said on this. If you are willing to learn and question yourself, and actually want to become a better rationalist and obtain more correct beliefs, the best way to start is to go read some of the articles that are already on LessWrong and actually read the material on the Singinst.org website, most of which is very readable even without prior technical knowledge or experience.

Comment author: Aeonios comment score below threshold [+] (8 children)

Comment author: Aeonios 20 July 2012 06:23:27AM -5 points [-]

I decided to read through the essays on facingthesingularity, and I have found more faults than I care to address. Also, I can see why you might think that the workings of the human mind are simple, given that the general attitude here is that you should go around maximizing your "utility function". That is utter and complete nonsense for reasons that deserve their own blog post. What I see more than anything is a bunch of ex-christians worshipping their newfound hypothetical machine god, and doing so by lowering themselves to the level of machine rather than raising machine to the level of man.

I'll give one good example to make clear what I mean: (from facingthesingularity) But that can’t possibly be correct. The probability of Linda being a bank teller can’t be less than the probability of her being a bank teller and a feminist.

This is my “Humans are crazy” Exhibit A: The laws of probability theory dictate that as a story gets more complicated, and depends on the truth of more and more claims, its probability of being true decreases. But for humans, a story often seems more likely as it is embellished with details that paint a compelling story: “Linda can’t be just a bank teller; look at her! She majored in philosophy and participated in antinuclear demonstrations. She’s probably a feminist bank teller.”

But, the thing is, context informs us that while a philosophy major is unlikely to work for a bank, a feminist is much more likely to work a "pink collar job" such as secretarial work or as a bank teller, where they can use the state to monger for positions, pay and benefits above and beyond what they deserve. A woman who otherwise would have no interest in business or finance, when indoctrinated by the feminist movement, will leap to take crappy office jobs so they can raise their fists in the air in onionistic fashion against the horrible man-oppression they righteously defeated with their superior women intellects. The simple fact that "philosophy" in a modern school amounts to "The History of Philosophy", and is utterly useless might also clue one in on the integrity or lack thereof that a person might have, although of course it isn't conclusive.

In short, impressive "logical" arguments about how probabilities of complements must be additive can only be justified in a vacuum without context, a situation that does not exist in the real world.

Comment author: RichardKennaway 20 July 2012 11:17:14AM 6 points [-]

But, the thing is, context informs us

None of that fog obscures the basic fact that the number of feminist female bank tellers cannot possibly be greater than the number of female bank tellers. The world is complex, but that does not mean that there are no simple truths about it. This is one of them.

People have thought up all manner of ways of exonerating people from the conjunction fallacy, but if you go back to Eliezer's two posts about it, you will find some details of the experiments that have been conducted. His conclusion:

The conjunction fallacy is probably the single most questioned bias ever introduced, which means that it now ranks among the best replicated. The conventional interpretation has been nearly absolutely nailed down. Questioning, in science, calls forth answers.

The conjunction error is an error, and people do make it.

Comment author: Aeonios comment score below threshold [+] (5 children)

Comment author: Aeonios 21 July 2012 09:22:05AM * -4 points [-]

I reread that section, and you are correct, given that they don't tell you whether or not she is a feminist, it cannot be used as a criterion to determine whether or not she is a banker. However, I would say that the example, in typical public education style, is loaded and begs an incorrect answer. Since the only data you are given is insufficient to draw any conclusions, the participant is lead to speculate without understanding the limitations of the question.

As for "utility function", there are at least three reasons why it is not just wrong, but entirely impossible.

1: Utility is heterogeneous. Which gives you more "utility", a bowl of ice cream or a chair? The question itself is nonsensical, the quality/type of utility gained from a chair and a bowl of ice cream are entirely different.

2: Utility is complementary. If I own a field, the field by itself may be useless to me. Add a picnic table, and some food and suddenly the field gains utility beyond the food, table, and field individually. Perhaps I could run horses through the field, or add some labor and intelligent work and turn it into a garden, but the utility I get from it depends on my preferences (which may change) and the combination with other resources and a plan. Another example, a person who owns a yaht would probably get more "utility" out of going to the ocean than someone who does not.

3: Utility is marginal. For the first three scoops of ice cream, I'd say I get equal "utility" from each. The fourth scoop yields comparably less "utility" than the previous three, and by the fifth the utility becomes negative, as I feel sick afterwards. By six scoops I'm throwing away ice cream. On the other hand, if I have 99 horses, whether I gain or lose one would not make much difference as to the utility I get from them, but if I only have 2 horses, losing one could mean losing more than half of my utility. Different things have different useful quantities in different situations depending on how they are used.

4: Utility cannot be measured. This should be obvious. Even if we were to invent a magical brain scanner that could measure brain activity in high resolution in vivo, utility is not always the same for the same thing every time it is experienced, and you still have the apples-oranges problem that makes the comparison meaningless to begin with.

5: Human psychology is not a mere matter of using logic correctly or not. In this case, it is definitely a misapplication, but it seems the only psychology that gets any attention around here is anecdotes from college textbooks on decisions and some oversimplified mechanistic theorizing from neuroscience. You talk about anchoring like it's some horrible disease, when it's the same fundamental process required for memory and mastery of concepts. You've probably heard of dissociation but you probably wouldn't believe me if I told you that memory can be flipped on and off like a light switch at the whim of your unconscious.

That aside, treating intelligence as a machine that optimizes things is missing the entire point of intelligence. If you had ever read Douglas Hofstadter's "Godel Escher Bach", or Christopher Alexander's "The Nature of Order" series, you might have a greater appreciation for the role that abstract pattern recognition and metaphor plays in intelligence.

Finally, I read two "papers" from SI, and found them entirely unprofessional. They were both full of vague terminology and unjustified assertions and were written in a colloquial style that pretty much begs the reader to believe the crap they're spewing. You get lots of special graphs showing how a superhuman AI would be something like two orders of magnitude more intelligent than humans, but no justification for how these machines will magically be able to produce the economic resources to reach that level of development "overnight". Comparing modern "AIs" to mice is probably the most absurd fallacy I've seen thus far. Even the most sophisticated AI for driving cars cannot drive on a real road, its "intelligence" is overall still lacking in sophistication compared to a honey bee, and the equipment required to produce its rudimentary driving skills far outweigh the benefits. Computer hardware may improve regularly by Moore's Law, but the field of AI research does not, and there is no evidence that we will see a jump in computer intelligence from below insects to above orangutans any time soon. When we do, it will probably take them 50-100 years to leave us fully at orangutan level.

Comment author: thomblake 23 July 2012 05:11:16PM 4 points [-]

Even the most sophisticated AI for driving cars cannot drive on a real road,

This is false. Though currently there are situations that may come up that will prompt it to give up control to the human driver, and there are some situations (such as high reflectivity / packed snow) that they can't handle yet.

Comment author: [deleted] 21 July 2012 06:00:25PM 4 points [-]

1: Utility is heterogeneous. Which gives you more "utility", a bowl of ice cream or a chair? The question itself is nonsensical, the quality/type of utility gained from a chair and a bowl of ice cream are entirely different.

It's not nonsensical; it means “would you rather have a bowl of ice cream or a chair?” Of course the answer is “it depends”, but no-one ever claimed that U(x + a bowl of ice cream) − U(x) doesn't depend on x.

Comment author: hairyfigment 21 July 2012 07:06:16PM 3 points [-]

To focus on one problem with this, you write:

treating intelligence as a machine that optimizes things is missing the entire point of intelligence. If you had ever read Douglas Hofstadter's "Godel Escher Bach", or Christopher Alexander's "The Nature of Order" series, you might have a greater appreciation for the role that abstract pattern recognition and metaphor plays in intelligence.

Eliezer has read GEB and praised it above the mountains (literally). So a charitable reader of him and his colleagues might suppose that they know the point about pattern recognition, but do not see the connection that you find obvious. And in fact I don't know what you're responding to, or what you think your second quoted sentence has to do with the first, or what practical conclusion you draw from it through what argument. Perhaps you could spell it out in detail for us mortals?

Comment author: wedrifid 21 July 2012 06:58:54PM 4 points [-]

As for "utility function", there are at least three reasons why it is not just wrong, but entirely impossible.

You don't understand what that term means.

Comment author: David_Gerard 21 July 2012 11:35:00AM 2 points [-]

Finally, I read two "papers" from SI, and found them entirely unprofessional.

Which two papers, by the way?

Comment author: DaFranker 20 July 2012 09:42:28AM * 1 point [-]

In short, impressive "logical" arguments about how probabilities of complements must be additive can only be justified in a vacuum without context, a situation that does not exist in the real world.

Every "context" can be described as a set of facts and parameters, AKA more data. Perfect data on the context means perfect information. Perfect information means perfect choice and perfect predictions. Sure, it might seem to you like the logical arguments expressed are "too basic to apply to the real world", but a utility function is really only ever "wrong" when it fails to apply the correct utility to the correct element ("sorting out your priorities"), whether that's by improper design, lack of self-awareness, missing information or some other hypothetical reason.

For every "no but theory doesn't apply to the real world" or "theory and practice are different" argument, there is always an explanation for the proposed difference between theory and reality, and this explanation can be included in the theory. The point isn't to throw out reality and use our own virtual-theoretical world. It's to update our model (the theory) in the most sane and rational way, over and over again (constantly and continuously) so that we get better.

Likewise, maximizing one's own utility function is not the reduce-oneself-to-machine-worshipper-of-the-machine-god that you seem to believe. I have emotions, I get angry, I get irritated (e.g. at your response*), I am happy, etc. Yet it appears that for several years, in hindsight, I've been maximizing my utility function without knowing that that's how it's called (and learning the terminology and more correct/formal ways of talking about it once I started reading LessWrong).

Your "utility function" is not one simple formula that you use to plug in values to variables, compute, and then call it a decision. The utility function of a person is the entire, general completeness of what that person wants and desires and values. If I tried to write down for you my own utility function, it would be both utterly incomprehensible and probably ridiculously ugly. That's assuming I'd even be capable of writing it all down - limited self-awareness, biases, continuous change, and all that stuff.

To put it all in perspective, "maximizing one's utility function" is very much equivalent to "according to what information you have, spend as much time as you think is worth taking deciding on the probably-best course of action available, and then act on it, such that in hindsight you'll have maximized your chances of reaching your own objectives". This doesn't mean obtaining perfect information or never being wrong or worshipping a formula. It simply means living your own life, in your own way, with better (and improving) awareness of yourself and updating (changing) your own beliefs when they're no longer correct so that you can act and behave more rationally. In this optic, LessWrong is essentially a large self-help group for normal people who just want to be better at knowing things and making decisions in general.

On a last note, FacingTheSingularity does not contain a bunch of scientific essays that would be the end answer to all singularity concerns. At best, it could be considered as one multi-chapter essay going through various points to support the primary thesis that the one author believes that the various experts are right about the Singularity being "imminent" (within this century at the outset). This is clearly stated on the front page, which is also the table of contents. As I've said in my previous reply, it's a good vulgarized introduction. However, the real meat comes from the SingInst articles, essays and theses, as well as some of the more official stuff on LessWrong. Eliezer's Timeless Decision Theory paper is a good example of more rigorous and technical writing, though it's by far not the most relevant, nor do I think it's the first one that a newcomer should read. If you're interested in possible AI decision-making techniques, though, it's a very interesting and pertinent reading.

*(I was slightly irritated that I failed to fully communicate my point and at the dismissal of long-thought-and-debated theories, including beliefs I've revalidated time and time again over the years, along with the childish comment on ex-christians and their "machine god". This does not mean, however, that I transpose this irritation towards you or some other, unrelated outlet. My irritation is my own and a product of my own mental models.)

Edit: Fixed some of the text and added missing footnote.

Comment author: homunq 23 July 2012 04:58:10PM 1 point [-]

I don't pretend I've read every refutation of Aeonios's arguments that's out there, but I've read a few. Generally, those "refutations" strike me as plausible arguments by smart people, but far from bulletproof. Thus, I think that your [DaFranker's] attitude of "I know better so I barely have time for this" isn't the best one.

(I'm sorry, I don't have time to get into the details of the arguments themselves, so this post is all meta. I realize that that's somewhat hypocritical, but "hypocrisy is the tribute vice pays to virtue" so I'm OK with that.)

Comment author: DaFranker 23 July 2012 10:26:33PM 0 points [-]

Indeed, most of them are nothing but smart arguments by smart people, and have not been formally proven. However, none of the arguments for anything in AI research is formally proven, except for some very primitive mathematics and computer science stuff. Basically, at the moment all we have to go on is a lot of thought, some circumstantial "evidence" and our sets of beliefs.

All I'm saying is that, if you watch the trend, it's much more likely (with my priors, at least) that the S.I. is "right" and that the arguments that keep being brought against it are unenlightened, in light of a few key observables; each argument against S.I. being "refuted" one after another historically, most of the critics of the S.I. not having spent nearly as much time thinking about the issues at hand and actually researching AIs, etc.

It's not that I know better, merely that with the evidence presented to me from "both sides" (if one were to arbitrarily delimit two specific opposing factions, for simplification) and my own knowledge of the world seem to indicate towards the "S.I. side" having propositions which are much more likely to be true. I'll admit that the end result does project that attitude, but this is mainly incidental from the fact that I actually was pressed for time when I wrote that particular post, and I did believe true that it be pointless to discuss and argument further for the benefit of an outsider that hadn't yet read the relevant material on the topic at hand.

Comment author: homunq 24 July 2012 02:13:25AM * 2 points [-]

But in this case, "more likely to be true" means something like "a good enough argument to move my priors by roughly an order of magnitude, or two at the outside". Since in the face of our ignorance of the future, reasonable priors could differ by several orders of magnitude, even the best arguments I've seen aren't enough to dismiss any "side" as silly or not worthy of further consideration (except stuff that was obviously silly to begin with).

Comment author: DaFranker 24 July 2012 01:56:31PM * 1 point [-]

That's a very good point.

I was intuitively tempted to retort a bunch of things about likelyness of exception and information taken into consideration, but I realized before posting that I was actually falling victim to several biases in that train of thought. You've actually given me a new way to think of the issue. I'm still of the intuition that any new way to think about it will only reinforce my beliefs and support the S.I. over time, though.

For now, I'm content to concede that I was weighing too heavily on my priors and my confidence in my own knowledge of the universe (on which my posteriors for AI issues inevitably depend, in one way or another), among possibly more mistakes. However, it seems at first glance to be even more evidence for the need of a new mathematical or logical language to discuss these questions more in depth, detail and formality.

Comment author: Nautilus 21 July 2012 02:11:33PM 0 points [-]

general lack of sensitivity to culture and art must be dealt with first

Where'd that come from? Are you an artists / anthropologist?

Aeonios comments on Reply to Holden on The Singularity Institute - Less Wrong

Article Navigation

Comments (213)

Nearest Meetups

Virtual Study Room

Recent Comments

Recent Posts

Recent Rationality Quotes

Recent Wiki Edits

Top Contributors, 30 Days

Recent Karma Awards

Comments (213)

Top Contributors, 30 Days

You'll need to login or register to do that

(Don't worry, it only takes a few seconds)

Create

Login