Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Is That Your True Rejection?

41 Post author: Eliezer_Yudkowsky 06 December 2008 02:26PM

It happens every now and then, that the one encounters some of my transhumanist-side beliefs—as opposed to my ideas having to do with human rationality—strange, exotic-sounding ideas like superintelligence and Friendly AI.  And the one rejects them.

If the one is called upon to explain the rejection, not uncommonly the one says,

"Why should I believe anything Yudkowsky says?  He doesn't have a PhD!"

And occasionally someone else, hearing, says, "Oh, you should get a PhD, so that people will listen to you."  Or this advice may even be offered by the same one who disbelieved, saying, "Come back when you have a PhD."

Now there are good and bad reasons to get a PhD, but this is one of the bad ones.

There's many reasons why someone actually has an adverse reaction to transhumanist theses.  Most are matters of pattern recognition, rather than verbal thought: the thesis matches against "strange weird idea" or "science fiction" or "end-of-the-world cult" or "overenthusiastic youth".

So immediately, at the speed of perception, the idea is rejected.  If, afterward, someone says "Why not?", this lanches a search for justification.  But this search will not necessarily hit on the true reason—by "true reason" I mean not the best reason that could be offered, but rather, whichever causes were decisive as a matter of historical fact, at the very first moment the rejection occurred.

Instead, the search for justification hits on the justifying-sounding fact, "This speaker does not have a PhD."

But I also don't have a PhD when I talk about human rationality, so why is the same objection not raised there?

And more to the point, if I had a PhD, people would not treat this as a decisive factor indicating that they ought to believe everything I say.  Rather, the same initial rejection would occur, for the same reasons; and the search for justification, afterward, would terminate at a different stopping point.

They would say, "Why should I believe you?  You're just some guy with a PhD! There are lots of those.  Come back when you're well-known in your field and tenured at a major university."

But do people actually believe arbitrary professors at Harvard who say weird things?  Of course not.  (But if I were a professor at Harvard, it would in fact be easier to get media attention.  Reporters initially disinclined to believe me—who would probably be equally disinclined to believe a random PhD-bearer—would still report on me, because it would be news that a Harvard professor believes such a weird thing.)

If you are saying things that sound wrong to a novice, as opposed to just rattling off magical-sounding technobabble about leptical quark braids in N+2 dimensions; and the hearer is a stranger, unfamiliar with you personally and with the subject matter of your field; then I suspect that the point at which the average person will actually start to grant credence overriding their initial impression, purely because of academic credentials, is somewhere around the Nobel Laureate level.  If that.  Roughly, you need whatever level of academic credential qualifies as "beyond the mundane".

This is more or less what happened to Eric Drexler, as far as I can tell.  He presented his vision of nanotechnology, and people said, "Where are the technical details?" or "Come back when you have a PhD!"  And Eric Drexler spent six years writing up technical details and got his PhD under Marvin Minsky for doing it.  And Nanosystems is a great book.  But did the same people who said, "Come back when you have a PhD", actually change their minds at all about molecular nanotechnology?  Not so far as I ever heard.

It has similarly been a general rule with the Singularity Institute that, whatever it is we're supposed to do to be more credible, when we actually do it, nothing much changes.  "Do you do any sort of code development?  I'm not interested in supporting an organization that doesn't develop code"—> OpenCog—> nothing changes.  "Eliezer Yudkowsky lacks academic credentials"—> Professor Ben Goertzel installed as Director of Research—> nothing changes.  The one thing that actually has seemed to raise credibility, is famous people associating with the organization, like Peter Thiel funding us, or Ray Kurzweil on the Board.

This might be an important thing for young businesses and new-minted consultants to keep in mind—that what your failed prospects tell you is the reason for rejection, may not make the real difference; and you should ponder that carefully before spending huge efforts.  If the venture capitalist says "If only your sales were growing a little faster!", if the potential customer says "It seems good, but you don't have feature X", that may not be the true rejection.  Fixing it may, or may not, change anything.

And it would also be something to keep in mind during disagreements.  Robin and I share a belief that two rationalists should not agree to disagree: they should not have common knowledge of epistemic disagreement unless something is very wrong.

I suspect that, in general, if two rationalists set out to resolve a disagreement that persisted past the first exchange, they should expect to find that the true sources of the disagreement are either hard to communicate, or hard to expose.  E.g:

  • Uncommon, but well-supported, scientific knowledge or math;
  • Long inferential distances;
  • Hard-to-verbalize intuitions, perhaps stemming from specific visualizations;
  • Zeitgeists inherited from a profession (that may have good reason for it);
  • Patterns perceptually recognized from experience;
  • Sheer habits of thought;
  • Emotional commitments to believing in a particular outcome;
  • Fear of a past mistake being disproven;
  • Deep self-deception for the sake of pride or other personal benefits.

If the matter were one in which all the true rejections could be easily laid on the table, the disagreement would probably be so straightforward to resolve that it would never have lasted past the first meeting.

"Is this my true rejection?" is something that both disagreers should surely be asking themselves, to make things easier on the Other Fellow.  However, attempts to directly, publicly psychoanalyze the Other may cause the conversation to degenerate very fast, in my observation.

Still—"Is that your true rejection?" should be fair game for Disagreers to humbly ask, if there's any productive way to pursue that sub-issue.  Maybe the rule could be that you can openly ask, "Is that simple straightforward-sounding reason your true rejection, or does it come from intuition-X or professional-zeitgeist-Y?"  While the more embarrassing possibilities lower on the table are left to the Other's conscience, as their own responsibility to handle.

Post scriptum:

This post is not really about PhDs in general, or their credibility value in particular.  But I've always figured that to the extent this was a strategically important consideration, it would make more sense to recruit an academic of existing high status, than spend a huge amount of time trying to achieve low or moderate academic status.

However, if any professor out there wants to let me come in and just do a PhD in analytic philosophy—just write the thesis and defend it—then I have, for my own use, worked out a general and mathematically elegant theory of Newcomblike decision problems.  I think it would make a fine PhD thesis, and it is ready to be written—if anyone has the power to let me do things the old-fashioned way.


Part of the Against Rationalization subsequence of How To Actually Change Your Mind

Next post: "Entangled Truths, Contagious Lies"

Previous post: "Fake Optimization Criteria"

Comments (81)

Sort By: Old
Comment author: RobinHanson 06 December 2008 03:08:37PM 22 points [-]

There need not be just one "true objection"; there can be many factors that together lead to an estimate. Whether you have a Ph.D., and whether folks with Ph.D. have reviewed your claims, and what they say, can certainly be relevant. Also remember that you should care lots more about the opinions of experts that could build on and endorse your work, than about average Joe opinions. Very few things ever convince average folks of anything unusual; target a narrower audience.

Comment author: buybuydandavis 15 December 2012 04:47:01PM 1 point [-]

There need not be just one "true objection"; there can be many factors that together lead to an estimate.

And more than one of them may be considered decisive by themselves.

Comment author: Vladimir_Slepnev 06 December 2008 03:18:57PM 4 points [-]

Immediate association: pick-up artists know well that when a girl rejects you, she often doesn't know the true reason and has to deceive herself. You could recruit some rationalists among PUAs. They wholeheartedly share your sentiment that "rational agents must WIN", and have accumulated many cynical but useful insights about human mating behaviour.

Comment author: poke 06 December 2008 03:37:45PM 24 points [-]

Most transhumanist ideas fall under the category of "not even wrong." Drexler's Nanosystems is ignored because it's a work of "speculative engineering" that doesn't address any of the questions a chemist would pose (i.e., regarding synthesis). It's a non-event. It shows that you can make fancy molecular structures under certain computational models. SI is similar. What do you expect a scientist to say about SI? Sure, they can't disprove the notion, but there's nothing for them to discuss either. The transhumanist community has a tendency to argue for its positions along the lines of "you can't prove this isn't possible" which is completely uninteresting from a practical viewpoint.

If I was going to depack "you should get a PhD" I'd say the intention is along the lines of: you should attempt to tackle something tractable before you start speculating on Big Ideas. If you had a PhD, maybe you'd be more cautious. If you had a PhD, maybe you'd be able to step outside the incestuous milieu of pop sci musings you find yourself trapped in. There's two things you get from a formal education: one is broad, you're exposed to a variety of subject matter that you're unlikely to encounter as an autodidact; the other is specific, you're forced to focus on problems you'd likely dismiss as trivial as an autodidact. Both offer strong correctives to preconceptions.

As for why people are less likely to express the same concern when the topic is rationality; there's a long tradition of disrespect for formal education when it comes to dispensing advice. Your discussions of rationality usually have the format of sage advice rather than scientific analysis. Nobody cares if Dr. Phil is a real doctor.

Comment author: ksvanhorn 18 January 2011 07:35:12PM 21 points [-]

"There's two things you get from a formal education: one is broad, you're exposed to a variety of subject matter that you're unlikely to encounter as an autodidact;"

As someone who has a Ph.D., I have to disagree here. Most of my own breadth of knowledge has come from pursuing topics on my own initiative outside of the classroom, simply because they interested me or because they seemed likely to help me solve some problem I was working on. In fact, as a grad student, most of the things I needed to learn weren't being taught in any of the classes available to me.

The choice isn't between being an autodidact or getting a Ph.D.; I don't think you can really earn the latter unless you have the skills of the former.

Comment author: RichardKennaway 18 April 2012 11:35:15AM 6 points [-]

In fact, as a grad student, most of the things I needed to learn weren't being taught in any of the classes available to me.

But being a grad student gave you the need to learn them.

Comment author: FeepingCreature 26 November 2012 03:50:28AM 8 points [-]

Or a common factor caused both.

Comment author: DanielLC 17 April 2012 04:33:46PM 13 points [-]

If you had a PhD, maybe you'd be able to step outside the incestuous milieu of pop sci musings you find yourself trapped in.

That sounds like it's less "Once you get a Ph.D., I'll believe you," than "Once you get a Ph.D., you'll stop believing that."

Comment author: komponisto 17 April 2012 10:42:12PM 10 points [-]

Of course, those aren't so different: if I expect that getting a Ph. D would make one less likely to believe X, then believing X after getting a Ph.D is a stronger signal than simply believing X.

Comment author: Eliezer_Yudkowsky 06 December 2008 03:39:51PM 8 points [-]

Vladimir, I don't quite think that's the "narrower audience" Robin is talking about...

Robin, see the Post Scriptum. I would be willing to get a PhD thesis if it went by the old rules and the old meaning of "Prove you can make an original, significant contribution to human knowledge and that you've mastered an existing field", rather than, "This credential shows you have spent X number of years in a building." (This particular theory would be hard enough to write up that I may not get around to it if a PhD credential isn't at stake.)

Comment author: kalla724 27 June 2011 08:11:14PM 10 points [-]


See poke's comment above (which is so on the nose, it actually inspired me to register). Then consider the following.

You will never get a PhD in the manner you propose, because that would fulfill only a part of the purpose of a PhD. The number of years spent in the building can be (and in too many cases is) wasted time - but if things are done in a proper manner, this time (which can be only three or four years) is critical.

For science PhDs specifically, the idea isn't to just come up with something novel and write it up. The idea is to go into the field with a question that you don't have an answer for, not yet. To find ways to collect data, and then to actually collect it. To build intricate, detailed models that answer your question precisely and completely, fitting all the available data. To design experiments specifically so you can test your models. And finally, to watch these models completely and utterly fail, nine times out of ten.

They won't fail because you missed something while building them. They will fail because you could only test them properly after making them. If you just built the model that fit everything, and then never tested it with specific experiments... you could spend a very long time convinced that you have found the truth. Several iterations of this process makes people far less willing to extrapolate beyond the available data - certainly not nearly as wildly and as far as transhumanists do.

A good philosophy PhD can do the same, but it is much more difficult to get an optimal result.

Don't take this the wrong way. I respect and admire your achievements, and I think getting a PhD would be a waste of time for you at this point. But it is entirely true that getting one - a real one - would increase the acceptance of your thoughts and ideas. Not (just) because a PhD would grant you prestige, but because your thoughts and ideas would actually get better.

Which finally brings us to the reason for the dichotomy you noted in your post. Your rationality musings are accepted because a) they are inspiring, and b) they can be actionable and provide solid feedback. A person can read them, try the ideas out, and see if those ideas work for them. Transhumanism, alas, falls under "half-baked" category; and the willingness to follow wildly speculative tangents from poorly constrained models... well, in order to have any weight there, you better either show concrete, practical results... or have credentials that show you have experienced significant model failure in the past. Repeatedly. And painfully. With significant cost to yourself.

Comment author: p4wnc6 28 June 2011 05:03:29AM *  30 points [-]

As a current grad student myself, I could not disagree with poke's comment and this comment more. I work for a very respected adviser in computer vision from a very prestigious university. The reason I was accepted to this lab is because I am an NDSEG fellow. Many other qualified people lost out because my attendance here frees up a lot of my adviser's money for more students. In the mean time, I have a lot of pretty worthwhile ideas in physical vision and theories of semantic visual representations. However, I spend most of my days building Python GUI widgets for a group of collaborating sociologists. They collect really mundane data by annotating videos and no off the shelf stuff does quite what they want... so guess who gets to do that grunt work for a summer? Grad students.

You should really read the good Economist article The Disposable Academic. Graduate studentships are business acquisitions in all but the utmost theoretical fields. Advisers want the most non-linear things imaginable. For example, I am a pure math guy, with heavy emphasis on machine learning methods and probability theory. Yet my day job is seriously creative-energy-draining Python programming. The programming isn't even related to original math, it's just a novel thing for some sociologists to use. My adviser doesn't want me to split my time between reading background theory, etc. He wants me to develop this code because it makes him look good in front of collaborators.

Academia is mostly a bad bad place. I think Eliezer's desire to circumvent all the crap of grad school is totally right. The old way was a real, true apprenticeship. It isn't like that anymore. Engineering is especially notorious for this. Minimize the amount of tenured positions, and balloon the number of grad students in order to farm out the work that profs don't want to do. For almost all of these people, they will just go through the motions, do mundane bullshit, and write a thesis not really worth the paper it gets printed on. The few who don't follow this route usually just take it upon themselves to go and read on their own and become experts across many different disciplines and then make interconnections between previously independent fields or results. Eliezer has certainly done this with discussions of Newcombe-like problems and friendly A.I. from a philosophical perspective. He's done more honest academic work here than almost anyone I know in academia.

When I used to work at MIT Lincoln Laboratory, a colleague of mine had a great saying about grad school: "Grad school is 99% about putting your ass in the chair." It is indeed about spending X years in building Y and getting Z publications. Pure mathematics is somewhat of an exception at elite schools. To boot, people don't take you any more seriously when you finish. It continues on as a political/social process where you must win grants and provide widgets for collaborators and funding agencies.

There is much more to be said, and of course, I am a grad student, so I must feel that at least for myself it is a good decision despite all of the issues. Well, that's not quite true. Part of it is that as an undergrad, all of my professors just paid attention to the fact that I was energetic and attentive when they talked about topics they liked, and it created a cumulative jazzed-up feeling for those topics. I expected grad school to be very different than it really is. I should also add that I have been in two different PhD programs, both Ivy League (I can talk more specifically in private). I transferred because the adviser situation at the first school was pretty grim. There was only one faculty member doing things close to what I wanted to study, and he was such a famous name that he had not the time of day for me. For example, I once scheduled a meeting with him to discuss possible research and when I arrived he let me know it was going to be a jointly held meeting with his current doctoral student. While that student wrote on a chalkboard, I got to ask questions. When the student was finished, this prof addressed the student for various intervals of time and then came back to me. This sort of pedantic garbage is the rule rather than the exception.

I find myself having to constantly fight the urge go home from a long, wrist-achingly terrible day of mundane Python programming and just mentally check out. Instead, I read about stuff here on LW, or I read physics books, or now A.I. complexity books. Hopefully my thesis will be a contribution that I enjoy and find interesting. Even better if it helps move science along in a "meaningful way", but the standard PhD process is absolutely not going to let that happen unless I actively intervene and do things all on my own.

Anyone considering a PhD should consider this heavily. My experience is that it is nothing like the description above or poke's comment. I think Bostrom should advise a thesis with Eliezer because it would be a great addition to philosophy, and I don't want Eliezer burdened with nuisance coursework requirements. We should be unyoking uncommonly smart people when we find them, not forcing them to jump through extra hoops just for the pedantic sake of standardization.

Comment author: kalla724 13 July 2011 11:40:07PM 5 points [-]

Ok, so - I hear what you're saying, but a) that is not the way it's supposed to be, and b) you are missing the point.

First, a), even in the current academia, you are in a bad position. If I were you, I would switch mentors or programs ASAP.

I understand where you're coming from perfectly. I had a very similar experience: I spent three years in a failed PhD (the lab I was working in went under at the same time as the department I was in), and I ended up getting a MS instead. But even in that position, which was all tedious gruntwork, I understood the hypothesis and had some input. I switched to a different field, and a different mentor, where most of my work was still tedious, but it was driven by ideas I came to while working with my adviser.

If your position is, as it seems to be, even worse - that you have NO input whatsoever, and are purely cheap labor - then you should switch mentors immediately. If you don't, you might finish your PhD with a great deal of bitterness, but it is much more likely that you will simply burn out and drop out.

Which brings me to b). As I said above, it would be pointless for Eliezer to go to grad school now. Even at best, it contains a lot of tedious, repetitive work. But the essential point stands: in a poorly constrained area such as transhumanism, grand ideas are not enough. That is where PhD does have a function, and does have a reason.

Comment author: p4wnc6 14 July 2011 01:09:20AM *  16 points [-]

Actually, my mentor is among one of the nicest guys around and is a good manager, offers good advice, and has a consistent record of producing successful students. It's just that almost no grad student gets to have real input in what they are doing. If you do have that, consider yourself lucky, because the dozens of grad students that I know aren't in a position like that. I just had a meeting today where my adviser talked to me about having to balance my time between "whatever needs doing" (for the utility of our whole research group rather than just my own dissertation) and doing my own reading/research. His idea (shared by many faculty members) is that for a few years at the front end of the PhD, you mostly do about 80% general utility work and infrastructure work, just to build experience, write code, get involved... then after you get into some publications a few years later, the roles switch and you shift to more like 80% writing and doing your own thing (research). The problem is that if you're a passionate student with good ideas, then that first few years of bullshit infrastructure work is a complete waste of time. The run-of-the-mill PhD student (who generally is not really all that smart or hardworking) might do fine just being told what to program for a few years, but the really intellectually curious people will die inside. Plus, for those really smart PhD students out there, I think it's in my best interest that they be cleared of all nuisance responsibilities and allowed to just work.

As for point (b), my reasoning would be like this: "in a poorly constrained area such as transhumanism, grand ideas are not enough, and that's exactly why a PhD is irrelevant and I have no reason to think that someone has better or more useful or more important ideas than Eliezer just because that person has been published in peer reviewed journals."

Comment author: RPMcMurphy 06 October 2014 10:26:16PM -2 points [-]
Comment author: billswift 06 December 2008 04:03:27PM 0 points [-]

Robin: Of course a PhD in "The Voodoo Sciences" isn't going to help convince anybody competent of much. I am actually more impressed with some of the fiction I vaguely remember you writing for Pournelle's "Endless Frontier" collections than a lot of what I've read recently here.

Poke: "formal education: one is broad, you're exposed to a variety of subject matter that you're unlikely to encounter as an autodidact"

I used to spend a lot of time around the Engineering Library at the University of Maryland, College Park before I moved away. In more than ten years I have never met anyone there as widely read as myself. This also brings to mind a quote from G Harry Stine's "The Hopeful Future" about how the self-taught are usually deficient in some areas of their learning - maybe so, but the self-taught will simply extend their knowledge when a lack appears to them. My lacks are from a lack of focus not breadth. Everybody I have ever met at the University was the other way around, narrow and all too often not even aware of how narrow they were.

Comment author: Aron 06 December 2008 04:23:43PM 3 points [-]

Perhaps you are marginally ahead of your time Eliezer, and the young individuals that will flush out the theory are still traipsing about in diapers. In which case, either being a billionare or a phD makes it more likely you can become their mentor. I'd do the former if you have a choice.

Comment author: michael_vassar3 06 December 2008 05:38:46PM 5 points [-]

Can't do basic derivatives? Seriously?!? I'm for kicking the troll out. His bragging about mediocre mathematical accomplishments isn't informative or entertaining to us readers.

Comment author: Jef_Allbright 06 December 2008 06:24:50PM -1 points [-]

billswift wrote:

‌but the self-taught will simply extend their knowledge when a lack appears to them.

Yes, this point is key to the topic at hand, as well as to the problem of meaningful growth of any intelligent agent, regardless of its substrate and facility for (recursive) improvement. But in this particular forum, due to the particular biases which tend to predominate among those whose very nature tends to enforce relatively narrow (albeit deep) scope of interaction, the emphasis should be not on "will simply extend" but on "when a lack appears."

In this forum, and others like it, we characteristically fail to distinguish between the relative ease of learning from the already abstracted explicit and latent regularities in our environment and the fundamentally hard (and increasingly harder) problem of extracting novelty of pragmatic value from an exponentially expanding space of possibilities.

Therein lies the problem—and the opportunity—of increasingly effective agency within an environment of even more rapidly increasing uncertainty. There never was or will be safety or certainty in any ultimate sense, from the point of view of any (necessarily subjective) agent. So let us each embrace this aspect of reality and strive, not for safety but for meaningful growth.

Comment author: Ian_C. 06 December 2008 06:39:59PM 9 points [-]

Eliezer, I'm sure if you complete your friendly AI design, there will be multiple honorary PhDs to follow.

Comment author: BrandNameThinker 06 December 2008 06:57:03PM 12 points [-]

I have spent years in the Amazon Basin perfecting the art of run-on sentences and hubris it helps remind others of my shining intellect it also helps me find attractive women who love the smell of rich leather furnishings and old books.

Between bedding supermodels a new one each night, I have developed a scientific thesis that supersedes your talk of Solomonoff and Kolmogorov and any other Russian name you can throw at me. Here are a random snippet of conclusions a supposedly intelligent person will arrive having been graced by my mathematical superpowers:

1. Everything you thought you knew about Probability is wrong. 2. Existence is MADE of Existence. 3. Einstien didn't know this, but slowly struggled toward my genius insight. 4. They mocked me when I called myself a modern day Galileo, but like Bean I will come back after they have gone soft.

I can off the tip of my rather distinguished salt-and-pepper beard name at least 108 other conclusions that would startle lesser minds such as the John BAEZ the very devil himself or Adolf Hitler I have really lost my patience with you ElIzer.

They called me mad when I reinvented calculus! They will call me mad no longer oh I have to go make the Sweaty Wildebeest with a delicately frowning Victoria's Secret model.

Comment author: p4wnc6 28 June 2011 05:11:16AM 5 points [-]

time cube!

Comment author: SpaceFrank 03 February 2012 03:27:02PM *  3 points [-]

I wish I could upvote this post back into the positive.

(It seems pretty obvious to me that is a direct satire of the previous post by a similar username. What, no love for sarcasm?)

Comment author: [deleted] 03 February 2012 07:32:33PM 7 points [-]

Poe's Law, anyone?

Comment author: SpaceFrank 03 February 2012 08:18:08PM 2 points [-]

I had to look it up, but I definitely agree. Especially considering how quickly the karma changes reversed after I edited in that footnote.

Comment author: DanielLC 17 April 2012 04:37:58PM 3 points [-]

I thought it was obvious.

Maybe I'm just spoiled by the consistently good comments on LessWrong, and don't realize that there actually are comments that bad and cliched.

Comment author: [deleted] 17 April 2012 09:36:02PM 2 points [-]

Given that particular misspelling of Einstein and the mention of Baez, it was nearly impossible that BrandNameThinker hadn't heard of the crackpot index... But what's obvious for me (and you) needn't be obvious for someone else. (Or maybe the downvoters did get the joke but just didn't find it funny.)

Comment author: Jef_Allbright 06 December 2008 07:03:30PM -1 points [-]

Crap. Will the moderator delete posts like that one, which appear to be so off the Mark?

Comment author: Richard4 06 December 2008 07:17:41PM 17 points [-]

Eliezer - 'I would be willing to get a PhD thesis if it went by the old rules and the old meaning of "Prove you can make an original, significant contribution to human knowledge and that you've mastered an existing field", rather than, "This credential shows you have spent X number of years in a building."'

British and Australasian universities don't require any coursework for their PhDs, just the thesis. If you think your work is good enough, write to Alan Hajek at ANU and see if he'd be willing to give it a look.

Comment author: Paul_Gowder2 06 December 2008 07:19:50PM 9 points [-]

Ignoring the highly unlikely slurs about your calculus ability:

However, if any professor out there wants to let me come in and just do a PhD in analytic philosophy - just write the thesis and defend it - then I have, for my own use, worked out a general and mathematically elegant theory of Newcomblike decision problems. I think it would make a fine PhD thesis, and it is ready to be written - if anyone has the power to let me do things the old-fashioned way.

British universities? That's the traditional place to do that sort of thing. Oxbridge.

Comment author: HalFinney 06 December 2008 08:56:35PM 1 point [-]

Specifically with regard to the apparent persistent disagreement between you and Robin, none of those things explain it. You guys could just take turns doing nothing but calling out your estimates on the issue in question (for example, the probability of a hard takeoff AI this century), and you should reach agreement within a few rounds. The actual reasoning behind your opinions has no bearing whatsoever on your ability to reach agreement (or more precisely, on your inability to maintain disagreement).

Now, this is assuming that you both are honest and rational, and view each other that way. Exposing your reasoning may give one or the other of you grounds to question these attributes, and that may explain a persistent disagreement.

It is also useful to discuss your reasoning, if case your goal is not to simply reach agreement, but to get the right answer. It is possible that this is the real explanation behind your apparent disagreement. You might be able to reach agreement relatively quickly by fiat, but one or both of you would still be left puzzled about how things could be so different from what your seemingly very valid reasoning led you to expect. You would still want to hash over the issues and talk things out.

Robin earlier posted, "since this topic is important, my focus here is on gaining a better understanding of it". I read this as suggesting that his goal is not merely to resolve the disagreement, and perhaps not to particularly pursue the disagreement aspects at all. He also pointed out, "you do not know that I have not changed my opinion since learning of Eliezer's opinion, and I do not assume that he has not changed his opinion." This is consistent with the possibility that there is no disagreement at all, and that Robin and possibly Eliezer have changed their views enough that they substantially agree.

Robin has also argued that there is no reason for agreement to limit vigorous dissension and debate about competing views. In effect, people would act as devil's advocates, advancing ideas and positions that they thought were probably wrong, but which still deserved a hearing. It's possible that he has come to share Eliezer's position yet continues to challenge it along the lines proposed in that posting.

One thing that bothers me, as an observer who is more interested in the nature of disagreement than the future course of humanity and its descendants, is that Robin and Eliezer have not taken more opportunity to clarify these matters and to lay out the time course of their disagreement more clearly. It would help too for them to periodically estimate how likely they think the other is to be behaving as a rational, honest, "Bayesian wannabe". As two of the most notable wannabe's around, both very familiar with the disagreement theorems, both highly intelligent and rational, this is a terrible missed opportunity. I understand that Robin's goal may be as stated, to air the issues, but I don't see why they can't simultaneously serve the community by shedding light on the nature of this disagreement.

Comment author: Eliezer_Yudkowsky 06 December 2008 09:21:14PM 3 points [-]

And with that lovely exhibition of math talent, combined with the assertion that he skipped straight to grad school in mathematics, I do hereby request GenericThinker to cease and desist from further commenting on Overcoming Bias.

Comment author: Jon2 06 December 2008 10:34:12PM 10 points [-]


The y appears on both sides of the equation, so these are differential equations. To avoid confusion, re-write as:

(1) (d/dt) F(t) = A*F(t) (2) (d/dt) F(t) = e^F(t)

Now plug e^At into (1) and -ln(C-t) into (2), and verify that they satisfy the condition.

Comment author: Brian7 07 December 2008 12:40:43AM -3 points [-]

You could recruit some rationalists among PUAs. They wholeheartedly share your sentiment that "rational agents must WIN"

Interesting. As a reasonable approximation, approaching women with confidence==one-boxing on Newcomb's problem. Eliezer's posts have increased my credence that the latter is correct, although it hasn't helped me with the former.

Comment author: frelkins 07 December 2008 01:37:51AM 0 points [-]


I think Alec Greven may be your man. Or perhaps like Lucy van Pelt I should set up office hours offering Love Advice, 5 cents?

Comment author: Daniel3 07 December 2008 03:27:34AM 2 points [-]

You could recruit some rationalists among PUAs. They wholeheartedly share your sentiment that "rational agents must WIN"

You have. We do. And yes, they must.

Comment author: Phil_Goetz7 07 December 2008 03:45:55AM 1 point [-]

"Drexler's Nanosystems is ignored because it's a work of "speculative engineering" that doesn't address any of the questions a chemist would pose (i.e., regarding synthesis)."

It doesn't address any of the questions a chemist would pose after reading Nanosystems.

"As a reasonable approximation, approaching women with confidence==one-boxing on Newcomb's problem."

Interesting. Although I would say "approaching women with confidence is an instance of a class of problems that Newcomb's problem is supposed to represent but does not." Newcomb's problem presents you a situation in which the laws of causality are broken, and then asks you to reason out a solution assuming the laws of causality are not broken.

Comment author: Broggly 21 November 2010 02:46:55PM 1 point [-]

It does? Assuming a deterministic universe (which would seem to be necessary for Omega to predict with 100% certainty whether someone 1-boxes or 2-boxes) then it isn't the act of taking one or two boxes that causes the $1M to be present or absent, but rather both those events share a common cause (namely whatever circumstances were present to cause the subject to pick their Newcombe strategy).

If Omega can't predict with certainty then two-boxing is possibly the right answer, depending on exactly how good Omega is at "reading" people and how much money is potentially in the boxes. If this really happened to me, I've got no idea how I would decide. I would probably think some kind of trick was going on and after checking all the fine print for loopholes and one-box the million with some kind of witnesses around to grab the old man behind the curtain and have him charged with fraud if the money wasn't there. If he had some legal escape hatch I could spot, then I'd probably one-box the $1k since I'd think it unlikely someone would give up $999,000 if they could get away with it.

Agree that picking up women isn't newcombelike, though. I get how being nervous can be like 1-boxing the $1k, and being honestly confident might be like one-boxing the $1M (maybe?) but I have no idea what corresponds to 2-boxing.

Comment author: Vladimir_Slepnev 07 December 2008 09:24:31AM 0 points [-]

Daniel, I knew it :-)

Phil, you can look at it another way: the commonality is that to win you have to make yourself believe a demonstrably false statement.

Comment author: Hopefully_Anonymous 07 December 2008 09:34:59AM 2 points [-]

"However, if any professor out there wants to let me come in and just do a PhD in analytic philosophy - just write the thesis and defend it - then I have, for my own use, worked out a general and mathematically elegant theory of Newcomblike decision problems. I think it would make a fine PhD thesis, and it is ready to be written - if anyone has the power to let me do things the old-fashioned way."

I think this is a good idea for you. But don't be surprised if finding the right one takes more work than an occasional bleg. And I do recommend getting it at Harvard or the equivalent. And if I'm not mistaken, you may still have to do a bachelors and masters?

Comment author: apophenia 10 July 2010 06:40:47PM 2 points [-]

My university does not require a masters' degree to get a PhD.

Comment author: Eliezer_Yudkowsky 07 December 2008 11:49:59AM 14 points [-]

If I have to do a bachelors degree, I expect that I can pick up an accredited degree quickly at that university that lets you test out of everything (I think it's called University of Phoenix these days?). No Masters, though, unless there's an org that will let me test out of that.

The rule of thumb here is pretty simple: I'm happy to take tests, I'm not willing to sit in a building for two years solely in order to get a piece of paper which indicates primarily that I sat in a building for two years.

Comment author: irrational 11 December 2013 01:09:31AM 0 points [-]

If you don't have a bachelor's degree, that makes it rather unlikely that you could get a PhD. I agree with folks that you shouldn't bother - if you are right, you'll get your honorary degrees and Nobel prizes, and if not, then not. (I know I am replying to a five-year-old comment).

I also think you are too quick to dismiss the point of getting these degrees, since you in fact have no experience in what that involves.

Comment author: RPMcMurphy 06 October 2014 10:45:10PM *  -2 points [-]

That's what John Gilmore did, among other cool things. http://www.toad.com/gnu/ http://papersplease.org/id.html

John Gilmore writes about his cut-rate degree, here: http://reason.com/archives/2005/04/01/letters

I would really like it if EY had the ability (money, engineering team, etc.) to begin producing brain-like hardware, replete with mirror neurons. I think that's the only way that the sociopaths (prosecutors, judges, politicians, bureaucrats, police) in the coercive sector will be challenged in a meaningful way. I think that it might be smarter for him to relocate MIRI to South Korea, because there's more of a culture of robotics there, and robotics is necessary for feedback regarding real-world problems.

These desires of mine aren't tyrannical. I wouldn't cling to them or try to prescribe actions for EY if he didn't share the same desires. I'm just stating what I would do, if I were suddenly to occupy a leadership position at MIRI, or some similarly-capable organization.

In many ways, this is a deep decision that is based on difficult to quantify innate value judgments. Hawkins was fascinated with brains, and logically, pursued brain design because attempts at getting intelligent, brainlike responses with technology have been so weak in the past, even given approximately adequate computing power. Deep-learning done by Schmidhuber has also recently been productive, given that all computational hardware is enabling far more intelligence, even from systems that were not optimal.

This leads me to believe there will be "many kinds of minds" in the coming singularity. Some, of course, will be superior to others in terms of ability to restructure their environments. Let's hope they aren't sociopathic, or "coercive-human-directed." Remember, even intelligent people can act as sociopaths given good intentions, but the wrong (coercive collectivist) ideas.

Comment author: Phil_Goetz7 07 December 2008 03:28:30PM 1 point [-]

Phil, you can look at it another way: the commonality is that to win you have to make yourself believe a demonstrably false statement.

But I don't. The problem, phrased in a real world situation that could possibly occur, is that a superintelligence is somehow figuring out what people are likely to do, or else is very lucky. The real-world solution is either

1. if you know ahead of time that you're going to be given this decision, either pre-commit to one-boxing, or try to game the superintelligence. Neither option is irrational; it doesn't take any fancy math; one-boxing is positing that your committing to one-boxing has a direct causal effect on what will be in the boxes.

2. if you didn't know ahead of time that you'd be given this decision, choose both boxes.

You can't, once the boxes are on the ground, decide to one-box and think that's going to change the past. That's not the real world, and describing the problem in a way that makes it seem convincing that choosing to one-box actually CAN change the past, is just spinning fantasies.

This is one of a class of apparent paradoxes that arise only because people posit situations that can't actually happen in our universe. Like the ultraviolet catastrophe, or being able to pick any point from a continuum.

Comment author: Vladimir_Nesov 07 December 2008 04:08:44PM 0 points [-]

Phil, your commitment ahead of time is your own private business, your own cognitive ritual. What you need in order to determine the past in the right way is that you are known to perform a certain action in the end. Whether you are arranging it so that you'll perform that action by making a prior commitment and then having to choose the actions because of the penalty, or simply following a timeless decision theory, so that you don't need to bother with prior commitments outside of your cognitive algorithm, is irrelevant. If you are known to follow timeless decision theory, it's just as good as if you are known to have made a commitment. You could say that embracing timeless decision theory is a global meta-commitment, that makes you act as if you made commitment in all the situations where you benefit from having made the commitment.

For example, one-off prisoner's dilemma can be resolved to mutual cooperation by both players making a commitment of huge negative utility in case the other player cooperates and you defect. This commitment doesn't fire in the real world, since its presence makes both players cooperate. The presence of this commitment leads to a better outcome for both players, so it's always rational to make it. Exactly the same effect can be achieved by both players following the timeless decision theory, only without the need to bother arranging that commitment in the environment (which could prove ruinous in case of advanced intelligence, since commitment is essentially intelligence playing adversarial game in the environment against itself).

Comment author: Phil_Goetz5 07 December 2008 05:00:59PM 0 points [-]

Vladimir, I understand the PD and similar cases. I'm just saying that the Newcomb paradox is not actually a member of that class. Any agent faced with either version - being told ahead of time that they will face the Predictor, or being told only once the boxes are on the ground - has a simple choice to make; there's no paradox and no PD-like situation. It's a puzzle only if you believe that there really is backwards causality.

Comment author: Vladimir_Nesov 07 December 2008 06:11:20PM 0 points [-]

Phil, you said "if you didn't know ahead of time that you'd be given this decision, choose both boxes", which is a wrong answer. You didn't know, but the predictor knew what you'll do, and if you one-box, that is your property that predictor knew, and you'll have your reward as a result.

The important part is what predictor knows about your action, not even what you yourself know about your action, and it doesn't matter how you convince the predictor. If predictor just calculates your final action by physical simulation or whatnot, you don't need anything else to convince it, you just need to make the right action. Commitment is a way of convincing, either yourself to make the necessary choice, or your opponent of the fact that you'll make that choice. In our current real world, a person usually can't just say "I promise", without any expected penalty for lying, however implicit, and expect to be trusted, which makes Newcomb's paradox counterintuitive, and which makes cooperating in one-off prisoner's dilemma without pre-commitment unrealistic. But it's a technical problem of communication, or of rationality, nothing more. If predictor can verify that you'll one-box (after you understand the rules of the game, yadda yadda), your property of one-boxing is communicated, and it's all it takes.

Comment author: Phil_Goetz5 07 December 2008 06:35:11PM 0 points [-]

"You didn't know, but the predictor knew what you'll do, and if you one-box, that is your property that predictor knew, and you'll have your reward as a result."

No. That makes sense only if you believe that causality can work backwards. It can't.

"If predictor can verify that you'll one-box (after you understand the rules of the game, yadda yadda), your property of one-boxing is communicated, and it's all it takes."

Your property of one-boxing can't be communicated backwards in time.

We could get bogged down in discussions of free will; I am assuming free will exists, since arguing about the choice to make doesn't make sense unless free will exists. Maybe the Predictor is always right. Maybe, in this imaginary universe, rationalists are screwed. I don't care; I don't claim that rationality is always the best policy in alternate universes where causality doesn't hold and 2+2=5.

What if I've decided I'm going to choose based on a coin flip? Is the Predictor still going to be right? (If you say "yes", then I'm not going to argue with you anymore on this topic; because that would be arguing about how to apply rules that work in this universe in a different universe.)

Comment author: Broggly 21 November 2010 02:59:08PM 0 points [-]

Presumably the Predictor would be smart enough to calculate the result of that coin flip.

If it was an actually random bit, then I don't know. In the real universe, as you require, then the Predictor would have a 50% certainty of being right. Probably if the Predictor thought you might do that it wouldn't offer you the challenge, in order to maintain its reputation for omniscience.

Comment author: Vladimir_Nesov 07 December 2008 06:50:31PM 1 point [-]

Compare: communicating the property of the timer that it will ring one hour in the future (that is, timer works according to certain principles that result in it ringing in the future) vs. communicating from the future the fact that timer ringed. If you can run a precise physical simulation of a coin, you can predict how it'll land. Usually, you can't do that. Not every difficult-seeming prediction requires things like simulation of physical laws, abstractions can be very powerful as well.

Comment author: Phil_Goetz7 08 December 2008 12:41:02AM 0 points [-]

Vladimir, I don't mean to diss you; but I am running out of weekend, and think it's better for me to not reply than to reply carelessly. I don't think I can do much more than repeat myself anyway.

Comment author: Phil_Goetz6 09 December 2008 03:08:34PM 1 point [-]

I may be wrong about Newcomb's paradox.

Comment author: Phil_Goetz6 09 December 2008 03:23:49PM 1 point [-]

You could say that embracing timeless decision theory is a global meta-commitment, that makes you act as if you made commitment in all the situations where you benefit from having made the commitment.

I think this is correct.

It's perplexing: This seems like a logic problem, and I expect to make progress on logic problems using logic. I would expect reading an explanation to be more helpful than having my subconscious mull over a logic problem. But instead, the first time I read it, I couldn't understand it properly because I was not framing the problem properly. Only after I suddenly understood the problem better, without consciously thinking about it, was I able to go back, re-read this, and understand it.

Comment author: Vladimir_Nesov 09 December 2008 03:38:12PM 0 points [-]

I'm glad that helped.

Comment author: Phil_Goetz6 09 December 2008 03:54:49PM 0 points [-]

I don't think it did help, though. I think I failed to comprehend it. I didn't file it away and think about it; I completely missed the point. Later, my subconscious somehow changed gears so that I was able to go back and comprehend it. But communication failed.

Buddhists say that great truths can't be communicated; they have to be experienced, only after which you can understand the communication. This was something like that. Discouraging.

Comment author: Vladimir_Nesov 09 December 2008 04:48:37PM 1 point [-]

From my experience, the most productive way to solve a problem on which I'm stuck (that is, hours of looking at it produce no new insight or promising directions of future investigation), is to keep it in the background for long time, while avoiding forgetting it by recalling what's it about and visualizing its different aspects and related conjectures from time to time. And sure enough, in a few days or weeks, triggered by some essentially unrelated cue, a little insight comes, that allows to develop a new line of thought. When there are several such problem in the background, it's more or less efficient.

Inferential distance can make communication a problem worthy of this kind of reflectively intractable insight.

Comment author: Douglas_Knight3 11 December 2008 06:20:23AM 0 points [-]

Phil, I think that's how logic (or math) normally works. You make progress on logic problems by using logic, but understanding another's solution usually feels completely different to me, completely binary.

Also, it's hard to say that your unconscious wasn't working on it. In particular, I don't know if communicating logic to me is as binary as it feels, whether I go through a search of complete dead ends, or whether intermediate progress is made but not reported.

Comment author: David_Gerard 28 November 2010 08:09:06PM *  3 points [-]

Still - "Is that your true rejection?" should be fair game for Disagreers to humbly ask, if there's any productive way to pursue that sub-issue.

Perhaps it should, but the problem is that answering this question is one of the big problems in salesmanship: working out the customer's true obstacle to wanting to buy from you. Salesmen would love to be able to get a true answer to this question - and some even ask it directly - but people tend to receive this as manipulation: finding out their inner thoughts for purposes of getting their money. This feeling happens when selling someone on an idea as much as it does when selling an item for money.

Vladimir Stepney also notes that they may not know the answer themselves. (Salesmen are aware of this problem too.)

As such, asking the question directly - as you note in your "if" - may end up being counterproductive. If this question occurs to you, you're already in sales mode. If you really want to make that sale, this is a question you have to divine the answer to yourself.

Comment author: epigeios 31 January 2012 07:37:04AM 0 points [-]

Here's something I'd love to put into an entire article, but can't because my karma's bad (see my other comment on this thread):

Many people make the false assumption that the scientific method starts with the hypothesis. They think: first hypothesize, then observe, then make a theory from the collection of hypotheses.

The reality is quite the opposite. The first point on the scientific method is the observation. Any hypotheses before observation will only diminish the pool of possible observations. Second is building a theory. Along the process, many things are observed without using a hypothesis to observe them; only needing a question and a desire to explore for the answer. Last is using the theory to create hypotheses; which are then used to confirm, alter, or reject the theory.

The exploration phase is where interesting new observations come up for the first time, and everyone who nears the end of this phase wants to talk to people about it; because it's really cool! Lots of amazing stuff is observed before any real theory can be made. Lots of plausible, valid, and important ideas come up which should indeed be discussed, as they are relevant to many subjects.

However, the people who take the wrong path, make the false assumption that the scientific method starts with the hypothesis, are afraid to take a step out of their reality. Taking in new information as an observation with no hypothesis or theory attached, which is what is available to them from the exploration stage, would be taking a step out of their reality.

the response you get of "why should I believe x? He doesn't have a PhD" is fundamentally the same as the response "prove it. Until you prove it, why should I listen to you?". These people don't want to observe or theorize. They don't want to explore. They don't want to work toward the goal I want to work toward. They want to stay in their world until they have a reason to involve something new in their world. The result of that isn't an improvement, either, because they still have the same problem of living in their own world.

This objection is a fallacy coming from false assumptions. The thing I have observed to be a/the source of this fallacy is the false start of the scientific method cycle.

FYI, "pseudo-scientists" have a related problem where they are either unable or unwilling to leave the exploration phase (often because the theory they are exploring is wrong, but sometimes because they're just looking in the wrong direction). Religions have another related problem where they are unable or unwilling to enter the exploration phase; which is not entirely unlike the problem I described about normal science, which is why many people call science a religion.

Comment author: Zackmarty 31 January 2012 09:32:29AM 1 point [-]

This isn't related to the entire post so much as it is a response to the problem with the scientific method. The scientific method does start with a hypothesis in relation to individual experiments. The hypothesis starts with general observations made from previous experiments or just some kind of general observation.

If we assume complete ignorance about NaCl (Sodium Chloride), but previously observed that pure sodium (Na) explodes in the presence of water, we might decide to devise an experiment to see what happens when we place other compounds with sodium in water. Our hypothesis might be something like, everything that has sodium explodes in water. It is not necessary to write down this hypothesis, because the hypotheses we generally make are made in our head when we are not in some kind of laboratory setting. By that I mean we constantly make hypotheses about subjects before we observe the results of our 'experiments.'

Anyway, we toss a block of sodium chloride in a bucket of water and observe that an explosion does not immediately follow. We had previous observations from former experiments, or general observations, to suggest that stuff with sodium will explode, but our observational evidence suggests that our previously made hypothesis is not true.

We make plenty of observations before making hypotheses, but we always make some sort of hypothesis before making some sort of observation when starting an experiment of any manner.

Comment author: epigeios 31 January 2012 09:28:59PM *  1 point [-]

Most people don't even go as far as to make a hypothesis there. It's arguable that every time someone forms a question, it is the same as forming a hypothesis; purely because questions can be reworded as hypotheses. However, in the case of someone who is just exploring, they would not go so far as to hypothesize.

It's normal for the person to say something like "I wonder if it happens with all sodium compounds" or, "I wonder if there are any sodium compounds that explode as well". But in these cases, there is no basis and no reason to form a hypothesis. one could argue that the person is making a hypothesis like "all sodium compounds explode in water"; but the person doesn't care. The person could just as easily make the hypothesis "no sodium compounds explode in water". And there's no reason to make either of these, or any hypothesis at all, because no theory has been formed that can be tested.

And further, making a hypothesis like this limits the amount of new information that can come in from these experiments. The information is now limited to "whether or not the substance explodes", when there are plenty of other reactions that can happen. The person who makes this hypothesis is liable to miss small bubbles appearing. That is anti-desired when exploring, when trying to observe as much as possible so as to build a theory.

The point is that the person in your example is not doing a hypothesis experiment, the person is doing an exploration experiment. Unless a theory exists, there's no basis for choosing any hypothesis at all.

Yet, let's say then that the person discovered some cool stuff and started to build a theory. He wants to tell you about his in-progress theory. Obviously he hasn't done any hypothesis experiments, because hypotheses haven't mattered yet. He tells you about his observations, and his conjectures. Many people, in response to this, say "can you prove it? Why should I believe you?". To which he has no answer, because he has nothing to prove yet. All of his observations are just observations, and he has no solid theories. Because he has no theories, any temporary hypotheses he makes continuously jump around, and to an outside observer have no coherence or meaning. Any attempt at proving something will prove futile, and will be a waste of time, purely because there is nothing to prove.

The higher-level or higher-class version of this response is: "what are your credentials? Why should I believe you?".

In this way my comment does relate to the entire post. Often times, there is no true objection. Often times, the objection is merely that someone is mentally lazy and doesn't want to think or explore. Often times, the objection is that I haven't formed a complete theory yet, only a list of observations and conjectures, so there's nothing the person can believe in. The difference in opinion there is that I want to work with them and believe in nothing, and they want to work on their own and believe in something. It's not that they object to the theory or observations or conjectures, they just object to thinking about it.

Comment author: halcyon 07 June 2012 12:42:41PM 1 point [-]

i'd say: you don't have a phd, therfore you're not qualified to judge whether or not yudkowsky should have a phd.

Comment author: paper-machine 07 June 2012 12:45:08PM 0 points [-]

How does getting a Ph. D. (even in a related field) give one the qualification to judge?

Comment author: halcyon 07 June 2012 12:58:33PM 1 point [-]

it doesn't, it's a jocular reductio ad absurdum based on the irrationality of the underlying premise. 9_9 and what field are we talking here, education? phdology?

Comment author: paper-machine 07 June 2012 12:59:59PM 0 points [-]

If we're commenting on the same article, a Ph. D. in (in decreaing order of relevance) AI, Machine Learning, CS, or mathematics...

Comment author: halcyon 07 June 2012 01:06:05PM 2 points [-]

naw, i'm talking about what field qualifies you to judge how much not having a phd disqualifies you from judging statements on that subject.

Comment author: asparisi 20 July 2012 07:02:08AM *  18 points [-]

When I was in Sales, we called this "finding their true objection."

Basically, if someone says "Well, I don't want it unless it has X!" You say "What if I could provide you with X?"

So if someone says "Come back when you have a PhD!" You say "What if I could provide you with PhDs who believe the same idea?" If they then say "There are tons of PhDs who believe crazy things!" then you say "Then what else would I need to convince you?"

Usually, between them dismissing their own criteria and the amount of ideas they can bring forward, you can bring it down to about three things. I've seen 5, but that was a hard case. Those aren't hard and fast rules: the rule is make sure you get them ALL, and make it specific, something like:

"So, if I can get you a published book by a PhD, respected in a field relevant to X, AND I can provide you with a for-profit organization that is working to accomplish goals relevant to X, AND I can make a flower appear out of my ear (or whatever)" THEN you will admit you were wrong and change your view?

And if you're REALLY invested, you should have been taking notes, and get them to 'initial' (not sign, people hate signing but will often initial: it feels like a smaller pain) the list. Consistency bias is also your friend here: if they say it aloud, they will probably also initial it.

And then, if you hand it all to them on a silver platter, with the right presentation you can get a "you were right, I was wrong" out of them. (If you screw it up, you can get begrudging acceptance. Occasionally hostility if you really botch it. But that's life in the interpersonal world.)

It sounds like a lot, but oddly, it isn't usually very hard to get people to change their minds this way. It takes some time, so you'd better be invested in making that change. If you know what to expect, handing it all early helps. But if you really want it to happen, this way works.

Comment author: Kenny 23 July 2013 12:52:35AM 1 point [-]

It would be awesome to see a transcript of a back and forth between Eliezer and Robin, or any two people denating, where they both provided this info!

Comment author: Jiro 23 July 2013 03:20:18PM 3 points [-]

"Come back when you have a PhD" typically doesn't mean "if you have a PhD, I'll believe you", it means "if you have a PhD, I'll assign a high enough probability to you having something worthwhile to say that it's worth even listening to you and doing the work to figure out where you might be wrong".

Also, it can't be answered by "I could show you some PhDs who agree with me". As there are a lot of PhDs in the world, finding one out of that large number who agrees with you doesn't update the probability of being right by much compared to combining PhD and agreement in a specific person named in advance (such as yourself).

Furthermore, human beings are neither unsafe genies nor stupid AIs, and nobody will take kindly to you trying to act like one by giving someone something that matches their literal request but any human being with common sense can figure out isn't what they meant.

My request would probably be something like "come back when you have a PhD and get your observations published in a peer-reviewed journal".

Comment author: ialdabaoth 23 July 2013 01:22:10AM 1 point [-]

The one thing that actually has seemed to raise credibility, is famous people associating with the organization, like Peter Thiel funding us, or Ray Kurzweil on the Board.

Well, yeah.

You spend a lot of your time worrying about how to get an AI to operate within the interests of lesser beings. You also seem to spend a certain amount of time laying out Schelling fences around "dark side tactics". It seems to me that these are closely related processes.

As you have said, "people are crazy, the world is mad". We are not operating with a population of rational decision makers. Even the subset of the population that appears to be rational decision makers often aren't; signaling rationality convincingly to other non-rational agents is often easier than actually being rational.

It seems that often, the things that you label "dark side" tactics are the only tools available for rational beings to extract resources out of non-rational animals, even if those resources are to be used explicitly for the non-rational animals' benefit. (I'll leave it for one of the Reactionaries to present the argument that you shouldn't bother worrying about their benefit; I happen to share most of your ethical goals, so I doubt I could do the argument justice.)

Over the past five years, have you found that it's become easier to navigate the wider social arena under your own rules, or are your goals and assertions still running into the same credibility problems? (I'm hoping it's the former, as I could use a hopeful glimmer, but reality is what it is and I will deal with it regardless.)

Comment author: RPMcMurphy 06 October 2014 09:30:17PM 0 points [-]
Comment author: Jiro 06 October 2014 10:37:27PM *  1 point [-]

I am not very impressed by that.

"Would you change your mind if you were convinced of X" carries the connotation "if I managed to give you an argument for X, and you couldn't rebut it, would you change your mind?" The answer to that should be "no" for many values of X even if the answer to the original question is "yes". The fact that you couldn't rebut the argument may mean that it's true. It may also just mean the argument is full of holes but the person is really good at convincing you. How do you know that the person who convinced you of X isn't another case of Eliezer convincing you to let the AI out of a box?

If a lot of scientists or other experts vetted the claim of such an X and it was not only personally convincing, but had a substantial following in the community of experts, then I might change my mind.

Comment author: RPMcMurphy 12 December 2014 04:04:15PM 0 points [-]

A third standard doesn't rely on "experts" or on your tendency to believe a third-party's argument. It can rely on your own capacity for intellectual honesty and your own assessment of your own knowledge. ...Which is really the final arbiter anyway.

Human social systems, as per the Milgram study (which very few people know in adequate detail) outlined in "Obedience to Authority: An Experimental View," are more subject to corruption than intelligent, curious, empathic individuals are. (Such individuals are very rare, but they do exist. Hayek was one, Harry Browne was one, a few people I've met in my life are others..) If you accept the current political system as legitimate, you've got a lot of company, all of whom are "experts" in a totally corrupted linked set of research, policy, and career communities. Those with sufficient interdisciplinary knowledge can understand this, but they have to be curious enough to note that the available data doesn't match the claims of the experts.

Do we live in a liberal democracy? No, we no longer do. But that's what is claimed by the experts. In the 1800s, proper jury trials were eliminated, thus eliminating liberal democracy --as a liberal democracy, the thing, not the label-- itself. The labels have survived, but their meaning has been stripped. Instead of meaning "pro-market, pro-evolution, pro-freedom," the term "liberal" is now taken to mean "anti-market, pro-evolution, pro-coercion." The dishonesty of that transition is confusing to the conformist, and useful to the exploiter of confusion (those seeking office). Adversarial cybernetic systems can benefit from and exploit confusion, and all human political systems (in terms of conflicts over power) do so.

So, aren't you just substituting "a community of experts" for "a person really good at convincing you"? If so, then you're basically saying that you believe that the decision-making of stultified academics is better than a single high-IQ person, who may or may not hold values that nearly zero "groups of academics" hold. Academics once dismissed the idea of "heavier than air flight" or thought it would take a lot longer than it did. Luckily, it was at a time when the Wright brothers and Hiram Maxim ignored "the community of experts" (including their earlier selves!). I'm not suggesting that engineers wouldn't have come up with flight anyway, I'm just suggesting it wouldn't have come about as quickly in a closed society subject to the condescension and derision of "the gatekeepers of knowledge" AKA "a community of experts."

Also, groups can be just as subject, if not more subject, to perverse incentives than individuals can. Stanley Milgram revealed this fact of uneducated (uncivilized) social organization, as did Solomon Asch, and Philip Zimbardo. (Especially when all individuals within a group are making the same perverse decision, due to the bystander effect, because none of them know the internal mental states of the others, and all are kept in line by conformity, fear, shyness, etc.) Many comedians, including George Carlin and Bill Hicks, and many more economists (Hayek, etc.) have commented on this at length, but it takes an empathic type of personality to implicitly comprehend it.

In some domains, there are no experts, or all of the experts are totally and completely corrupted by perverse incentives that are obvious to minority viewpoints whose brains are structured differently making them sensitive to different patterns than the less thoughtful and detail-oriented majority. Monica Anderson, Temple Grandin, Ray Kurzweil (In "Age of Spiritual Machines"), and J Storrs Hall have pointed this out many times, but people don't generally comment on their views in this area, I suspect because they either cannot process the information (majority) or because they are convinced by either sociopathic arguments (preventing them from processing the information by accessing the portion of their brain that obeys authority from an inborn now-irrational desire to survive), or the minority "dissenting" arguments (and then become a part of that quieter dissenting minority).

Monica Anderson writes:

But why is there so much more activity in the Majority View corner? There are several reasons. One is that if you are a programmer, then this is where you start out. In order to build, say, ontology based systems you need to learn very little; you can immediately start programming. And the mere fact that you are a programmer makes it likely that you prefer Reductionist methods. In order to move to the other corner you’d have to study quite a lot of Philosophy and Epistemology, some Neuroscience, etc., which constitutes a barrier that keeps most people from shifting viewpoint.

People in the Minority View corner may have started as Biologists, Psychologists or even Philosophers. They are less inclined to start AI projects since they may not have sufficient background and interest in programming and computer science. Proponents of the Minority View can claim that their theories are more Biologically Plausible than the Majority view theories. If you build a system according to the Minority View, it will have more similarities with brains than Majority View systems.

When I discuss my minority view AI theory with people from the majority camp I have to explain everything down to the basics of holistic thinking and I never seem to get through to them; they are often very skeptical. My “unproven theory of AI” is about as consistent as any Reductionist/majority view “unproven theory of AI” so I rarely get any outright arguments against my ideas; a typical response is “show me”.

In contrast, when I discuss my theories with competent people from other fields, such as Biology, they will nod their heads vigorously and say, “Of course, all along I’ve been thinking it has to be something like this”. This is also true for many people without any science education, since these theories are quite congruent with naive ideas of how the mind works.

I'm in rough agreement with Peter Voss, Hugo de Garis, J.S. Hall, Jeff Hawkins, Kwabena Boahen, and Monica Anderson's view of artificial general intelligence. Massively parallel emergent systems will become intelligent, and can become intelligent by evolved competition, and filling of market and ecosystem niches. This will happen whether anyone likes it or not, and whether it's "Friendly" or not. Likely such brains will be both Friendly and Unfriendly, Weak and Strong. Obviously, strong minds won't be able to be controlled, as Peter Voss believes. Also, if there's any advantage to copying near-human intelligences, then they will be copied to gain the group advantage.

My "true rejection" is the lack of empathy from any brain: the willing cause of unnecessary and undeserved suffering. The suffering caused by the drug war is one example: it's totally sub-optimal in every way, unless one is a conformist and therefore rejects the truth "out of hand." Once one understands that coercion is the thing that causes all the unnecessary suffering, it's easy to be a libertarian, a classical liberal. When you see that religion provides the reliable access to political power exploited by sociopaths, you can also see the damage that faith causes enough to be an atheist, and not say "But what does it matter if people are religious?"

However, to gain direct knowledge of any of the prior things, you need to actually be curious enough with them to interact with them scientifically. Most people are not intellectually honest enough to do this. Some people are. Those who are should work together, because logically, everyone here has a "true rejection" of totalitarianism, and historically, totalitarianism is very likely, and reliably recurrent.

"Why Does the World Feel Wrong?" (...to empaths, the rest of you sociopaths, and enslaved chattel, need not inquire.)

When we reject sociopathic domination at the local level, we can be civilized, we can "ramp up," and we can have our singularity, and a benevolent SL4 society. Not before.

Comment author: Jiro 12 December 2014 04:46:16PM 0 points [-]

That seems to suggest you believe peer review is a bad idea. Is that true?

Comment author: RPMcMurphy 06 October 2014 10:06:41PM *  -3 points [-]

I'll believe you're a gold miner when I see your gold.

Numenta http://www.numenta.org is building smart software brains that simulate hardware brains, with the ultimate goal of building energy-inexpensive hardware brains. Vicarious is building smart hardware brains. Robobrain is building --and educating-- smart hardware brains. IBM is building smart hardware brains, with its Neuromorphic computing project. OpenCog is building a software system, which is very good, but of unknown (to me) competitiveness with the other projects. Honda is building smart hardware brains, and teaching them. Jürgen Schmidhuber and Andrew Ng are building non-human brain-like machines and educating them. Hugo de Garis was building smart hardware brains (and dropped out to be an educator, I guess).

In any event, building context-independent brainlike systems seems to be the way to go. In much the same way that decent parents don't allow one child to beat up and even murder their other kids, I imagine that such synthetic intelligences, given time, would disallow the stagnation caused by omnipresent and dominant sociopathy.

This is also why it's important to measure the success of organizations by their stated goals, and by their incremental results on the path to those goals. A more nimble, more adventurous organization would be manufacturing chipsets now, based on the latest science. Such an organization would give up some control in order to retain influence. You will never be in control of a superhuman synthetic intelligence.

But almost all parents have influence (not control) over their children. The smartest parents steer their children into self-building directions.

That is all the power that anyone at the human level has, in comparison to superhuman, self-directing intelligences.

Most or all of the prior brain projects listed, by the way, lack mirror neurons, and do what sociopaths or non-sociopath humans tell them to do. They will increasingly do what sociopaths tell them to do, because sociopaths print the money, and dominate government positions of top-down control. The original intention of LessWrong is valuable, but its standards may be too much of a top-down variety, and too little of a bottom-up variety.

You can't stop the sociopath from existing. He exists. And, he has intentionality, and will seek to corrupt your systems. But there are lots of Tim Murphys and George Washingtons out there. There are lots of Henry David Thoreaus, George Orwells, Lysander Spooners, and Doug Stanhopes out there, even if they are vastly outnumbered by neutral-by-default brains, and sociopaths. The two ends of the spectrum of intelligence compete for the allegiance of the neutral-by-default brains. There are so many human minds that they can't all be Round pegs beaten into square holes, no matter how much the education bureaucrats try.

Really, at a deep level, LessWrong should be an educational organization that seeks to build hardware brains and then educate them properly. This then solves previously intractable problems like rising totalitarianism, space travel, etc., if only out of boredom. Perhaps it even solves the problem of human stupidity, supermodifying us all.

Ray Kurzweil has the right idea, but there are too few Kurzweilian projects. Evolution got it right by making lots of human kids. If human lifespan was 1,000, there would have been more family planning, and fewer kids, and therefore, fewer geniuses of the caliber necessary to adapt to brutal and changing environments, including those of Manichean sociopathy. (Also, the Manichean devils themselves, like Stalin and Pol Pot, would have not died off.)

Human brains have proven that it's possible to build very smart PhD scientists who don't care if the world around them becomes a Stalinist hellhole, so long as the temperature of the water is raised incrementally, and they have some reason to think they will be shielded from its boiling. So, the goal of LessWrong is to build classical liberal robots, who oppose all taxation, but especially property taxation (so that large numbers of humans aren't rendered homeless and displaced).

Smart doesn't necessarily = "libertarian." However, "smartest, given reality" does. This is why virtually everyone thinks it's a good thing that slavery ended. Once a system has decided correctly, opposition drops to nearly zero.

Right now, the libertarian and single-issue resisters are growing in number, because it makes no sense to allow powerful sociopaths to siphon away the value of everyone who trades in "Federal Reserve Notes." It makes no sense to allow them to educate our kids, or subject our kids to a dumbed-down society/world of their creation.

Comment author: SanguineEmpiricist 15 October 2014 12:20:44AM 0 points [-]

As the years go on. I'm glad to review this and appreciate that you understand this. You definitely have a group of people that love you more than like you, and it is somewhat disheartening to see these people so vehemently they insist on their rejection without even giving some modicum of chance.

I only get more motivation to put in my extraordinary effort and see in what ways I can help.