Filter Last three months

Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

[moderator action] Eugine_Nier is now banned for mass downvote harassment

95 Kaj_Sotala 03 July 2014 12:04PM

As previously discussed, on June 6th I received a message from jackk, a Trike Admin. He reported that the user Jiro had asked Trike to carry out an investigation to the retributive downvoting that Jiro had been subjected to. The investigation revealed that the user Eugine_Nier had downvoted over half of Jiro's comments, amounting to hundreds of downvotes.

I asked the community's guidance on dealing with the issue, and while the matter was being discussed, I also reviewed previous discussions about mass downvoting and looked for other people who mentioned being the victims of it. I asked Jack to compile reports on several other users who mentioned having been mass-downvoted, and it turned out that Eugine was also overwhelmingly the biggest downvoter of users David_Gerard, daenarys, falenas108, ialdabaoth, shminux, and Tenoke. As this discussion was going on, it turned out that user Ander had also been targeted by Eugine.

I sent two messages to Eugine, requesting an explanation. I received a response today. Eugine admitted his guilt, expressing the opinion that LW's karma system was failing to carry out its purpose of keeping out weak material and that he was engaged in a "weeding" of users who he did not think displayed sufficient rationality.

Needless to say, it is not the place of individual users to unilaterally decide that someone else should be "weeded" out of the community. The Less Wrong content deletion policy contains this clause:

Harrassment of individual users.

If we determine that you're e.g. following a particular user around and leaving insulting comments to them, we reserve the right to delete those comments. (This has happened extremely rarely.)

Although the wording does not explicitly mention downvoting, harassment by downvoting is still harassment. Several users have indicated that they have experienced considerable emotional anguish from the harassment, and have in some cases been discouraged from using Less Wrong at all. This is not a desirable state of affairs, to say the least.

I was originally given my moderator powers on a rather ad-hoc basis, with someone awarding mod privileges to the ten users with the highest karma at the time. The original purpose for that appointment was just to delete spam. Nonetheless, since retributive downvoting has been a clear problem for the community, I asked the community for guidance on dealing with the issue. The rough consensus of the responses seemed to authorize me to deal with the problem as I deemed appropriate.

The fact that Eugine remained quiet about his guilt until directly confronted with the evidence, despite several public discussions of the issue, is indicative of him realizing that he was breaking prevailing social norms. Eugine's actions have worsened the atmosphere of this site, and that atmosphere will remain troubled for as long as he is allowed to remain here.

Therefore, I now announce that Eugine_Nier is permanently banned from posting on LessWrong. This decision is final and will not be changed in response to possible follow-up objections.

Unfortunately, it looks like while a ban prevents posting, it does not actually block a user from casting votes. I have asked jackk to look into the matter and find a way to actually stop the downvoting. Jack indicated earlier on that it would be technically straightforward to apply a negative karma modifier to Eugine's account, and wiping out Eugine's karma balance would prevent him from casting future downvotes. Whatever the easiest solution is, it will be applied as soon as possible.

EDIT 24 July 2014: Banned users are now prohibited from voting.

[Sequence announcement] Introduction to Mechanism Design

54 badger 30 April 2014 04:21PM

Mechanism design is the theory of how to construct institutions for strategic agents, spanning applications like voting systems, school admissions, regulation of monopolists, and auction design. Think of it as the engineering side of game theory, building algorithms for strategic agents. While it doesn't have much to say about rationality directly, mechanism design provides tools and results for anyone interested in world optimization.

In this sequence, I'll touch on

  • The basic mechanism design framework, including the revelation principle and incentive compatibility.
  • The Gibbard-Satterthwaite impossibility theorem for strategyproof implementation (a close analogue of Arrow's Theorem), and restricted domains like single-peaked or quasilinear preference where we do have positive results.
  • The power and limitations of Vickrey-Clarke-Groves mechanisms for efficiently allocating goods, generalizing Vickrey's second-price auction.
  • Characterizations of incentive-compatible mechanisms and the revenue equivalence theorem.
  • Profit-maximizing auctions.
  • The Myerson-Satterthwaite impossibility for bilateral trade.
  • Two-sided matching markets à la Gale and Shapley, school choice, and kidney exchange.

As the list above suggests, this sequence is going to be semi-technical, but my foremost goal is to convey the intuition behind these results. Since mechanism design builds on game theory, take a look at Yvain's Game Theory Intro if you want to brush up.

Various resources:

I plan on following up on this sequence with another focusing on group rationality and information aggregation, surveying scoring rules and prediction markets among other topics.

Suggestions and comments are very welcome.

Moving on from Cognito Mentoring

50 VipulNaik 16 May 2014 10:42PM

Back in December 2013, Jonah Sinick and I launched Cognito Mentoring, an advising service for intellectually curious students. Our goal was to improve the quality of learning, productivity, and life choices of the student population at large, and we chose to focus on intellectually curious students because of their greater potential as well as our greater ability to relate with that population. We began by offering free personalized advising. Jonah announced the launch in a LessWrong post, hoping to attract the attention of LessWrong's intellectually curious readership.

Since then, we feel we've done a fair amount, with a lot of help from LessWrong. We've published a few dozen blog posts and have an information wiki. Slightly under a hundred people contacted us asking us for advice (many from LessWrong), and we had substantive interactions with over 50 of them. As our reviews from students and parents suggest, we've made a good impression and have had a positive impact on many of the people we've advised. We're proud of what we've accomplished and grateful for the support and constructive criticism we've received on LessWrong.

However, what we've learned in the last few months has led us to the conclusion that Cognito Mentoring is not ripe for being a full-time work opportunity for the two of us.

For the last few months, we've eschewed regular jobs and instead done contract work that provides us the flexibility to work on Cognito Mentoring, eating into our savings somewhat to cover the cost of living differences. This is a temporary arrangement and is not sustainable. We therefore intend to scale back our work on Cognito Mentoring to "maintenance mode" so that people can continue to benefit from the resources we've already collected, with minimal additional effort on our part, freeing us up to take regular jobs with more demanding time requirements.

We might revive Cognito Mentoring as a part-time or full-time endeavor in the future if there are significant changes to our beliefs about the traction, impact, and long-run financial viability of Cognito Mentoring. Part of the purpose of "maintenance mode" will be to leave open the possibility of such a revival if the idea does indeed have potential.

In this post, I discuss some of the factors that led us to change our view, the conditions under which we might revive Cognito Mentoring, and more details about how "maintenance mode" for Cognito Mentoring will look.

Reason #1: Downward update on social value

We do think that the work we've done on Cognito Mentoring so far has generated social value, and the continued presence of the website will add more value over time. However, our view has shifted in the direction of lower marginal social value from working on Cognito Mentoring full-time, relative to simply keeping the website live and doing occasional work to improve it. Specifically:

  • It's quite possible that the lowest-hanging fruit with respect to the advisees who would be most receptive to our advice has already been plucked. We received the bulk of our advisees through LessWrong within the month after our initial posting. Other places where we've posted about our service have led to fewer advisees (more here).
  • Of our website content, only a small fraction of the content gets significant traction (see our list of popular pages), so honing and promoting our best content might be a better strategy for improving social value than trying to create a comprehensive resource. This can be done while in maintenance mode, and does not require full-time effort on our part.

What might lead us to change our minds: If we continue to be contacted by large numbers of potentially high-impact people, or we get evidence that the advising we've already done has had significantly greater impact than we think it did, we'll update our social value upward.

Reason #2: Downward update on long-run financial viability

We have enough cash to go on for a few more months. But for Cognito Mentoring to be something that we work full time on, we need an eventual steady source of income from it. Around mid-March 2014, we came to the realization that charging advisees is not a viable revenue source, as Jonah described at the end of his post about how Cognito Mentoring can do the most good (see also this comment by Luke Muehlhauser and Jonah's response to it below the comment). At that point, we decided to focus more on our informational content and on looking for philanthropic funding.

Our effort at looking into philanthropic funding did give us a few leads, and some of them could plausibly result in us getting small grants. However, none of the leads we got pointed to potential steady long-term income sources. In other words, we don't think philanthropic funding is a viable long-term revenue model for Cognito Mentoring.

Our (anticipated) difficulty in getting philanthropic funding arises from two somewhat different reasons.

  1. What we're doing is somewhat new and does not fit the standard mold of educational grants. Educational foundations tend to give grants for fairly specific activities, and what we're doing does not seem to fit those.
  2. We haven't demonstrated significant traction or impact yet (even though we've had a reasonable amount of per capita impact, the total number of people we've influenced so far is relatively small). This circles back to Reason #1: funders' reluctance to fund us may in part stem from their belief that we won't have much social value, given our lack of traction so far. Insofar as funders' judgment carries some information value, this should also strengthen Reason #1.

What might lead us to change our minds: If we are contacted by a funder who is willing to bankroll us for over a year and also offer a convincing reason for why he/she thinks bankrolling us is a good idea (so that we're convinced that our funding can be sustained beyond a year) we'll change our minds.

Reason #3: Acquisition of knowledge and skills

One of the reasons we've been able to have an impact through Cognito Mentoring so far is that both Jonah and I have knowledge of many diverse topics related to the questions that our advisees have posed to us. But our knowledge is still woefully inadequate in a number of areas. In particular, many advisees have asked us questions in the realms of technology, entrepreneurship, and the job environment, and while we have pointed them to resources on these, firsthand experience, or close secondhand experience, would help us more effectively guide advisees. We intend to take jobs related to computer technology (in fields such as programming or data science), and these jobs might be at startups or put us in close contact with startups. This will better position us to return to mentoring later if we choose to resume it part-time or full-time.

Knowledge and skills we acquire working in the technology sector could also help us design better interfaces or websites that can more directly address the needs of our audience. So far, we've thought of ourselves as content-oriented people, so we've used standard off-the-shelf software such as WordPress (for our main website and blog) and MediaWiki (for our information wiki). Part of the reason is that we wanted to focus on content creation rather than interface design, but part of the reason we've stuck to these is that we didn't think we could design interfaces. Once we've acquired more programming and design experience, we might be more open to the idea of designing interfaces and software that can meet particular needs of our target audience.We might design an interface that helps people study more effectively, make better life decisions, or share reviews of courses and colleges, in a manner similar to softwares or websites such as Anki or Beeminder or Goodreads. There might also be potential for a more effective online resource that teaches programming than those in existence (e.g. Codecademy). It's not clear right now whether there exists a useful opportunity of this sort that we are particularly well-suited to, but with more coding experience, we'll at least be able to implement an idea of this sort if we decide it has promise.

Reason #4: Letting it brew in the background can give us a better idea of the potential

If we continue to gradually add content to the wiki, and continue to get links and traffic to it from other sources, it's likely that the traffic will grow slowly and steadily. The extent of organic growth will help us figure out how much promise Cognito Mentoring has. If our wiki gets to the point of steadily receiving thousands of pageviews a day, we will reconsider reviving Cognito Mentoring as a part-time or full-time endeavor. If, on the other hand, traffic remains at approximately the current level (about a hundred pageviews a day, once we exclude spikes arising from links from LessWrong and Marginal Revolution) then the idea is probably not worth revisiting, and we'll leave it in maintenance mode.

In addition, by maintaining contact with the people we've advised, we can get more insight into the sort of impact we've had, whether it is significant over the long term, and how it can be improved. This again can tell us whether our impact is sufficiently large as to make Cognito Mentoring worth reviving.

What "maintenance mode" entails

  1. We'll continue to have contact information available, but will scale back on personalized advising: People are welcome to contact us with questions and suggestions about content, but we will not generally offer detailed personalized responses or do research specific to individuals who contact us. We'll attempt to point people to relevant content we've already written, or to other resources we're already aware of that can address their concerns.
  2. The information wiki will remain live, and we will continue to make occasional improvements, but we won't have a time schedule of when particular improvements have to be implemented by.
  3. Existing blog posts will remain, but we probably won't be making many new blog posts. New blog posts will happen only if one of us has an idea that really seems worth sharing and for which the Cognito Mentoring blog is an ideal forum.
  4. We'll continue our administrative roles in the communities of existing Cognito Mentoring advisees
  5. We'll continue periodically reviewing the progress of people we've advised so far: This will help us get a better sense of how valuable our work has been, and can be useful should we choose to revive Cognito Mentoring.
  6. We'll continue to correspond with advisees we have so far (time permitting), though we'll give more priority to advisees who continue to maintain contact of their own accord and those whose activities seem to have higher impact potential.
  7. We'll try to get our best content linked from other sources, such as about.com: Sources like about.com are targeted at the general population. We can try to get linked to from there as an additional resource for the more intellectually curious population that's outside the core focus of about.com.
  8. We'll link more extensively to other sources that people can use: For instance, we can more emphatically point to 80,000 Hours for people who are interested in career advising in relation to effective altruist pursuits. We can point to about.com and College Confidential for more general information about mainstream institutions. We already make a number of recommendations on our website, but as we stop working actively, it becomes all the more important that people who come to us are appropriately redirected to other sources that can help them.

Conclusion and summary (TL;DR)

We (qua Cognito Mentoring) are grateful to LessWrong for being welcoming of our posts, offering constructive criticism, and providing us with some advisees we've enjoyed working with. We think that the work we've done has value, but don't think that there's enough marginal value from full-time work on Cognito Mentoring. We think we can do more good for ourselves and the world by switching Cognito Mentoring to maintenance mode and freeing our time currently spent on Cognito Mentoring for other pursuits. The material that we have already produced will continue to remain in the public domain and we hope that people will benefit from it. We may revisit our "maintenance mode" decision if new evidence changes our view regarding traction, impact, and long-run financial viability.

False Friends and Tone Policing

43 palladias 18 June 2014 06:20PM

TL;DR: It can be helpful to reframe arguments about tone, trigger warnings, and political correctness as concerns about false cognates/false friends.  You may be saying something that sounds innocuous to you, but translates to something much stronger/more vicious to your audience.  Cultivating a debating demeanor that invites requests for tone concerns can give you more information about about the best way to avoid distractions and have a productive dispute.

 

When I went on a two-week exchange trip to China, it was clear the cultural briefing was informed by whatever mistakes or misunderstandings had occurred on previous trips, recorded and relayed to us so that we wouldn't think, for example, that our host siblings were hitting on us if they took our hands while we were walking.

But the most memorable warning had to do with Mandarin filler words.  While English speakers cover gaps with "uh" "um" "ah" and so forth, the equivalent filler words in Mandarin had an African-American student on a previous trip pulling aside our tour leader and saying he felt a little uncomfortable since his host family appeared to be peppering all of their comments with "nigga, nigga, nigga..."

As a result, we all got warned ahead of time.  The filler word (那个 - nèige) was a false cognate that, although innocuous to the speaker, sounded quite off-putting to us.  It helped to be warned, but it still required some deliberate, cognitive effort to remind myself that I wasn't actually hearing something awful and to rephrase it in my head.

When I've wound up in arguments about tone, trigger warnings, and taboo words, I'm often reminded of that experience in China.  Limiting language can prompt suspicion of closing off conversations, but in a number of cases, when my friends have asked me to rephrase, it's because the word or image I was using was as distracting (however well meant) as 那个 was in Beijing.

It's possible to continue a conversation with someone who's every statement is laced with "nigga" but it takes effort.  And no one is obligated to expend their energy on having a conversation with me if I'm making it painful or difficult for them, even if it's as the result of a false cognate (or, as the French would say, false friend) that sounds innocuous to me but awful to my interlocutor.  If I want to have a debate at all, I need to stop doing the verbal equivalent of assaulting my friend to make any progress.

It can be worth it to pause and reconsider your language even if the offensiveness of a word or idea is exactly the subject of your dispute.  When I hosted a debate on "R: Fire Eich" one of the early speakers made it clear that, in his opinion, opposing gay marriage was logically equivalent to endorsing gay genocide (he invoked a slippery slope argument back to the dark days of criminal indifference to AIDS).

Pretty much no one in the room (whatever their stance on gay marriage) agreed with this equivalence, but we could all agree it was pretty lucky that this person had spoken early in the debate, so that we understood how he was hearing our speeches.  If every time someone said "conscience objection," this speaker was appending "to enable genocide," the fervor and horror with which he questioned us made a lot more sense, and didn't feel like personal viciousness.  Knowing how high the stakes felt to him made it easier to have a useful conversation.

This is a large part of why I objected to PZ Myers's deliberate obtuseness during the brouhaha he sparked when he asked readers to steal him a consecrated Host from a Catholic church so that he could desecrate it.  PZ ridiculed Catholics for getting upset that he was going to "hurt" a piece of bread, even though the Eucharist is a fairly obvious example of a false cognate that is heard/received differently by Catholics and atheists.  (After all, if it wasn't holy to someone, he wouldn't be able to profane it).  In PZ's incident, it was although we had informed our Chinese hosts about the 那个/nigga confusion, and they had started using it more boisterously, so that it would be clearer to us that they didn't find it offensive.

We were only able to defuse the awkwardness in China for two reasons.

  1. The host family was so nice, aside from this one provocation, that the student noticed he was confused and sought advice.
  2. There was someone on hand who understood both groups well enough to serve as an interpreter.

In an ordinary argument (especially one that takes place online) it's up to you to be visibly virtuous enough that, if you happen to be using a vicious false cognate, your interlocutor will find that odd, not of a piece with your other behavior.

That's one reason my debating friend did bother explaining explicitly the connection he saw between opposition to gay marriage and passive support of genocide -- he trusted us enough to think that we wouldn't endorse the implications of our arguments if he made them obvious.  In the P.Z. dispute, when Catholic readers found him as the result of the stunt, they didn't have any such trust.

It's nice to work to cultivate that trust, and to be the kind of person your friends do approach with requests for trigger warnings and tone shifts.  For one thing, I don't want to use emotionally intense false cognates and not know it, any more than I would want to be gesticulating hard enough to strike my friend in the face without noticing.  For the most part, I prefer to excise the distraction, so it's easier for both of us to focus on the heart of the dispute, but, even if you think that the controversial term is essential to your point, it's helpful to know it causes your friend pain, so you have the opportunity to salve it some other way.  

 

P.S. Arnold Kling's The Three Languages of Politics is a short read and a nice introduction to what political language you're using that sounds like horrible false cognates to people rooted in different ideologies.

P.P.S. I've cross-posted this on my usual blog, but am trying out cross-posting to Discussion sometimes.

[Meta] The Decline of Discussion: Now With Charts!

42 Gavin 04 June 2014 10:02PM

[Based on Alexandros's excellent dataset.]

I haven't done any statistical analysis, but looking at the charts I'm not sure it's necessary. The discussion section of LessWrong has been steadily declining in participation. My fairly messy spreadsheet is available if you want to check the data or do additional analysis.

Enough talk, you're here for the pretty pictures.

The number of posts has been steadily declining since 2011, though the trend over the last year is less clear. Note that I have excluded all posts with 0 or negative Karma from the dataset.

 

The total Karma given out each month has similarly been in decline.

Is it possible that there have been fewer posts, but of a higher quality?

No, at least under initial analysis the average Karma seems fairly steady. My prior here is that we're just seeing less visitors overall, which leads to fewer votes being distributed among fewer posts for the same average value. I would have expected the average karma to drop more than it did--to me that means that participation has dropped more steeply than mere visitation. Looking at the point values of the top posts would be helpful here, but I haven't done that analysis yet.

These are very disturbing to me, as someone who has found LessWrong both useful and enjoyable over the past few years. It raises several questions:

 

  1. What should the purpose of this site be? Is it supposed to be building a movement or filtering down the best knowledge?
  2. How can we encourage more participation?
  3. What are the costs of various means of encouraging participation--more arguing, more mindkilling, more repetition, more off-topic threads, etc?

 

Here are a few strategies that come to mind:

Idea A: Accept that LessWrong has fulfilled its purpose and should be left to fade away, or allowed to serve as a meetup coordinator and repository of the highest quality articles. My suspicion is that without strong new content and an online community, the strength of the individual meetup communities may wane as fewer new people join them. This is less of an issue for established communities like Berkeley and New York, but more marginal ones may disappear.

Idea B: Allow and encourage submission of rationalism, artificial intelligence, transhumanism etc related articles from elsewhere, possibly as a separate category. This is how a site like Hacker News stays high engagement, even though many of the discussions are endless loops of the same discussion. It can be annoying for the old-timers, but new generations may need to discover things for themselves. Sometimes "put it all in one big FAQ" isn't the most efficient method of teaching.

Idea C: Allow and encourage posts on "political" topics in Discussion (but probably NOT Main). The dangers here might be mitigated by a ban on discussion of current politicians, governments, and issues. "Historians need to have had a decade to mull it over before you're allowed to introduce it as evidence" could be a good heuristic. Another option would be a ban on specific topics that cause the worst mindkilling. Obviously this is overall a dangerous road.

Idea D: Get rid of Open Threads and create a new norm that a discussion post as short as a couple sentences is acceptable. Open threads get stagnant within a day or two, and are harder to navigate than the discussion page. Moving discussion from the Open Threads to the Discussion section would increase participation if users could be convinced thatit was okay to post questions and partly-formed ideas there.

The challenge with any of these ideas is that they will require strong moderation. 

At any rate, this data is enough to convince me that some sort of change is going to be needed in order to put the community on a growth trajectory. That is not necessarily the goal, but at its core LessWrong seems like it has the potential to be a powerful tool for the spreading of rational thought. We just need to figure out how to get it started into its next evolution.

Downvote stalkers: Driving members away from the LessWrong community?

39 Ander 02 July 2014 12:40AM

Last month I saw this post: http://lesswrong.com/lw/kbc/meta_the_decline_of_discussion_now_with_charts/ addressing whether the discussion on LessWrong was in decline.  As a relatively new user who had only just started to post comments, my reaction was: “I hope that LessWrong isn’t in decline, because the sequences are amazing, and I really like this community.  I should try to write a couple articles myself and post them!  Maybe I could do an analysis/summary of certain sequences posts, and discuss how they had helped me to change my mind”.   I started working on writing an article.

Then I logged into LessWrong and saw that my Karma value was roughly half of what it had been the day before.   Previously I hadn’t really cared much about Karma, aside from whatever micro-utilons of happiness it provided to see that the number slowly grew because people generally liked my comments.   Or at least, I thought I didn’t really care, until my lizard brain reflexes reacted to what it perceived as an assault on my person.

 

Had I posted something terrible and unpopular that had been massively downvoted during the several days since my previous login?  No, in fact my ‘past 30 days’ Karma was still positive.  Rather, it appeared that everything I had ever posted to LessWrong now had a -1 on it instead of a 0. Of course, my loss probably pales in comparison to that of other, more prolific posters who I have seen report this behavior.

So what controversial subject must I have commented on in order to trigger this assault?  Well, let’s see, in the past week  I had asked if anyone had any opinions of good software engineer interview questions I could ask a candidate.  I posted in http://lesswrong.com/lw/kex/happiness_and_children/ that I was happy to not have children, and finally, here in what appears to me to be by far the most promising candidate:http://lesswrong.com/r/discussion/lw/keu/separating_the_roles_of_theory_and_direct/  I replied to a comment about global warming data, stating that I routinely saw headlines about data supporting global warming. 

 

Here is our scenario: A new user is attempting to participate on a message board that values empiricism and rationality, posted that evidence supports that climate change is real.  (Wow, really rocking the boat here!)    Then, apparently in an effort to ‘win’ this discussion by silencing opposition, someone went and downvoted every comment this user had ever made on the site.   Apparently they would like to see LessWrong be a bastion of empiricism and rationality and [i]climate change denial[/i] instead? And the way to achieve this is not to have a fair and rational discussion of the existing empirical data, but rather to simply Karmassassinate anyone who would oppose them?

 

Here is my hypothesis: The continuing problem of karma downvote stalkers is contributing to the decline of discussion on the site.    I definitely feel much less motivated to try and contribute anything now, and I have been told by multiple other people at LessWrong meetings things such as “I used to post a lot on LessWrong, but then I posted X, and got mass downvoted, so now I only comment on Yvain’s blog”.  These anecdotes are, of course, only very weak evidence to support my claim.  I wish I could provide more, but I will have to defer to any readers who can supply more.

 

Perhaps this post will simply trigger more retribution, or maybe it will trigger an outswelling of support, or perhaps just be dismissed by people saying I should’ve posted it to the weekly discussion thread instead.   Whatever the outcome, rather than meekly leaving LessWrong and letting my 'stalker' win, I decided to open a discussion about the issue.  Thank you!

Against utility functions

38 Qiaochu_Yuan 19 June 2014 05:56AM

I think we should stop talking about utility functions.

In the context of ethics for humans, anyway. In practice I find utility functions to be, at best, an occasionally useful metaphor for discussions about ethics but, at worst, an idea that some people start taking too seriously and which actively makes them worse at reasoning about ethics. To the extent that we care about causing people to become better at reasoning about ethics, it seems like we ought to be able to do better than this.

The funny part is that the failure mode I worry the most about is already an entrenched part of the Sequences: it's fake utility functions. The soft failure is people who think they know what their utility function is and say bizarre things about what this implies that they, or perhaps all people, ought to do. The hard failure is people who think they know what their utility function is and then do bizarre things. I hope the hard failure is not very common. 

It seems worth reflecting on the fact that the point of the foundational LW material discussing utility functions was to make people better at reasoning about AI behavior and not about human behavior. 

Mathematics as a lossy compression algorithm gone wild

35 shminux 06 June 2014 11:53PM

This is yet another half-baked post from my old draft collection, but feel free to Crocker away.

 

There is an old adage from Eugene Wigner known as the "Unreasonable Effectiveness of Mathematics". Wikipedia:

the mathematical structure of a physical theory often points the way to further advances in that theory and even to empirical predictions.

The way I interpret is that it is possible to find an algorithm to compress a set of data points in a way that is also good at predicting other data points, not yet observed. In yet other words, a good approximation is, for some reason, sometimes also a good extrapolation. The rest of this post elaborates on this anti-Platonic point of view.

Now, this point of view is not exactly how most people see math. They imagine it as some near-magical thing that transcends science and reality and, when discovered, learned and used properly, gives one limited powers of clairvoyance. While only the select few wizard have the power to discover new spells (they are known as scientists), the rank and file can still use some of the incantations to make otherwise impossible things to happen (they are known as engineers). 

This metaphysical view is colorfully expressed by Stephen Hawking:

What is it that breathes fire into the equations and makes a universe for them to describe? The usual approach of science of constructing a mathematical model cannot answer the questions of why there should be a universe for the model to describe. Why does the universe go to all the bother of existing?

Should one interpret this as if he presumes here that math, in the form of "the equations" comes first and only then there is a physical universe for math to describe, for some values of "first" and "then", anyway? Platonism seems to reach roughly the same conclusions:

Wikipedia defines platonism as

the philosophy that affirms the existence of abstract objects, which are asserted to "exist" in a "third realm distinct both from the sensible external world and from the internal world of consciousness, and is the opposite of nominalism

In other words, math would have "existed" even if there were no humans around to discover it. In this sense, it is "real", as opposed to "imagined by humans". Wikipedia on mathematical realism:

mathematical entities exist independently of the human mind. Thus humans do not invent mathematics, but rather discover it, and any other intelligent beings in the universe would presumably do the same. In this point of view, there is really one sort of mathematics that can be discovered: triangles, for example, are real entities, not the creations of the human mind.

Of course, the debate on whether mathematics is "invented" or "discovered" is very old. Eliezer-2008 chimes in in http://lesswrong.com/lw/mq/beautiful_math/:

To say that human beings "invented numbers" - or invented the structure implicit in numbers - seems like claiming that Neil Armstrong hand-crafted the Moon.  The universe existed before there were any sentient beings to observe it, which implies that physics preceded physicists. 

and later:

The amazing thing is that math is a game without a designer, and yet it is eminently playable.

In the above, I assume that what Eliezer means by physics is not the science of physics (a human endeavor), but the laws according to which our universe came into existence and evolved. These laws are not the universe itself (which would make the statement "physics preceded physicists" simply "the universe preceded physicists", a vacuous tautology), but some separate laws governing it, out there to be discovered. If only we knew them all, we could create a copy of the universe from scratch, if not "for real", then at least as a faithful model. This universe-making recipe is then what physics (the laws, not science) is.

And these laws apparently require mathematics to be properly expressed, so mathematics must "exist" in order for the laws of physics to exist.

Is this the only way to think of math? I don't think so. Let us suppose that the physical universe is the only "real" thing, none of those Platonic abstract objects. Let is further suppose that this universe is (somewhat) predictable. Now, what does it mean for the universe to be predictable to begin with? Predictable by whom or by what? Here is one approach to predictability, based on agency: a small part of the universe (you, the agent) can construct/contain a model of some larger part of the universe (say, the earth-sun system, including you) and optimize its own actions (to, say, wake up the next morning just as the sun rises). 

Does waking up on time count as doing math? Certainly not by the conventional definition of math. Do migratory birds do math when they migrate thousands of miles twice a year, successfully predicting that there would be food sources and warm weather once they get to their destination? Certainly not by the conventional definition of math. Now, suppose a ship captain lays a course to follow the birds, using maps and tables and calculations? Does this count as doing math? Why, certainly the captain would say so, even if the math in question is relatively simple. Sometimes the inputs both the birds and the humans are using are the same: sun and star positions at various times of the day and night, the magnetic field direction, the shape of the terrain.

What is the difference between what the birds are doing and what humans are doing? Certainly both make predictions about the universe and act on them. Only birds do this instinctively and humans consciously, by "applying math". But this is a statement about the differences in cognition, not about some Platonic mathematical objects. One can even say that birds perform the relevant math instinctively. But this is a rather slippery slope. By this definition amoebas solve the diffusion equation when they move along the sugar gradient toward a food source. While this view has merits, the mathematicians analyzing certain aspects of the Navier-Stokes equation might not take kindly being compared to a protozoa. 

So, like JPEG is a lossy image compression algorithm of the part of the universe which creates an image on our retina when we look at a picture, the collection of the Newton's laws is a lossy compression algorithm which describes how a thrown rock falls to the ground, or how planets go around the Sun. in both cases we, a tiny part of the universe, are able to model and predict a much larger part, albeit with some loss of accuracy.

What would it mean then for a Universe to not "run on math"? In this approach it means that in such a universe no subsystem can contain a model, no matter how coarse, of a larger system. In other words, such a universe is completely unpredictable from the inside. Such a universe cannot contain agents, intelligence or even the simplest life forms. 

Now, to the "gone wild" part of the title. This is where the traditional applied math, like counting sheep, or calculating how many cannons you can arm a ship with before it sinks, or how to predict/cause/exploit the stock market fluctuations, becomes "pure math", or math for math's sake, be it proving the Pythagorean theorem or solving a Millennium Prize problem. At this point the mathematician is no longer interested in modeling a larger part of the universe (except insofar as she predicts that it would be a fun thing to do for her, which is probably not very mathematical).

Now, there is at least one serious objection to this "math is jpg" epistemology. It goes as follows: "in any universe, no matter how convoluted, 1+1=2, so clearly mathematics transcends the specific structure of a single universe". I am skeptical of this logic, since to me 1,+,= and 2 are semi-intuitive models running in our minds, which evolved to model the universe we live in. I can certainly imagine a universe where none of these concepts would be useful in predicting anything, and so they would never evolve in the "mind" of whatever entity inhabits it. To me mathematical concepts are no more universal than moral concepts: sometimes they crystallize into useful models, and sometimes they do not. Like the human concept of honor would not be useful to spiders, the concept of numbers (which probably is useful to spiders) would not be useful in a universe where size is not a well-defined concept (like something based on a Conformal Field Theory).

So the "Unreasonable Effectiveness of Mathematics" is not at all unreasonable: it reflects the predictability of our universe. Nothing "breathes fire into the equations and makes a universe for them to describe", the equations are but one way a small part of the universe predicts the salient features of a larger part of it. Rather, an interesting question is what features of a predictable universe enable agents to appear in it, and how complex and powerful can these agents get.

 

[LINK] Claustrum Stimulation Temporarily Turns Off Consciousness in an otherwise Awake Patient

34 shminux 04 July 2014 08:00PM

This paper, or more often the New Scientist's exposition of it is being discussed online and is rather topical here. In a nutshell, stimulating one small but central area of the brain reversibly rendered one epilepsia patient unconscious without disrupting wakefulness. Impressively, this phenomenon has apparently been hypothesized before, just never tested (because it's hard and usually unethical). A quote from the New Scientist article (emphasis mine):

One electrode was positioned next to the claustrum, an area that had never been stimulated before.

When the team zapped the area with high frequency electrical impulses, the woman lost consciousness. She stopped reading and stared blankly into space, she didn't respond to auditory or visual commands and her breathing slowed. As soon as the stimulation stopped, she immediately regained consciousness with no memory of the event. The same thing happened every time the area was stimulated during two days of experiments (Epilepsy and Behavior, doi.org/tgn).

To confirm that they were affecting the woman's consciousness rather than just her ability to speak or move, the team asked her to repeat the word "house" or snap her fingers before the stimulation began. If the stimulation was disrupting a brain region responsible for movement or language she would have stopped moving or talking almost immediately. Instead, she gradually spoke more quietly or moved less and less until she drifted into unconsciousness. Since there was no sign of epileptic brain activity during or after the stimulation, the team is sure that it wasn't a side effect of a seizure.

If confirmed, this hints at several interesting points. For example, a complex enough brain is not sufficient for consciousness, a sort-of command and control structure is required, as well, even if relatively small. A low-consciousness state of late-stage dementia sufferers might be due to the damage specifically to the claustrum area, not just the overall brain deterioration. The researchers speculates that stimulating the area in vegetative-state patients might help "push them out of this state". From an AI research perspective, understanding the difference between wakefulness and consciousness might be interesting, too.

 

This is why we can't have social science

33 Costanza 13 July 2014 09:04PM

Jason Mitchell is [edit: has been] the John L. Loeb Associate Professor of the Social Sciences at Harvard. He has won the National Academy of Science's Troland Award as well as the Association for Psychological Science's Janet Taylor Spence Award for Transformative Early Career Contribution.

Here, he argues against the principle of replicability of experiments in science. Apparently, it's disrespectful, and presumptively wrong.

Recent hand-wringing over failed replications in social psychology is largely pointless, because unsuccessful experiments have no meaningful scientific value.

Because experiments can be undermined by a vast number of practical mistakes, the likeliest explanation for any failed replication will always be that the replicator bungled something along the way. Unless direct replications are conducted by flawless experimenters, nothing interesting can be learned from them.

Three standard rejoinders to this critique are considered and rejected. Despite claims to the contrary, failed replications do not provide meaningful information if they closely follow original methodology; they do not necessarily identify effects that may be too small or flimsy to be worth studying; and they cannot contribute to a cumulative understanding of scientific phenomena.

Replication efforts appear to reflect strong prior expectations that published findings are not reliable, and as such, do not constitute scientific output.

The field of social psychology can be improved, but not by the publication of negative findings. Experimenters should be encouraged to restrict their “degrees of freedom,” for example, by specifying designs in advance.

Whether they mean to or not, authors and editors of failed replications are publicly impugning the scientific integrity of their colleagues. Targets of failed replications are justifiably upset, particularly given the inadequate basis for replicators’ extraordinary claims.

This is why we can't have social science. Not because the subject is not amenable to the scientific method -- it obviously is. People are conducting controlled experiments and other people are attempting to replicate the results. So far, so good. Rather, the problem is that at least one celebrated authority in the field hates that, and would prefer much, much more deference to authority.

What resources have increasing marginal utility?

33 Qiaochu_Yuan 14 June 2014 03:43AM

Most resources you might think to amass have decreasing marginal utility: for example, a marginal extra $1,000 means much more to you if you have $0 than if you have $100,000. That means you can safely apply the 80-20 rule to most resources: you only need to get some of the resource to get most of the benefits of having it.

At the most recent CFAR workshop, Val dedicated a class to arguing that one resource in particular has increasing marginal utility, namely attention. Initially, efforts to free up your attention have little effect: the difference between juggling 10 things and 9 things is pretty small. But once you've freed up most of your attention, the effect is larger: the difference between juggling 2 things and 1 thing is huge. Val also argued that because of this funny property of attention, most people likely undervalue the value of freeing up attention by orders of magnitude.

During a conversation later in the workshop I suggested another resource that might have increasing marginal utility, namely trust. A society where people abide by contracts 80% of the time is not 80% as good as a society where people abide by contracts 100% of the time; most of the societal value of trust (e.g. decreasing transaction costs) doesn't seem to manifest until people are pretty close to 100% trustworthy. The analogous way to undervalue trust is to argue that e.g. cheating on your spouse is not so bad, because only one person gets hurt. But cheating on spouses in general undermines the trust that spouses should have in each other, and the cumulative impact of even 1% of spouses cheating on the institution of marriage as a whole could be quite negative. (Lots of things about the world make more sense from this perspective: for example, it seems like one of the main practical benefits of religion is that it fosters trust.) 

What other resources have increasing marginal utility? How undervalued are they? 

Flowers for Algernon

32 Anatoly_Vorobey 18 June 2014 09:16AM

Daniel Keyes, the author of the short story Flowers for Algernon, and a novel of the same title that is its expanded version, died three days ago.

Keyes wrote many other books in the last half-century, but none achieved nearly as much prominence as the original short story (published in 1959) or the novel (came out in 1966). 

It's probable that many or even most regulars here at Less Wrong read Flowers for Algernon: it's a very famous SF story, it's about enhanced intelligence, and it's been a middle/high school literature class staple in the US. But most != all, and past experience showed me that assumptions of cultural affinity are very frequently wrong. So in case you haven't read the story, I'd like to invite you explicitly to do so. It's rather short, and available at this link:

Flowers for Algernon

(I was surprised to find out that the original story is not available on Amazon. The expanded novelization is. If you wonder which version is better to read, I have no advice to offer)

(I will edit this post in a week or so to remove the link to the story and this remark)

 

Change Contexts to Improve Arguments

30 palladias 08 July 2014 03:51PM

On a recent trip to Ireland, I gave a talk on tactics for having better arguments (video here).  There's plenty in the video that's been discussed on LW before (Ideological Turing Tests and other reframes), but I thought I'd highlight one other class of trick I use to have more fruitful disagreements.

It's hard, in the middle of a fight, to remember, recognize, and defuse common biases, rhetorical tricks, emotional triggers, etc.  I'd rather cheat than solve a hard problem, so I put a lot of effort into shifting disagreements into environments where it's easier for me and my opposite-number to reason and argue well, instead of relying on willpower.  Here's a recent example of the kind of shift I like to make:

A couple months ago, a group of my friends were fighting about the Brendan Eich resignation on facebook. The posts were showing up fast; everyone was, presumably, on the edge of their seats, fueled by adrenaline, and alone at their various computers. It’s a hard place to have a charitable, thoughtful debate.

I asked my friends (since they were mostly DC based) if they’d be amenable to pausing the conversation and picking it up in person.  I wanted to make the conversation happen in person, not in front of an audience, and in a format that let people speak for longer and ask questions more easily. If so, I promised to bake cookies for the ultimate donnybrook.  

My friends probably figured that I offered cookies as a bribe to get everyone to change venues, and they were partially right. But my cookies had another strategic purpose. When everyone arrived, I was still in the process of taking the cookies out of the oven, so I had to recruit everyone to help me out.

“Alice, can you pour milk for people?”

“Bob, could you pass out napkins?”

“Eve, can you greet people at the door while I’m stuck in the kitchen with potholders on?”

Before we could start arguing, people on both sides of the debate were working on taking care of each other and asking each others’ help. Then, once the logistics were set, we all broke bread (sorta) with each other and had a shared, pleasurable experience. Then we laid into each other.

Sharing a communal experience of mutual service didn’t make anyone pull their intellectual punches, but I think it made us more patient with each other and less anxiously fixated on defending ourselves. Sharing food and seating helped remind us of the relationships we enjoyed with each other, and why we cared about probing the ideas of this particular group of people.

I prefer to fight with people I respect, who I expect will fight in good faith.  It's hard to remember that's what I'm doing if I argue with them in the same forums (comment threads, fb, etc) that I usually see bad fights.  An environment shift and other compensatory gestures makes it easier to leave habituated errors and fears at the door.

 

Crossposted/adapted from my blog.

Look for the Next Tech Gold Rush?

29 Wei_Dai 19 July 2014 10:08AM

In early 2000, I registered my personal domain name weidai.com, along with a couple others, because I was worried that the small (sole-proprietor) ISP I was using would go out of business one day and break all the links on the web to the articles and software that I had published on my "home page" under its domain. Several years ago I started getting offers, asking me to sell the domain, and now they're coming in almost every day. A couple of days ago I saw the first six figure offer ($100,000).

In early 2009, someone named Satoshi Nakamoto emailed me personally with an announcement that he had published version 0.1 of Bitcoin. I didn't pay much attention at the time (I was more interested in Less Wrong than Cypherpunks at that point), but then in early 2011 I saw a LW article about Bitcoin, which prompted me to start mining it. I wrote at the time, "thanks to the discussion you started, I bought a Radeon 5870 and started mining myself, since it looks likely that I can at least break even on the cost of the card." That approximately $200 investment (plus maybe another $100 in electricity) is also worth around six figures today.

Clearly, technological advances can sometimes create gold rush-like situations (i.e., first-come-first-serve opportunities to make truly extraordinary returns with minimal effort or qualifications). And it's possible to stumble into them without even trying. Which makes me think, maybe we should be trying? I mean, if only I had been looking for possible gold rushes, I could have registered a hundred domain names optimized for potential future value, rather than the few that I happened to personally need. Or I could have started mining Bitcoins a couple of years earlier and be a thousand times richer.

I wish I was already an experienced gold rush spotter, so I could explain how best to do it, but as indicated above, I participated in the ones that I did more or less by luck. Perhaps the first step is just to keep one's eyes open, and to keep in mind that tech-related gold rushes do happen from time to time and they are not impossibly difficult to find. What other ideas do people have? Are there other past examples of tech gold rushes besides the two that I mentioned? What might be some promising fields to look for them in the future?

[meta] Policy for dealing with users suspected/guilty of mass-downvote harassment?

28 Kaj_Sotala 06 June 2014 05:46AM

Below is a message I just got from jackk. Some specifics have been redacted 1) so that we can discuss general policy rather than the details of this specific case 2) because presumption of innocence, just in case there happens to be an innocuous explanation to this.

Hi Kaj_Sotala,

I'm Jack, one of the Trike devs. I'm messaging you because you're the moderator who commented most recently. A while back the user [REDACTED 1] asked if Trike could look into retributive downvoting against his account. I've done that, and it looks like [REDACTED 2] has downvoted at least [over half of REDACTED 1's comments, amounting to hundreds of downvotes] ([REDACTED 1]'s next-largest downvoter is [REDACTED 3] at -15).

What action to take is a community problem, not a technical one, so we'd rather leave that up to the moderators. Some options:

1. Ask [REDACTED 2] for the story behind these votes
2. Use the "admin" account (which exists for sending scripted messages, &c.) to apply an upvote to each downvoted post
3. Apply a karma award to [REDACTED 1]'s account. This would fix the karma damage but not the sorting of individual comments
4. Apply a negative karma award to [REDACTED 2]'s account. This makes him pay for false downvotes twice over. This isn't possible in the current code, but it's an easy fix
5. Ban [REDACTED 2]

For future reference, it's very easy for Trike to look at who downvoted someone's account, so if you get questions about downvoting in the future I can run the same report.

If you need to verify my identity before you take action, let me know and we'll work something out.

-- Jack

So... thoughts? I have mod powers, but when I was granted them I was basically just told to use them to fight spam; there was never any discussion of any other policy, and I don't feel like I have the authority to decide on the suitable course of action without consulting the rest of the community.

Skills and Antiskills

26 katydee 26 April 2014 06:54AM

One useful little concept that a friend and I have is that of the antiskill. Like a normal skill, an antiskill gives you both the ability and the affordance to do things that you wouldn't otherwise be able to do. The difference between a skill and an antiskill is that a skill gives you the ability and affordance to do things that are positive on net, while an antiskill gives you the ability and affordance to do things that are negative on net.

For instance, my friend believes that dancing is often an antiskill, because it gives you an affordance to dance rather than have interesting conversations while at parties, and he considers having interesting conversations to be much more valuable than dancing-- therefore, knowing how to dance serves primarily to enable choices that are bad on net.

I disagree with the specific point in this case, but I nevertheless think it's a good example because it illustrates another key principle of skills and antiskills-- whether something is a skill or an antiskill is context-dependent. If dancing will largely prevent you from having interesting conversations, it may well be an antiskill-- but if you go to a lot of nightclubs where loud music makes conversation difficult, knowing how to dance seems very useful indeed!

Another example is the skill of knowing how to fix computers. In many respects this is very useful, and can indeed lead to a profitable career in IT. But-- as I'm sure many of you may have experienced-- having your friends and family know that you know how to fix computers can be very negative on net!

Overall, I find the skill/antiskill framework quite useful when it comes to navigating what sorts of skills, abilities, and knowledge I should acquire. Before choosing my next priority, I often pause to think:

  • What affordances will learning this give me?
  • In what contexts will those affordances be most relevant?
  • Will this be positive or negative on net?

Using this framework has enabled me to discern strengths and weaknesses that I had previously not considered, and in some cases those strengths and weaknesses have proven decisive to my planning.

Arguments and relevance claims

25 Kaj_Sotala 05 May 2014 04:49PM
The following once happened: I posted a link to some article on an IRC channel. A friend of mine read the article in question and brought up several criticisms. I felt that her criticisms were mostly correct though not very serious, so I indicated agreement with them.

Later on the same link was posted again. My friend commented something along the lines of "that was already posted before, we discussed this with Kaj and we found that the article was complete rubbish". I was surprised - I had thought that I had only agreed to some minor criticisms that didn't affect the main point of the article. But my friend had clearly thought that the criticisms were decisive and had made the article impossible to salvage.

--

Every argument actually has two parts, even if people often only state the first part. There's the argument itself, and an implied claim of why the argument would matter if it were true. Call this implied part the relevance claim.

Suppose that I say "Martians are green". Someone else says, "I have seen a blue Martian", and means "I have seen a blue Martian (argument), therefore your claim of all Martians being green is false (relevance claim)". But I might interpret this as them saying, "I have seen a blue Martian (argument), therefore your claim of most Martians being green is less likely (relevance claim)". I then indicate agreement. Now I will be left with the impression that the other person made a true-but-not-very-powerful claim that left my argument mostly intact, whereas the other person is left with the impression that they made a very powerful claim that I agreed with, and therefore I admitted that I was wrong.

We could also say that the relevance claim is a claim of how much the probability of the original statement would be affected if the argument in question were true. So, for example "I have seen a blue martian (argument), therefore the probability of 'Martians are green' is less than .01 (relevance claim)", or equivalently, "I have seen a blue martian" & "P(martians are green|I have seen a blue martian) < .01".

If someone says something that I feel is entirely irrelevant to the whole topic, inferential silence may follow.

Therefore, if someone makes an argument that I agree with, but I suspect that we might disagree about its relevance, I now try to explicitly comment on what my view of the relevance is. Example.

Notice that people who are treating arguments as soldiers are more likely to do this automatically, without needing to explicitly remind themselves of it. In fact, for every argument that their opponent makes that they're forced to concede, they're likely to immediately say "but that doesn't matter because X!". Because we like to think that we're not treating arguments as soldiers, we also try to avoid automatically objecting "but that doesn't matter because X" whenever our favored position gets weakened. This is a good thing, but it also means that we're probably less likely than average to comment about an argument's relevance even in cases where we should comment on it.

(Cross-posted from my blog.)

The Correct Use of Analogy

24 SilentCal 16 July 2014 09:07PM

In response to: Failure by AnalogySurface Analogies and Deep Causes

Analogy gets a bad rap around here, and not without reason. The kinds of argument from analogy condemned in the above links fully deserve the condemnation they get. Still, I think it's too easy to read them and walk away thinking "Boo analogy!" when not all uses of analogy are bad. The human brain seems to have hardware support for thinking in analogies, and I don't think this capability is a waste of resources, even in our highly non-ancestral environment. So, assuming that the linked posts do a sufficient job detailing the abuse and misuse of analogy, I'm going to go over some legitimate uses.

 

The first thing analogy is really good for is description. Take the plum pudding atomic model. I still remember this falsified proposal of negative 'raisins' in positive 'dough' largely because of the analogy, and I don't think anyone ever attempted to use it to argue for the existence of tiny subnuclear particles corresponding to cinnamon. 

But this is only a modest example of what analogy can do. The following is an example that I think starts to show the true power: my comment on Robin Hanson's 'Don't Be "Rationalist"'. To summarize, Robin argued that since you can't be rationalist about everything you should budget your rationality and only be rational about the most important things; I replied that maybe rationality is like weightlifting, where your strength is finite yet it increases with use. That comment is probably the most successful thing I've ever written on the rationalist internet in terms of the attention it received, including direct praise from Eliezer and a shoutout in a Scott Alexander (yvain) post, and it's pretty much just an analogy.

Here's another example, this time from Eliezer. As part of the AI-Foom debate, he tells the story of Fermi's nuclear experiments, and in particular his precise knowledge of when a pile would go supercritical.

What do the above analogies accomplish? They provide counterexamples to universal claims. In my case, Robin's inference that rationality should be spent sparingly proceeded from the stated premise that no one is perfectly rational about anything, and weightlifting was a counterexample to the implicit claim 'a finite capacity should always be directed solely towards important goals'. If you look above my comment, anon had already said that the conclusion hadn't been proven, but without the counterexample this claim had much less impact.

In Eliezer's case, "you can never predict an unprecedented unbounded growth" is the kind of claim that sounds really convincing. "You haven't actually proved that" is a weak-sounding retort; "Fermi did it" immediately wins the point. 

The final thing analogies do really well is crystallize patterns. For an example of this, let's turn to... Failure by Analogy. Yep, the anti-analogy posts are themselves written almost entirely via analogy! Alchemists who glaze lead with lemons and would-be aviators who put beaks on their machines are invoked to crystallize the pattern of 'reasoning by similarity'. The post then makes the case that neural-net worshippers are reasoning by similarity in just the same way, making the same fundamental error.

It's this capacity that makes analogies so dangerous. Crystallizing a pattern can be so mentally satisfying that you don't stop to question whether the pattern applies. The antidote to this is the question, "Why do you believe X is like Y?" Assessing the answer and judging deep similarities from superficial ones may not always be easy, but just by asking you'll catch the cases where there is no justification at all.

Separating the roles of theory and direct empirical evidence in belief formation: the examples of minimum wage and anthropogenic global warming

23 VipulNaik 25 June 2014 09:47PM

I recently asked two questions on Quora with similar question structures, and the similarities and differences between the responses were interesting.

Question #1: Anthropogenic global warming, the greenhouse effect, and the historical weather record

I asked the question here. Question statement:

If you believe in Anthropogenic Global Warming (AGW), to what extent is your belief informed by the theory of the greenhouse effect, and to what extent is it informed by the historical temperature record?

In response to some comments, I added the following question details:

Due to length limitations, the main question is a bit simplistically framed. But what I'm really asking for is the relative importance of theoretical mechanisms and direct empirical evidence. Theoretical mechanisms are of course also empirically validated, but the empirical validation could occur in different settings.

For instance, the greenhouse effect is a mechanism, and one may get estimates of the strength of the greenhouse effect based on an understanding of the underlying physics or by doing laboratory experiments or simulations.

Direct empirical evidence is evidence that is as close to the situation we are trying to predict as possible. In this case, it would involve looking at the historical records of temperature and carbon dioxide concentrations, and perhaps some other confounding variables whose role needs to be controlled for (such as solar activity).

Saying that your belief is largely grounded in direct empirical evidence is basically saying that just looking at the time series of temperature, carbon dioxide concentrations and the other variables can allow one to say with fairly high confidence (starting from very weak priors) that increased carbon dioxide concentrations, due to human activity, are responsible for temperature increases. In other words, if you ran a regression and tried to do the usual tricks to infer causality, carbon dioxide would come out as the culprit.

Saying that your belief is largely grounded in theory is basically saying that the science of the greenhouse effect is sufficiently convincing that the historical temperature and weather record isn't an important factor in influencing your belief: if it had come out differently, you'd probably just have thought the data was noisy or wrong and wouldn't update away from believing in the AGW thesis.

I also posted to Facebook here asking my friends about the pushback to my use of the term "belief" in my question.

Question #2: Effect of increase in the minimum wage on unemployment

I asked the question here. Question statement:

If you believe that raising the minimum wage is likely to increase unemployment, to what extent is your belief informed by the theory of supply and demand and to what extent is it informed by direct empirical evidence?

I added the following question details:

By "direct empirical evidence" I am referring to empirical evidence that  directly pertains to the relation between minimum wage raises and  employment level changes, not empirical evidence that supports the  theory of supply and demand in general (because transferring that to the  minimum wage context would require one to believe the transferability  of the theory).

Also, when I say "believe that raising the minimum wage is likely to increase unemployment" I am talking about minimum wage increases of the sort often considered in legislative measures, and by "likely" I just mean that it's something that should always be seriously considered whenever a proposal to raise the minimum wage is made. The belief would be consistent with believing that in some cases minimum wage raises have no employment effects.

I also posted the question to Facebook here.

Similarities between the questions

The questions are structurally similar, and belong to a general question type of considerable interest to the LessWrong audience. The common features to the questions:

  • In both cases, there is a theory (the greenhouse effect for Question #1, and supply and demand for Question #2) that is foundational to the domain and is supported through a wide range of lines of evidence.
  • In both cases, the quantitative specifics of the extent to which the theory applies in the particular context are not clear. There are prima facie plausible arguments that other factors may cancel out the effect and there are arguments for many different effect sizes.
  • In both cases, people who study the broad subject (climate scientists for Question #1, economists for Question #2) are more favorably disposed to the belief than people who do not study the broad subject.
  • In both cases, a significant part of the strength of belief of subject matter experts seems to be their belief in the theory. The data, while consistent with the theory, does not seem to paint a strong picture in isolation. For the minimum wage, consider the Card and Krueger study. Bryan Caplan discusses how Bayesian reasoning with strong theoretical priors can lead one to continue believing that minimum wage increases cause unemployment to rise, without addressing Card and Krueger at the object level. For the case of anthropogenic global warming, consider the draft by Kesten C. Green (addressing whether a warming-based forecast has higher forecast accuracy than a no-change forecast) or the paper AGW doesn't cointegrate by Beenstock, Reingewertz, and Paldor (addressing whether, looking at the data alone, we can get good evidence that carbon dioxide concentration increases are linked with temperature increases).
  • In both cases, outsiders to the domain, who nonetheless have expertise in other areas that one might expect gives them insight into the question, are often more skeptical of the belief. A number of weather forecasters, physicists, and forecasting experts are skeptical of long-range climate forecasting or confident assertions about anthropogenic global warming. A number of sociologists, lawyers, and politicians often are disparaging of the belief that minimum wage increases cause unemployment levels to rise. The criticism is similar: namely, that a basically correct theory is being overstretched or incorrectly applied to a situation that is too complex, is similar.
  • In both cases, the debate is somewhat politically charged, largely because one's beliefs here affect one's views of proposed legislation (climate change mitigation legislation and minimum wage increase legislation). The anthropogenic global warming belief is more commonly associated with environmentalists, social democrats, and progressives, and (in the United States) with Democrats, whereas opposition to it is more common among conservatives and libertarians. The minimum wage belief is more commonly associated with free market views and (in the United States) with conservatives and Republicans, and opposition to it is more common among progressives and social democrats.

Looking for help

I'm interested in thoughts from the people here on these questions:

  • Thoughts on the specifics of Question #1 and Question #2.
  • Other possible questions in the same reference class (where a belief arises from a mix of theory and data, and the theory plays a fairly big role in driving the belief, while the data on its own is very ambiguous).
  • Other similarities between Question #1 and Question #2.
  • Ways that Question #1 and Question #2 are disanalogous.
  • General thoughts on how this relates to Bayesian reasoning and other modes of belief formation based on a combination of theory and data.

 

Tapestries of Gold

23 Shane_Patt 27 April 2014 07:23AM

(Nothing here is actually new, but a short explanation with pictures would have been helpful to me a while ago, so I thought I'd make an attempt.)

Let me start with a patch of territory: a set of things that exist. The number of rows is far from clear, but I'll use six candidates as a sample; and of course the diagram ought to be a tree, with many elements on each row converging to fewer on the row above, but you'll have to imagine that part.



The blue line that runs through the column is not causation, but identity. It took me a long time (and many knocks about the head from smarter people) to realize that this line is directionless. If someone labels the top Meaningful and the bottom Meaningless, or the top Important and the bottom Unimportant, we see this at once for an error; but the same labels are still errors, if applied in the opposite order. If someone labels the top Contingent and the bottom Necessary, this is another error; if the top Subjective and the bottom Objective, another; or if the top Less Real and the bottom More Real, another still. (Some errors of this type have been called "reductionism," but they aren't the thing people mean when they say "reductionism" around here.) Whatever is, is real—and equally so, wherever it appears along the blue line.

At one time I would have labelled the top Emergent and the bottom Fundamental, but David Deutsch convinced me that even this was a mistake. Suppose we grant that a mind of arbitrary power, given only the bottom row, could deduce all the others—a popular hypothesis for which we have some good evidence (though not too much). Even then: could not this same mind, given only the complete row for Physiology, deduce the contents of Chemistry no less readily? The blue line has no direction; if I forget this I forget what identity means, and cast myself into confusion—the same type of confusion afflicting one who says, "Science believes that morality's not real!" Better, then, to unlabel the blue line entirely, and when someone wants to know what ontological difference exists between the higher rows and the lower, say "Mu." (Until I realized this I did not understand the metaethics sequence—but that isn't the topic of this post.)

Where does that leave reductionism? Right where it was, untouched. As finite entities, we never perceive the blue line as a whole—not a single azure band of the infinite expanse in which we live. We have to divide the line into graspable segments, and therefore must explain how each segment connects to the others; we must spin the green threads of explanation, drawing a map to overlay the territory. (The diagram is simplified here as well; a green thread is not as simple as "compounds explain synapse," but an intricate dance of analysis and synthesis.) In dividing the line we introduce relation between its divisions, and in introducing relation we introduce direction; emergence is a feature of maps, not of territories. (I would not say "The mind emerges from the brain," but "The active brain is the mind, and models of the mind emerge from models of the brain.") Reductionism proper is just this: noticing that green arrows are always present, and always point up. The whole is never more, nor less, than the sum of its parts; it only seems that way, if some parts have escaped our notice.



Now let's add a few more columns; again, we'll simplify the structure so we can see it, leaving out all worlds but one.



The violet lines are causation—how things are; together with the blue of what things are, they form indigo Reality: the World That Is. (Maybe the violet lines too have no direction; this sounds like timeless physics, of which I don't feel I can wisely speak.) In any case, they extend far beyond our reach, as each effect in turn becomes a cause; once again we find ourselves dividing the lines into portions we can grasp, and to restore the continuity we once removed, we spin threads of red. These too are explanations—though of another kind than the green. Just as violet lines connect blue lines to form a complete territory, red threads connect strands of green threads to form a complete map.



There are traps to fall into here, too. If we believed that the only violet lines (or the only red threads) that counted as real or meaningful were the ones on the bottom row, we would commit another error; this error also has been called "reductionism"—small wonder that it's sometimes deployed as a term of abuse! Or—because we are fallible and the true multiply-branching structure is hard to perceive—we might draw a red arrow pointing left, and be guilty of mere illogic.

But if we can braid red and green together, our best strength is here—in threads of gold. Only a golden thread is knowledge made whole, and no golden thread is ever spun but of green and red in harmony; until I know what a thing is and how it comes to be, both, I do not understand that thing.



If you're wondering about the empty space beneath, remember that the number of rows in the true structure is far above six; I suspect it is infinite. The number of columns is far higher than shown as well, so a golden thread connects, not a single point to a single point, but a wide expanse of one row to a wide expanse of the row above it. Golden threads are far-reaching theories and models—spun of many smaller explanations.

Often, those who find their cloth too threadbare for their taste will turn to another source of material: the beige threads of supernaturalism. Beige looks a bit like gold, if not examined too closely, and these threads have one great advantage: to spin them is the easiest thing in the world. Since they're unanchored to reality, you're free to craft them in any length or shape you like, lay them with arrows pointing wherever suits you, and even cover threads of red or green or gold whose lustre seems displeasing. Some have been taught to weave with beige alone, and in years of toil wrought patterns of strange and desolate beauty; but every hour of labour made their work, not more akin to fact, but less.



In spinning green and red, and in braiding them as gold, we become scientists; in cutting loose the snarls of beige, we become naturalists; in weaving our many threads into sturdy cloth, we become rationalists. Then we join our separate cloths as one, and in such tapestries—if all goes well—we glimpse truth: the harmony of indigo and gold.

A Parable of Elites and Takeoffs

22 gwern 30 June 2014 11:04PM

Let me tell you a parable of the future. Let’s say, 70 years from now, in a large Western country we’ll call Nacirema.

One day far from now: scientific development has continued apace, and a large government project (with, unsurprisingly, a lot of military funding) has taken the scattered pieces of cutting-edge research and put them together into a single awesome technology, which could revolutionize (or at least, vastly improve) all sectors of the economy. Leading thinkers had long forecast that this area of science’s mysteries would eventually yield to progress, despite theoretical confusion and perhaps-disappointing initial results and the scorn of more conservative types and the incomprehension (or outright disgust, for ‘playing god’) of the general population, and at last - it had! The future was bright.

Unfortunately, it was hurriedly decided to use an early prototype outside the lab in an impoverished foreign country. Whether out of arrogance, bureaucratic inertia, overconfidence on the part of the involved researchers, condescending racism, the need to justify the billions of grant-dollars that cumulative went into the project over the years by showing some use of it - whatever, the reasons no longer mattered after the final order was signed. The technology was used, but the consequences turned out to be horrific: over a brief period of what seemed like mere days, entire cities collapsed and scores - hundreds - of thousands of people died. (Modern economies are extremely interdependent and fragile, and small disruptions can have large consequences; more people died in the chaos of the evacuation of the areas around Fukushima than will die of the radiation.)

continue reading »

[LINK] Scott Aaronson on Integrated Information Theory

22 DanielFilan 22 May 2014 08:40AM

Scott Aaronson, complexity theory researcher, disputes Tononi's theory of consciousness, that a physical system is conscious if and only if it has a high value of "integrated information". Quote:

So, this is the post that I promised to Max [Tegmark] and all the others, about why I don’t believe IIT.  And yes, it will contain that quantitative calculation [of the integrated information of a system that he claims is not conscious].

...

But let me end on a positive note.  In my opinion, the fact that Integrated Information Theory is wrong—demonstrably wrong, for reasons that go to its core—puts it in something like the top 2% of all mathematical theories of consciousness ever proposed.  Almost all competing theories of consciousness, it seems to me, have been so vague, fluffy, and malleable that they can only aspire to wrongness.

http://www.scottaaronson.com/blog/?p=1799

A Story of Kings and Spies

21 Joshua_Blaine 11 June 2014 11:54PM

There exists an old Kingdom with a peculiar, but no altogether uncommon, trait. It is overwhelmingly defensible given adequate forewarning. Its fields are surrounded by rivers on 3 sides and an impassable mountain to the South. The series of bridges commonly used by merchants and farmers to pass over the river can be completely removed by an impressive feat of engineering, unrivaled by any other kingdom, involving elaborate systems of levers and pulleys and large crews of men. This retracting, given the co-operation of all able men, can be done in the time of a single day across the entire length of river. The water is also deep, chilled, and very fast moving all throughout, making crossing without the bridges all but impossible. Fortifications on the inner banks of the river exist for archers and catapults to lease barrages against any foe that dare approach their land. It is this challenge that the enemies of the kingdom try to find a way to overcome.

It is acknowledged by both the King and his enemies that a surprise attack, one with so little warning that the bridges remain in place, would be successful against what is otherwise a poorly defensible region. Even a force of only moderate size could slaughter anyone within the rivers with ease. With this in mind, the King and his cabinet have a large espionage network that's infiltrated every major kingdom's decision making process. Their spies should, and have many times in the past, notified the King long before any attack, and allow for defenses to be raised, and victory to be assured. The King is very happy with his spies. They've never once failed to bring advance notice of any attack, and his network of informants have proven themselves resilient against counter-infiltration. He is, however, a very paranoid king, and wishes there was some way to be even more certain of his kingdom's safety. He is, as he sits upon his throne, ruminating on some such plans when a man of small stature is brought before him by some guards. The little man is wearing mostly simple clothes, but with some vibrant accents in the trimming.

"Why have you brought this fellow before me?" Asked the King of his guards.

"He claims to have word of an attack on the kingdom, sire." A guard said.

"He seems believable enough, sire, that we thought it best to bring him before you instead of merely dismissing him. You have had more training in detecting the truth of matters." The second guard said.

"Very well." He said, gesturing for the guards to relax. "Sir, may I have your name?" The King spoke directly to his small guest.

"Orin Eldirh, my king." He barely manages to say as he stammers on, "I've been told by a f-friend... a very close friend in-indeed... a t-t-trustworthy sort of fellow, you know... the kind who'd n-never lie, you see... And he says, and he's the employee of a very well off member of the Northern Kingdom's leadership, s-so I trust this information is accurate... He says th-that his boss was part of a meeting to plan a surprise attack on our kingdom. And very soon, I might add. He said the meeting was a pre-planned sort of thing, was going to be on a random day, so our spies wouldn't have time to figure things out, and that they'd have an army ready in less than a day! A day, sire! They're surely marching here now, as I speak."

The king quietly held the man's gaze for several moments before speaking. "And, holding what you've said is what you've heard, why would a noble betray his kingdom by speaking of such a secret meeting?"

"Is that important, sire? We have such little time to prepare for the invasion." Orin says. "Surely what I've said is enough to warrant removing the bridges, whatever his motivations." He paused uncertainly as he looked upon his king. "Isn't it?"

The King heaved a sigh before responding, "No. It really isn't." The King began to elaborate, "You see, removing those bridges cost more than you may realize. It takes every able man in the kingdom to work as fast as you claim we need to. That's an entire day's worth of labor used up. With the bridges up, that's maybe a weeks worth of trade and messages that wont be coming or going, seeing as the men wont work themselves so hard for two days in a row to put things back. What you personally lose may well be small, but it will make our kingdom and its stores suffer."

"But what are those costs to the lives of those people, those women, those children, lost to an attack?" Orin admonished.

"There is more at work here than you think, Orin." The King firmly answered, "You do not know how much thought I have put into the defense of my people." Orin's outrage slowly began diminishing as he took a sheepish stature. "Imagine, if you will, that I heed the word of every beggar and peasant who claimed some terrible force was underway. It's a much more common experience than you seem to think. Not a week goes by without someone offering their wisdom of an attack that my spies have somehow missed hearing of. The people of the kingdom would spend more time cowering in fear of an impending attack than doing anything else if I listened to every such piece of obvious paranoia or subterfuge. My people would tire of removing the bridges. Traders would tire of so frequent delays in their travels. It would spell our eventual doom, I'm sure of it." The King took a deep breathe and frowned, calmly continuing, "And yet, how could I ever forgive myself if I left us undefended from a legitimate attack? My spies are not perfect. Such a random meeting as you described may elude them, if we were unlucky and it was well implemented." The King pinched the bridge of his nose and closed his eyes before continuing, "I have to determine, to the best of my abilities, whether or not this threat is legitimate. So I'll ask you again, as I must know, why might this noble betray his kingdom?"

Orin swallowed and said, "If what my friend says is to be believed, and I consider it so, then this noble is not motivated by loyalty for his kingdom. I was told that he was not born into his position, but bought it himself. He has quite a fortune from his ownership of many kinds of businesses and guilds. War hurts him more than it helps the businesses of his kingdom, I've been told he believes. He'd wish to avoid starting any kind of fight, I'd think, if this were true."

"I know of a man of the Northern Kingdom who fits that description. It's possible, not likely, but possible, he's heard things our spies have not." The King said, "And that he might also decide to warn us if he heard such a thing. But there's still the matter of *your* trustworthiness. How should I know that you are not a lying spy, sent form the North to deceive?"

Orin's eyes grew wide with fear as he attempted to speak, "ple-please, s-s-sire, I w-would n-n-never be-betray my kingdom!"

"So a spy would say." Orin opened his mouth to protest but the King interrupted "No, nothing you can say will persuade me you aren't just a well trained spy." The King smiled, "But I have been giving thought to how I may judge your information's usefulness. You are an artisan of some skill, right? You're better dressed than a peasant can afford."

Orin spoke "A potter, sir." After a pause he then bashfully admits, "Of kinds both functional and beautiful, as I've been told by my more affluent clients."

The King smiled wider, "Then you are well off, yes? How much would you consider your current wares and savings worth?"

Nervous about the King's sudden eagerness, Orin hesitantly replies, "700 coins, but p-perhaps even 800 c-coins... If I sold my s-shop and everything w-within."

"Very well. I propose a wager. 20 to 1 against this invasion being real." The king laughed as he saw Orin's shocked face. "What, surprised your King is a betting man? If you'd like to convince me you're not lying, then put your money where your mouth is, I say. If you'd also like to convince me you're right about this, you'll have to bet big. If you're willing to put up 1000 coins I'm willing to call for the removal of the bridges." Orin just stood there silently, jaw agape. The King continued speaking, "That's 200 coins of debt if you're wrong about this, a lifetime of payments for someone of your skills. If you're right, however, you will be rewarded handsomely. 20,000 coins is enough to keep you from working the rest of your life, if you'd like. I think that's fair compensation for saving the kingdom."

The King just smiled as he waited for Orin to speak.

Orin remained silent as he fervently thought about his options. He swallowed several times and wrung his hands together. After several minutes of silence he took a breath and spoke, "I'll take it." The King's smile grew as Orin spoke, "I'll take the bet. After considering his trustworthiness, it seems like my friend is right. I am willing to risk myself for his word. I am not willing to risk my kingdom."

"Very well." The King said before looking towards the two men standing to Orin's side, "Guards, one of you notify the city that an attack is impending. We have 2 days at most before the Northern Kingdom is here." The left one nods at once and left the chamber. "Orin, I hope you understand why you should stay here for the night. We can't have you running off." Orin nodded stiffly in understanding. Looking at the remaining guard, the King said, "Orin here is your responsibility. Keep him occupied and within the castle until you have my word to release him. You may send out someone to notify his family of the circumstances surrounding his stay. They are welcome to come visit as soon as their duties for preparation are complete." After a short thought he said, "Orin is a guest here, not a prisoner, so treat him as such."

Standing up from his throne, the King walked over to where Orin was standing, petrified by what was happening around him. The King, towering over the small man, said, "If you are right about this, I am incredibly grateful that you came to me." The King reached out and grabbed Orin's shoulder, looking into his eyes with his own, and smiled wide. He then released him and returned to his seat. "You may go, I'm soon to be swamped by my bureaucracy for the coming hours as we prepare for this fight." Orin and his escort made their way from the room. As the door closed behind them another one opened as several official looking men rushed in, chatting loudly. The King straightened his stature and forced a smile as he prepared himself for dealing with his government for the next several days.

Fifty Shades of Self-Fulfilling Prophecy

20 PhilGoetz 24 July 2014 12:17AM

The official story: "Fifty Shades of Grey" was a Twilight fan-fiction that had over two million downloads online. The publishing giant Vintage Press saw that number and realized there was a huge, previously-unrealized demand for stories like this. They filed off the Twilight serial numbers, put it in print, marketed it like hell, and now it's sold 60 million copies.

The reality is quite different.

continue reading »

Questioning and Respect

20 jkaufman 10 June 2014 10:52AM
A: [Surprising fact]
B: [Question]

When someone has a claim questioned, there are two common responses. One is to treat the question as a challenge, intended as an insult or indicating a lack of trust. If you have this model of interaction you think people should take your word for things, and feel hurt when they don't. Another response is to treat the question as a signal of respect: they take what you're saying seriously and are trying to integrate it into their understanding of the world. If you have this model of interaction then it's the people who smile, nod, and give no indication of their disagreement that are being disrespectful.

Within either of these groups you can just follow the social norm, but it's harder across groups. Recently I was talking to a friend who claimed that in their state income taxes per dollar went down as you earned more. This struck me as really surprising and kind of unlikely: usually it goes the other way around. [1] I'm very much in the latter group described above, while I was pretty sure my friend was in the former. Even though I suspected they would treat it as disrespectful if I asked for details and tried to confirm their claim, it would have felt much more disrespectful for me to just pretend to accept it and move on. What do you do in situations like this?

(Especially given that I think the "disagreement as respect" version builds healthier communities...)


[1] Our tax system does have regressive components, where poor people sometimes pay a higher percentage of their income as tax than richer people, but it's things like high taxes on cigarettes (which rich people don't consume as much), sales taxes (rich people spend less of their income), and a lower capital gains tax rate (poorer people earn way less in capital gains). I tried to clarify to see if this is what my friend meant, but they were clear that they were talking about "report your income to the state, get charged a higher percentage as tax if your income is lower".

I also posted this on my blog.

Brainstorming for post topics

20 NancyLebovitz 31 May 2014 03:08PM

I suggested  recently that part of the problem with with LW was a lock of discussion posts which was caused by people not thinking of much to post about.

When I ask myself "what might be a good topic for a post?", my mind goes blank, but surely not everything that's worth saying that's related to rationality has been said.

So, is there something at the back of your mind which might be interesting? A topic which got some discussion in an open thread that could be worth pursuing?

If you've found anything which helps you generate useable ideas, please comment about it-- or possibly write a post on the subject.

Exploring Botworld

20 So8res 30 April 2014 10:29PM

Many people have been asking me this question:

But what am I supposed to do with Botworld?

This indicates a failure of communication on my part. In this post, I'll try to resolve that question. As part of this attempt, I've made some updates to the Botworld code (which is now v1.1) which make Botworld a bit more approachable. A changelog and some documentation are included below.

continue reading »

Some alternatives to “Friendly AI”

18 lukeprog 15 June 2014 07:53PM

Cross-posted from my blog.

What does MIRI's research program study?

The most established term for this was coined by MIRI founder Eliezer Yudkowsky: "Friendly AI." The term has some advantages, but it might suggest that MIRI is trying to build C-3PO, and it sounds a bit whimsical for a serious research program.

What about safe AGI or AGI safety? These terms are probably easier to interpret than Friendly AI. Also, people like being safe, and governments like saying they're funding initiatives to keep the public safe.

A friend of mine worries that these terms could provoke a defensive response (in AI researchers) of "Oh, so you think me and everybody else in AI is working on unsafe AI?" But I've never actually heard that response to "AGI safety" in the wild, and AI safety researchers regularly discuss "software system safety" and "AI safety" and "agent safety" and more specific topics like "safe reinforcement learning" without provoking negative reactions from people doing regular AI research.

I'm more worried that a term like "safe AGI" could provoke a response of "So you're trying to make sure that a system which is smarter than humans, and able to operate in arbitrary real-world environments, and able to invent new technologies to achieve its goals, will be safe? Let me save you some time and tell you right now that's impossible. Your research program is a pipe dream."

My reply goes something like "Yeah, it's way beyond our current capabilities, but lots of things that once looked impossible are now feasible because people worked really hard on them for a long time, and we don't think we can get the whole world to promise never to build AGI just because it's hard to make safe, so we're going to give AGI safety a solid try for a few decades and see what can be discovered." But that's probably not all that reassuring.

How about high-assurance AGI? In computer science, a "high assurance system" is one built from the ground up for unusually strong safety and/or security guarantees, because it's going to be used in safety-critical applications where human lives — or sometimes simply billions of dollars — are at stake (e.g. autopilot software or Mars rover software). So there's a nice analogy to MIRI's work, where we're trying to figure out what an AGI would look like if it was built from the ground up to get the strongest safety guarantees possible for such an autonomous and capable system.

I think the main problem with this term is that, quite reasonably, nobody will believe that we can ever get anywhere near as much assurance in the behavior of an AGI as we can in the behavior of, say, the relatively limited AI software that controls the European Train Control System. "High assurance AGI" sounds a bit like "Totally safe all-powerful demon lord." It sounds even more wildly unimaginable to AI researchers than "safe AGI."

What about superintelligence control or AGI control, as in Bostrom (2014)? "AGI control" is perhaps more believable than "high-assurance AGI" or "safe AGI," since it brings to mind AI containment methods, which sound more feasible to most people than designing an unconstrained AGI that is somehow nevertheless safe. (It's okay if they learn later that containment probably isn't an ultimate solution to the problem.)

On the other hand, it might provoke a reaction of "What, you don't think sentient robots have any rights, and you're free to control and confine them in any way you please? You're just repeating the immoral mistakes of the old slavemasters!" Which of course isn't true, but it takes some time to explain how I can think it's obvious that conscious machines have moral value while also being in favor of AGI control methods.

How about ethical AGI? First, I worry that it sounds too philosophical, and philosophy is widely perceived as a confused, unproductive discipline. Second, I worry that it sounds like the research assumes moral realism, which many (most?) intelligent people reject. Third, it makes it sound like most of the work is in selecting the goal function, which I don't think is true.

What about beneficial AGI? That's better than "ethical AGI," I think, but like "ethical AGI" and "Friendly AI," the term sounds less like a serious math and engineering discipline and more like some enclave of crank researchers writing a flurry of words (but no math) about how AGI needs to be "nice" and "trustworthy" and "not harmful" and oh yeah it must be "virtuous" too, whatever that means.

So yeah, I dunno. I think "AGI safety" is my least-disliked term these days, but I wish I knew of some better options.

Politics is hard mode

17 RobbBB 21 July 2014 10:14PM

Summary: I don't think 'politics is the mind-killer' works well rthetorically. I suggest 'politics is hard mode' instead.


 

Some people in and catawampus to the LessWrong community have objected to "politics is the mind-killer" as a framing (/ slogan / taunt). Miri Mogilevsky explained on Facebook:

My usual first objection is that it seems odd to single politics out as a “mind-killer” when there’s plenty of evidence that tribalism happens everywhere. Recently, there has been a whole kerfuffle within the field of psychology about replication of studies. Of course, some key studies have failed to replicate, leading to accusations of “bullying” and “witch-hunts” and what have you. Some of the people involved have since walked their language back, but it was still a rather concerning demonstration of mind-killing in action. People took “sides,” people became upset at people based on their “sides” rather than their actual opinions or behavior, and so on.

Unless this article refers specifically to electoral politics and Democrats and Republicans and things (not clear from the wording), “politics” is such a frightfully broad category of human experience that writing it off entirely as a mind-killer that cannot be discussed or else all rationality flies out the window effectively prohibits a large number of important issues from being discussed, by the very people who can, in theory, be counted upon to discuss them better than most. Is it “politics” for me to talk about my experience as a woman in gatherings that are predominantly composed of men? Many would say it is. But I’m sure that these groups of men stand to gain from hearing about my experiences, since some of them are concerned that so few women attend their events.

In this article, Eliezer notes, “Politics is an important domain to which we should individually apply our rationality — but it’s a terrible domain in which to learn rationality, or discuss rationality, unless all the discussants are already rational.” But that means that we all have to individually, privately apply rationality to politics without consulting anyone who can help us do this well. After all, there is no such thing as a discussant who is “rational”; there is a reason the website is called “Less Wrong” rather than “Not At All Wrong” or “Always 100% Right.” Assuming that we are all trying to be more rational, there is nobody better to discuss politics with than each other.

The rest of my objection to this meme has little to do with this article, which I think raises lots of great points, and more to do with the response that I’ve seen to it — an eye-rolling, condescending dismissal of politics itself and of anyone who cares about it. Of course, I’m totally fine if a given person isn’t interested in politics and doesn’t want to discuss it, but then they should say, “I’m not interested in this and would rather not discuss it,” or “I don’t think I can be rational in this discussion so I’d rather avoid it,” rather than sneeringly reminding me “You know, politics is the mind-killer,” as though I am an errant child. I’m well-aware of the dangers of politics to good thinking. I am also aware of the benefits of good thinking to politics. So I’ve decided to accept the risk and to try to apply good thinking there. [...]

I’m sure there are also people who disagree with the article itself, but I don’t think I know those people personally. And to add a political dimension (heh), it’s relevant that most non-LW people (like me) initially encounter “politics is the mind-killer” being thrown out in comment threads, not through reading the original article. My opinion of the concept improved a lot once I read the article.

In the same thread, Andrew Mahone added, “Using it in that sneering way, Miri, seems just like a faux-rationalist version of ‘Oh, I don’t bother with politics.’ It’s just another way of looking down on any concerns larger than oneself as somehow dirty, only now, you know, rationalist dirty.” To which Miri replied: “Yeah, and what’s weird is that that really doesn’t seem to be Eliezer’s intent, judging by the eponymous article.”

Eliezer replied briefly, to clarify that he wasn't generally thinking of problems that can be directly addressed in local groups (but happen to be politically charged) as "politics":

Hanson’s “Tug the Rope Sideways” principle, combined with the fact that large communities are hard to personally influence, explains a lot in practice about what I find suspicious about someone who claims that conventional national politics are the top priority to discuss. Obviously local community matters are exempt from that critique! I think if I’d substituted ‘national politics as seen on TV’ in a lot of the cases where I said ‘politics’ it would have more precisely conveyed what I was trying to say.

But that doesn't resolve the issue. Even if local politics is more instrumentally tractable, the worry about polarization and factionalization can still apply, and may still make it a poor epistemic training ground.

A subtler problem with banning “political” discussions on a blog or at a meet-up is that it’s hard to do fairly, because our snap judgments about what counts as “political” may themselves be affected by partisan divides. In many cases the status quo is thought of as apolitical, even though objections to the status quo are ‘political.’ (Shades of Pretending to be Wise.)

Because politics gets personal fast, it’s hard to talk about it successfully. But if you’re trying to build a community, build friendships, or build a movement, you can’t outlaw everything ‘personal.’

And selectively outlawing personal stuff gets even messier. Last year, daenerys shared anonymized stories from women, including several that discussed past experiences where the writer had been attacked or made to feel unsafe. If those discussions are made off-limits because they relate to gender and are therefore ‘political,’ some folks may take away the message that they aren’t allowed to talk about, e.g., some harmful or alienating norm they see at meet-ups. I haven’t seen enough discussions of this failure mode to feel super confident people know how to avoid it.

Since this is one of the LessWrong memes that’s most likely to pop up in cross-subcultural dialogues (along with the even more ripe-for-misinterpretation “policy debates should not appear one-sided“…), as a first (very small) step, my action proposal is to obsolete the ‘mind-killer’ framing. A better phrase for getting the same work done would be ‘politics is hard mode’:

1. ‘Politics is hard mode’ emphasizes that ‘mind-killing’ (= epistemic difficulty) is quantitative, not qualitative. Some things might instead fall under Middlingly Hard Mode, or under Nightmare Mode…

2. ‘Hard’ invites the question ‘hard for whom?’, more so than ‘mind-killer’ does. We’re used to the fact that some people and some contexts change what’s ‘hard’, so it’s a little less likely we’ll universally generalize.

3. ‘Mindkill’ connotes contamination, sickness, failure, weakness. In contrast, ‘Hard Mode’ doesn’t imply that a thing is low-status or unworthy. As a result, it’s less likely to create the impression (or reality) that LessWrongers or Effective Altruists dismiss out-of-hand the idea of hypothetical-political-intervention-that-isn’t-a-terrible-idea. Maybe some people do want to argue for the thesis that politics is always useless or icky, but if so it should be done in those terms, explicitly — not snuck in as a connotation.

4. ‘Hard Mode’ can’t readily be perceived as a personal attack. If you accuse someone of being ‘mindkilled’, with no context provided, that smacks of insult — you appear to be calling them stupid, irrational, deluded, or the like. If you tell someone they’re playing on ‘Hard Mode,’ that’s very nearly a compliment, which makes your advice that they change behaviors a lot likelier to go over well.

5. ‘Hard Mode’ doesn’t risk bringing to mind (e.g., gendered) stereotypes about communities of political activists being dumb, irrational, or overemotional.

6. ‘Hard Mode’ encourages a growth mindset. Maybe some topics are too hard to ever be discussed. Even so, ranking topics by difficulty encourages an approach where you try to do better, rather than merely withdrawing. It may be wise to eschew politics, but we should not fear it. (Fear is the mind-killer.)

7. Edit: One of the larger engines of conflict is that people are so much worse at noticing their own faults and biases than noticing others'. People will be relatively quick to dismiss others as 'mindkilled,' while frequently flinching away from or just-not-thinking 'maybe I'm a bit mindkilled about this.' Framing the problem as a challenge rather than as a failing might make it easier to be reflective and even-handed.

This is not an attempt to get more people to talk about politics. I think this is a better framing whether or not you trust others (or yourself) to have productive political conversations.

When I playtested this post, Ciphergoth raised the worry that 'hard mode' isn't scary-sounding enough. As dire warnings go, it's light-hearted—exciting, even. To which I say: good. Counter-intuitive fears should usually be argued into people (e.g., via Eliezer's politics sequence), not connotation-ninja'd or chanted at them. The cognitive content is more clearly conveyed by 'hard mode,' and if some group (people who love politics) stands to gain the most from internalizing this message, the message shouldn't cast that very group (people who love politics) in an obviously unflattering light. LW seems fairly memetically stable, so the main issue is what would make this meme infect friends and acquaintances who haven't read the sequences. (Or Dune.)

If you just want a scary personal mantra to remind yourself of the risks, I propose 'politics is SPIDERS'. Though 'politics is the mind-killer' is fine there too.

If you and your co-conversationalists haven’t yet built up a lot of trust and rapport, or if tempers are already flaring, conveying the message ‘I’m too rational to discuss politics’ or ‘You’re too irrational to discuss politics’ can make things worse. In that context, ‘politics is the mind-killer’ is the mind-killer. At least, it’s a needlessly mind-killing way of warning people about epistemic hazards.

‘Hard Mode’ lets you speak as the Humble Aspirant rather than the Aloof Superior. Strive to convey: ‘I’m worried I’m too low-level to participate in this discussion; could you have it somewhere else?’ Or: ‘Could we talk about something closer to Easy Mode, so we can level up together?’ More generally: If you’re worried that what you talk about will impact group epistemology, you should be even more worried about how you talk about it.

An Experiment In Social Status: Software Engineer vs. Data Science Manager

17 JQuinton 15 July 2014 08:24PM

Here is an interesting blog post about a guy who did a resume experiment between two positions which he argues are by experience identical, but occupy different "social status" positions in tech: A software engineer and a data manager.

Interview A: as Software Engineer

Bill faced five hour-long technical interviews. Three went well. One was so-so, because it focused on implementation details of the JVM, and Bill’s experience was almost entirely in C++, with a bit of hobbyist OCaml. The last interview sounds pretty hellish. It was with the VP of Data Science, Bill’s prospective boss, who showed up 20 minutes late and presented him with one of those interview questions where there’s “one right answer” that took months, if not years, of in-house trial and error to discover. It was one of those “I’m going to prove that I’m smarter than you” interviews...

Let’s recap this. Bill passed three of his five interviews with flying colors. One of the interviewers, a few months later, tried to recruit Bill to his own startup. The fourth interview was so-so, because he wasn’t a Java expert, but came out neutral. The fifth, he failed because he didn’t know the in-house Golden Algorithm that took years of work to discover. When I asked that VP/Data Science directly why he didn’t hire Bill (and he did not know that I knew Bill, nor about this experiment) the response I got was “We need people who can hit the ground running.” Apparently, there’s only a “talent shortage” when startup people are trying to scam the government into changing immigration policy. The undertone of this is that “we don’t invest in people”.

Or, for a point that I’ll come back to, software engineers lack the social status necessary to make others invest in them.

Interview B: as Data Science manager.

A couple weeks later, Bill interviewed at a roughly equivalent company for the VP-level position, reporting directly to the CTO.

Worth noting is that we did nothing to make Bill more technically impressive than for Company A. If anything, we made his technical story more honest, by modestly inflating his social status while telling a “straight shooter” story for his technical experience. We didn’t have to cover up periods of low technical activity; that he was a manager, alone, sufficed to explain those away.

Bill faced four interviews, and while the questions were behavioral and would be “hard” for many technical people, he found them rather easy to answer with composure. I gave him the Golden Answer, which is to revert to “There’s always a trade-off between wanting to do the work yourself, and knowing when to delegate.” It presents one as having managerial social status (the ability to delegate) but also a diligent interest in, and respect for, the work. It can be adapted to pretty much any “behavioral” interview question...

Bill passed. Unlike for a typical engineering position, there were no reference checks. The CEO said, “We know you’re a good guy, and we want to move fast on you”. As opposed tot he 7-day exploding offers typically served to engineers, Bill had 2 months in which to make his decision. He got a fourth week of vacation without even having to ask for it, and genuine equity (about 75% of a year’s salary vesting each year)...

It was really interesting, as I listened in, to see how different things are once you’re “in the club”. The CEO talked to Bill as an equal, not as a paternalistic, bullshitting, “this is good for your career” authority figure. There was a tone of equality that a software engineer would never get from the CEO of a 100-person tech company.

The author concludes that positions that are labeled as code-monkey-like are low status, while positions that are labeled as managerial are high status. Even if they are "essentially" doing the same sort of work.

Not sure about this methodology, but it's food for thought.

Identification of Force Multipliers for Success

17 Nick5a1 21 June 2014 05:15AM

For a while now I've been very interested in learning useful knowledge and acquiring useful skills. Of course there's no shortage of useful knowledge and skills to acquire, and so I've often thought about how best to spend my limited time learning.

When I came across the concept of Force Multiplication, it seemed like an appropriate metaphor for a strategy to apply to choosing where to invest my time and energy in acquiring useful skills and knowledge. I started to think about what areas or skills would make sense to learn about or acquire first, to:

  1. increase speed or ease of further learning/skill acquisition,
  2. help me achieve success not only in my current goals, but in later goals that I have not yet developed, and
  3. lead to interesting downstream options or other knowledge/skills to acquire.

There have been a small number of skills/areas that have helped me surge forward in progress towards my goals. I look back at these areas and wish only that I had come across them sooner. As most of my adult life has been focused on business, most of those areas that have had a tremendous impact on my progress have been business related, but not all.

So far I've found it hard to identify these areas in advance. Almost all of the skills or knowledge  that I learned, that had a large impact on progress towards success, I pursued for unrelated reasons, or I had no concept of how truly useful they would be. The only solution I currently have for identifying force multipliers is to ask other people, and especially those more accomplished than me, what they've learned that had the most impact on their progress towards success.

So, what have you learned that had the most impact on your progress towards success (whatever that might be)?

Can you think of any other ways to identify areas of force multiplication?

[LINK] Scott Aaronson on Google, Breaking Circularity and Eigenmorality

17 shminux 19 June 2014 08:17PM

Scott suggests that ranking morality is similar to ranking web pages. A quote:

Philosophers from Socrates on, I was vaguely aware, had struggled to define what makes a person “moral” or “virtuous,” without tacitly presupposing the answer.  Well, it seemed to me that, as a first attempt, one could do a lot worse than the following:

A moral person is someone who cooperates with other moral people, and who refuses to cooperate with immoral people.

Proposed solution:

Just like in CLEVER or PageRank, we can begin by giving everyone in the community an equal number of “morality starting credits.”  Then we can apply an iterative update rule, where each person A can gain morality credits by cooperating with each other person B, and A gains more credits the more credits B has already.  We apply the rule over and over, until the number of morality credits per person converges to an equilibrium.  (Or, of course, we can shortcut the process by simply finding the principal eigenvector of the “cooperation matrix,” using whatever algorithm we like.)  We then have our objective measure of morality for each individual, solving a 2400-year-old open problem in philosophy.

He then talks about "eigenmoses and eigenjesus" and other fun ideas, like Plato at the Googleplex.

One final quote:

All that's needed to unravel the circularity is a principal eigenvector computation on the matrix of trust.

EDIT: I am guessing that after judicious application of this algorithm one would end up with the other Scott A's loosely connected components with varying definitions of morality, the Archipelago. UPDATE: He chimes in.

EDIT2: The obvious issue of equating prevailing mores with morality is discussed to death in the comments. Please read them first before raising it yet again here.

 

Strawman Yourself

17 katydee 18 May 2014 05:28AM

One good way to ensure that your plans are robust is to strawman yourself. Look at your plan in the most critical, contemptuous light possible and come up with the obvious uncharitable insulting argument for why you will fail.

In many cases, the obvious uncharitable insulting argument will still be fundamentally correct.

If it is, your plan probably needs work. This technique seems to work not because it taps into some secret vault of wisdom (after all, making fun of things is easy), but because it is an elegant way to shift yourself into a critical mindset.

For instance, I recently came up with a complex plan to achieve one of my goals. Then I strawmanned myself; the strawman version of why this plan would fail was simply "large and complicated plans don't work." I thought about that for a moment, concluded "yep, large and complicated plans don't work," and came up with a simple, elegant plan to achieve the same ends.

You may ask "why didn't you just come up with a simple, elegant plan in the first place?" The answer is that elegance is hard. It's easier to add on special case after special case, not realizing how much complexity debt you've added. Strawmanning yourself is one way to safeguard against this risk, as well as many others.

Rebutting radical scientific skepticism

17 asr 30 April 2014 07:40PM

Suppose you distrusted everything you had ever read about science. How much of modern scientific knowledge could you verify for yourself, using only your own senses and the sort of equipment you could easily obtain?  How about if you accept third-party evidence when many thousands of people can easily check the facts?

 

continue reading »

MIRI Donation Collaboration Station

17 Skeptityke 29 April 2014 02:11PM

As you may know, on May 6, there will be a large one-day price-matching fundraiser for Bay Area Charities.

The relevant details are right here at MIRI's official website.

And this is the webpage to visit to donate.

For those of you who didn't read the two links above, here's the important information.

On May 6, MIRI is participating in Silicon Valley Gives...

Why is this exciting for supporters of MIRI? Many reasons, but here are a few.

 

  • Over $250,000 of matching prizes and funds up for grabs, from sources that normally wouldn't contribute to MIRI:
  • Two-to-one dollar match up to $50,000 during the midnight hour.
  • $2,000 dollar prize for the nonprofit that has the most individual gift in an hour, every hour, for 24 hours.
  • $150 added to a random donation each hour, every hour for 24 hours.
  • Dollar for Dollar match up to $35,000 during the 7AM hour, and $50,000 during the noon, 6 PM, and 7 PM hours.
  • Local NBC stations, radio stations, businesses, and Bay Area foundations will be promoting the Silicon Valley Day of Giving on May 6th. So if MIRI is making a splash with our fundraising that day, it's possible we'll draw attention from media and by extension new donors.

 

Making the most of this opportunity will require some cleverness and a lot of coordination. We are going to need all the help we can get...

 

  1. If you are interested in supporting MIRI with a large donation during the fundraiser... Get in touch with Malo at malo@intelligence.org
  2. All MIRI supporters have the potential to make a big impact if we can all work together in a coordinated manner. Sign up below (The sign up sheet is on the MIRI announcement page) to receive updates on our strategy leading up to the event, and updates throughout the fundraiser on the best times to give and promote the event.

 

A group coordination problem? We can do that easily, right?
So, in the comments, feel free to discuss donation strategy, coordinate groups of people, provide positive reinforcement for donors, etc.

With a preliminary scan through the prize list, a viable strategy seems to be to funnel most of the donations in during the first hour (midnight) for the two-to-one matching until the matching limit, and do the the rest of the donations during 6 and 7 PM. The small donors can all coordinate on a few hours where few people will be donating (inconvenient morning hours should work well for this) to secure a 2000 dollar prize or two, and since 1000 dollars will be given to the first 50 organizations to receive a gift in the 5 PM hour, getting a handful of people together to donate right at 5 also seems useful.

Mark your calendars!

 

 

How do you notice when you are ignorant of necessary alternative hypotheses?

16 eli_sennesh 24 June 2014 06:12PM

So I just wound up in a debate with someone over on Reddit about the value of conventional academic philosophy.  He linked me to a book review, in which both the review and the book are absolutely godawful.  That is, the author (and the reviewer following him) start with ontological monism (the universe only contains a single kind of Stuff: mass-energy), adds in the experience of consciousness, reasons deftly that emergence is a load of crap... and then arrives to the conclusion of panpsychism.

WAIT HOLD ON, DON'T FLAME YET!

Of course panpsychism is bunk.  I would be embarrassed to be caught upholding it, given the evidence I currently have, but what I want to talk about is the logic being followed.

1) The universe is a unified, consistent whole.  Good!

2) The universe contains the experience/existence of consciousness.  Easily observable.

3) If consciousness exists, something in the universe must cause or give rise to consciousness.  Good reasoning!

4) "Emergence" is a non-explanation, so that can't be it.  Good!

5) Therefore, whatever stuff the unified universe is made of must be giving rise to consciousness in a nonemergent way.

6) Therefore, the stuff must be innately "mindy".

What went wrong in steps (5) and (6)?  The man was actually reasoning more-or-less correctly!  Given the universe he lived in, and the impossibility of emergence, he reallocated his probability mass to the remaining answer.  When he had eliminated the impossible, whatever remained, however low its prior, must be true.

The problem was, he eliminated the impossible, but left open a huge vast space of possible hypotheses that he didn't know about (but which we do): the most common of these is the computational theory of mind and consciousness, which says that we are made of cognitive algorithms.  A Solomonoff Inducer can just go on to the next length of bit-strings describing Turing machines, but we can't.

Now, I can spot the flaw in the reasoning here.  What frightens me is: what if I'm presented with some similar argument, and I can't spot the flaw?  What if, instead, I just neatly and stupidly reallocate my belief to what seems to me to be the only available alternative, while failing to go out and look for alternatives I don't already know about?  Notably, it seems like expected evidence is conserved, but expecting to locate new hypotheses means I should be reducing my certainty about all currently-available hypotheses now to have some for dividing between the new possibilities.

If you can notice when you're confused, how do you notice when you're ignorant?

Request for concrete AI takeover mechanisms

16 KatjaGrace 28 April 2014 01:04AM

Any scenario where advanced AI takes over the world requires some mechanism for an AI to leverage its position as ethereal resident of a computer somewhere into command over a lot of physical resources.

One classic story of how this could happen, from Eliezer:

  1. Crack the protein folding problem, to the extent of being able to generate DNA strings whose folded peptide sequences fill specific functional roles in a complex chemical interaction.
  2. Email sets of DNA strings to one or more online laboratories which offer DNA synthesis, peptide sequencing, and FedEx delivery. (Many labs currently offer this service, and some boast of 72-hour turnaround times.)
  3. Find at least one human connected to the Internet who can be paid, blackmailed, or fooled by the right background story, into receiving FedExed vials and mixing them in a specified environment.
  4. The synthesized proteins form a very primitive “wet” nanosystem which, ribosomelike, is capable of accepting external instructions; perhaps patterned acoustic vibrations delivered by a speaker attached to the beaker.
  5. Use the extremely primitive nanosystem to build more sophisticated systems, which construct still more sophisticated systems, bootstrapping to molecular nanotechnology—or beyond.

You can do a lot of reasoning about AI takeover without any particular picture of how the world gets taken over. Nonetheless it would be nice to have an understanding of these possible routes. For preparation purposes, and also because a concrete, plausible pictures of doom are probably more motivating grounds for concern than abstract arguments.

So MIRI is interested in making a better list of possible concrete routes to AI taking over the world. And for this, we ask your assistance.

What are some other concrete AI takeover mechanisms? If an AI did not have a solution to the protein folding problem, and a DNA synthesis lab to write off to, what else might it do? 

We would like suggestions that take an AI from being on an internet-connected computer to controlling substantial physical resources, or having substantial manufacturing ability.

We would especially like suggestions which are plausible given technology that normal scientists would expect in the next 15 years. So limited involvement of advanced nanotechnology and quantum computers would be appreciated. 

We welcome partial suggestions, e.g. 'you can take control of a self-driving car from the internet - probably that could be useful in some schemes'. 

Thank you!

R support group and the benefits of applied statistics

15 sixes_and_sevens 26 June 2014 02:11PM

Following the interest in this proposal a couple of weeks ago, I've set up a Google Group for the purpose of giving people a venue to discuss R, talk about their projects, seek advice, share resources, and provide a social motivator to hone their skills. Having done this, I'd now like to bullet-point a few reasons for learning applied statistical skills in general, and R in particular:

The General Case:

- Statistics seems to be a subject where it's easy to delude yourself into thinking you know a lot about it. This is visibly apparent on Less Wrong. Although there are many subject experts on here, there are also a lot of people making bold pronouncements about Bayesian inference who wouldn't recognise a beta distribution if it sat on them. Don't be that person! It's hard to fool yourself into thinking you know something when you have to practically apply it.

- Whenever you think "I wonder what kind of relationship exists between [x] and [y]", it's within your power to investigate this.

- Statistics has a rich conceptual vocabulary for reasoning about how observations generalise, and how useful those generalisations might be when making inferences about future observations. These are the sorts of skills we want to be practising as aspiring rationalists.

- Scientific literature becomes a lot more readable when you appreciate the methods behind them. You'll have a much greater understanding of scientific findings if you appreciate what the finding means in the context of statistical inference, rather than going off whatever paraphrased upshot is given in the abstract.

- Statistical techniques make use of fundamental mathematical methods in an applicable way. If you're learning linear algebra, for example, and you want an intuitive understanding of eigenvectors, you could do a lot worse than learning about principal component analysis.

R in particular:

- It's non-proprietary, (read "free"). Many competitive products are ridiculously expensive to license.

- Since it's common in academia, newer or more exotic statistical tools and procedures are more likely to have been implemented and made available in R than proprietary statistical packages or other software libraries.

- R skills are a strong signal of technical competence that will distinguish you from SPSS mouse-jockeys.

- There are many out-of-the-box packages for carrying out statistical procedures that you'd probably have to cobble together yourself if you were working in Python or Java.

- Having said that, popular languages such as Python and Java have libraries for interfacing with R.

- There's a discussion / support group for R with Less Wrong users in it. :-)

[LINK] Elon Musk interested in AI safety

15 TakisMichel 18 June 2014 10:56PM

http://www.businessinsider.com/musk-on-artificial-intelligence-2014-6

Summary: The only non-Tesla/SpaceX/SolarCity companies that Musk is invested in are DeepMind and Vicarious, due to vague feelings of wanting AI to not unintentionally go Terminator. The best part of the article is the end, where he acknowledges that Mars isn't a get-out-of-jail-free card any more: "KE: Or escape to mars if there is no other option. MUSK: The A.I. will chase us there pretty quickly." Thinking of SpaceX not as a childhood dream, but as one specific arms supplier in the war against existential risks, puts things into perspective for him.

How has technology changed social skills?

15 philh 08 June 2014 12:41PM

At LW London last week, someone mentioned the possibility of a Google Glass app doing face recognition on people. If you've met someone before, it tells you their name, how you know them, etc. Someone else mentioned that this could reduce the social capital of people who are already good at this.

A third person said that something similar happened when Facebook started telling everyone when everyone else's birthday was. Previously he got points by making an effort to remember, but those points are no longer available.

Are there other social skills that technology has made obsolete? And the reverse question that it only just occured to me to ask, are there social skills that are only useful because of technology?

I'm not really sure what sorts of things I'm looking for here. "Ability to ask for directions" seems like one example, but it feels kind of noncentral to me, I don't know why. But I'm mostly just curious.

View more: Next