Filter Last three months

Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

I Want To Live In A Baugruppe

44 Alicorn 17 March 2017 01:36AM

Rationalists like to live in group houses.  We are also as a subculture moving more and more into a child-having phase of our lives.  These things don't cooperate super well - I live in a four bedroom house because we like having roommates and guests, but if we have three kids and don't make them share we will in a few years have no spare rooms at all.  This is frustrating in part because amenable roommates are incredibly useful as alloparents if you value things like "going to the bathroom unaccompanied" and "eating food without being screamed at", neither of which are reasonable "get a friend to drive for ten minutes to spell me" situations.  Meanwhile there are also people we like living around who don't want to cohabit with a small child, which is completely reasonable, small children are not for everyone.

For this and other complaints ("househunting sucks", "I can't drive and need private space but want friends accessible", whatever) the ideal solution seems to be somewhere along the spectrum between "a street with a lot of rationalists living on it" (no rationalist-friendly entity controls all those houses and it's easy for minor fluctuations to wreck the intentional community thing) and "a dorm" (sorta hard to get access to those once you're out of college, usually not enough kitchens or space for adult life).  There's a name for a thing halfway between those, at least in German - "baugruppe" - buuuuut this would require community or sympathetic-individual control of a space and the money to convert it if it's not already baugruppe-shaped.

Maybe if I complain about this in public a millionaire will step forward or we'll be able to come up with a coherent enough vision to crowdfund it or something.  I think there is easily enough demand for a couple of ten-to-twenty-adult baugruppen (one in the east bay and one in the south bay) or even more/larger, if the structures materialized.  Here are some bulleted lists.

Desiderata:

  • Units that it is really easy for people to communicate across and flow between during the day - to my mind this would be ideally to the point where a family who had more kids than fit in their unit could move the older ones into a kid unit with some friends for permanent sleepover, but still easily supervise them.  The units can be smaller and more modular the more this desideratum is accomplished.
  • A pricing structure such that the gamut of rationalist financial situations (including but not limited to rent-payment-constraining things like "impoverished app academy student", "frugal Google engineer effective altruist", "NEET with a Patreon", "CfAR staffperson", "not-even-ramen-profitable entrepreneur", etc.) could live there.  One thing I really like about my house is that Spouse can pay for it himself and would by default anyway, and we can evaluate roommates solely on their charming company (or contribution to childcare) even if their financial situation is "no".  However, this does require some serious participation from people whose financial situation is "yes" and a way to balance the two so arbitrary numbers of charity cases don't bankrupt the project.
  • Variance in amenities suited to a mix of Soylent-eating restaurant-going takeout-ordering folks who only need a fridge and a microwave and maybe a dishwasher, and neighbors who are not that, ideally such that it's easy for the latter to feed neighbors as convenient.
  • Some arrangement to get repairs done, ideally some compromise between "you can't do anything to your living space, even paint your bedroom, because you don't own the place and the landlord doesn't trust you" and "you have to personally know how to fix a toilet".
  • I bet if this were pulled off at all it would be pretty easy to have car-sharing bundled in, like in Benton House That Was which had several people's personal cars more or less borrowable at will.  (Benton House That Was may be considered a sort of proof of concept of "20 rationalists living together" but I am imagining fewer bunk beds in the baugruppe.)  Other things that could be shared include longish-term storage and irregularly used appliances.
  • Dispute resolution plans and resident- and guest-vetting plans which thread the needle between "have to ask a dozen people before you let your brother crash on the couch, let alone a guest unit" and "cannot expel missing stairs".  I think there are some rationalist community Facebook groups that have medium-trust networks of the right caution level and experiment with ways to maintain them.

Obstacles:

  • Bikeshedding.  Not that it isn't reasonable to bikeshed a little about a would-be permanent community edifice that you can't benefit from or won't benefit from much unless it has X trait - I sympathize with this entirely - but too much from too many corners means no baugruppen go up at all even if everything goes well, and that's already dicey enough, so please think hard on how necessary it is for the place to be blue or whatever.
  • Location.  The only really viable place to do this for rationalist population critical mass is the Bay Area, which has, uh, problems, with new construction.  Existing structures are likely to be unsuited to the project both architecturally and zoningwise, although I would not be wholly pessimistic about one of those little two-story hotels with rooms that open to the outdoors or something like that.
  • Principal-agent problems.  I do not know how to build a dormpartment building and probably neither do you.
  • Community norm development with buy-in and a good match for typical conscientiousness levels even though we are rules-lawyery contrarians.

Please share this wherever rationalists may be looking; it's definitely the sort of thing better done with more eyes on it.

Effective altruism is self-recommending

41 Benquo 21 April 2017 06:37PM

A parent I know reports (some details anonymized):

Recently we bought my 3-year-old daughter a "behavior chart," in which she can earn stickers for achievements like not throwing tantrums, eating fruits and vegetables, and going to sleep on time. We successfully impressed on her that a major goal each day was to earn as many stickers as possible.

This morning, though, I found her just plastering her entire behavior chart with stickers. She genuinely seemed to think I'd be proud of how many stickers she now had.

The Effective Altruism movement has now entered this extremely cute stage of cognitive development. EA is more than three years old, but institutions age differently than individuals.

What is a confidence game?

In 2009, investment manager and con artist Bernie Madoff pled guilty to running a massive fraud, with $50 billion in fake return on investment, having outright embezzled around $18 billion out of the $36 billion investors put into the fund. Only a couple of years earlier, when my grandfather was still alive, I remember him telling me about how Madoff was a genius, getting his investors a consistent high return, and about how he wished he could be in on it, but Madoff wasn't accepting additional investors.

What Madoff was running was a classic Ponzi scheme. Investors gave him money, and he told them that he'd gotten them an exceptionally high return on investment, when in fact he had not. But because he promised to be able to do it again, his investors mostly reinvested their money, and more people were excited about getting in on the deal. There was more than enough money to cover the few people who wanted to take money out of this amazing opportunity.

Ponzi schemes, pyramid schemes, and speculative bubbles are all situations in investors' expected profits are paid out from the money paid in by new investors, instead of any independently profitable venture. Ponzi schemes are centrally managed – the person running the scheme represents it to investors as legitimate, and takes responsibility for finding new investors and paying off old ones. In pyramid schemes such as multi-level-marketing and chain letters, each generation of investor recruits new investors and profits from them. In speculative bubbles, there is no formal structure propping up the scheme, only a common, mutually reinforcing set of expectations among speculators driving up the price of something that was already for sale.

The general situation in which someone sets themself up as the repository of others' confidence, and uses this as leverage to acquire increasing investment, can be called a confidence game.

Some of the most iconic Ponzi schemes blew up quickly because they promised wildly unrealistic growth rates. This had three undesirable effects for the people running the schemes. First, it attracted too much attention – too many people wanted into the scheme too quickly, so they rapidly exhausted sources of new capital. Second, because their rates of return were implausibly high, they made themselves targets for scrutiny. Third, the extremely high rates of return themselves caused their promises to quickly outpace what they could plausibly return to even a small share of their investor victims.

Madoff was careful to avoid all these problems, which is why his scheme lasted for nearly half a century. He only promised plausibly high returns (around 10% annually) for a successful hedge fund, especially if it was illegally engaged in insider trading, rather than the sort of implausibly high returns typical of more blatant Ponzi schemes. (Charles Ponzi promised to double investors' money in 90 days.) Madoff showed reluctance to accept new clients, like any other fund manager who doesn't want to get too big for their trading strategy.

He didn't plaster stickers all over his behavior chart – he put a reasonable number of stickers on it. He played a long game.

Not all confidence games are inherently bad. For instance, the US national pension system, Social Security, operates as a kind of Ponzi scheme, it is not obviously unsustainable, and many people continue to be glad that it exists. Nominally, when people pay Social Security taxes, the money is invested in the social security trust fund, which holds interest-bearing financial assets that will be used to pay out benefits in their old age. In this respect it looks like an ordinary pension fund.

However, the financial assets are US Treasury bonds. There is no independently profitable venture. The Federal Government of the United States of America is quite literally writing an IOU to itself, and then spending the money on current expenditures, including paying out current Social Security benefits.

The Federal Government, of course, can write as large an IOU to itself as it wants. It could make all tax revenues part of the Social Security program. It could issue new Treasury bonds and gift them to Social Security. None of this would increase its ability to pay out Social Security benefits. It would be an empty exercise in putting stickers on its own chart.

If the Federal government loses the ability to collect enough taxes to pay out social security benefits, there is no additional capacity to pay represented by US Treasury bonds. What we have is an implied promise to pay out future benefits, backed by the expectation that the government will be able to collect taxes in the future, including Social Security taxes.

There's nothing necessarily wrong with this, except that the mechanism by which Social Security is funded is obscured by financial engineering. However, this misdirection should raise at least some doubts as to the underlying sustainability or desirability of the commitment. In fact, this scheme was adopted specifically to give people the impression that they had some sort of property rights over their social Security Pension, in order to make the program politically difficult to eliminate. Once people have "bought in" to a program, they will be reluctant to treat their prior contributions as sunk costs, and willing to invest additional resources to salvage their investment, in ways that may make them increasingly reliant on it.

Not all confidence games are intrinsically bad, but dubious programs benefit the most from being set up as confidence games. More generally, bad programs are the ones that benefit the most from being allowed to fiddle with their own accounting. As Daniel Davies writes, in The D-Squared Digest One Minute MBA - Avoiding Projects Pursued By Morons 101:

Good ideas do not need lots of lies told about them in order to gain public acceptance. I was first made aware of this during an accounting class. We were discussing the subject of accounting for stock options at technology companies. […] One side (mainly technology companies and their lobbyists) held that stock option grants should not be treated as an expense on public policy grounds; treating them as an expense would discourage companies from granting them, and stock options were a vital compensation tool that incentivised performance, rewarded dynamism and innovation and created vast amounts of value for America and the world. The other side (mainly people like Warren Buffet) held that stock options looked awfully like a massive blag carried out my management at the expense of shareholders, and that the proper place to record such blags was the P&L account.

Our lecturer, in summing up the debate, made the not unreasonable point that if stock options really were a fantastic tool which unleashed the creative power in every employee, everyone would want to expense as many of them as possible, the better to boast about how innovative, empowered and fantastic they were. Since the tech companies' point of view appeared to be that if they were ever forced to account honestly for their option grants, they would quickly stop making them, this offered decent prima facie evidence that they weren't, really, all that fantastic.

However, I want to generalize the concept of confidence games from the domain of financial currency, to the domain of social credit more generally (of which money is a particular form that our society commonly uses), and in particular I want to talk about confidence games in the currency of credit for achievement.

If I were applying for a very important job with great responsibilities, such as President of the United States, CEO of a top corporation, or head or board member of a major AI research institution, I could be expected to have some relevant prior experience. For instance, I might have had some success managing a similar, smaller institution, or serving the same institution in a lesser capacity. More generally, when I make a bid for control over something, I am implicitly claiming that I have enough social credit – enough of a track record – that I can be expected to do good things with that control.

In general, if someone has done a lot, we should expect to see an iceberg pattern where a small easily-visible part suggests a lot of solid but harder-to-verify substance under the surface. One might be tempted to make a habit of imputing a much larger iceberg from the combination of a small floaty bit, and promises. But, a small easily-visible part with claims of a lot of harder-to-see substance is easy to mimic without actually doing the work. As Davies continues:

The Vital Importance of Audit. Emphasised over and over again. Brealey and Myers has a section on this, in which they remind callow students that like backing-up one's computer files, this is a lesson that everyone seems to have to learn the hard way. Basically, it's been shown time and again and again; companies which do not audit completed projects in order to see how accurate the original projections were, tend to get exactly the forecasts and projects that they deserve. Companies which have a culture where there are no consequences for making dishonest forecasts, get the projects they deserve. Companies which allocate blank cheques to management teams with a proven record of failure and mendacity, get what they deserve.

If you can independently put stickers on your own chart, then your chart is no longer reliably tracking something externally verified. If forecasts are not checked and tracked, or forecasters are not consequently held accountable for their forecasts, then there is no reason to believe that assessments of future, ongoing, or past programs are accurate. Adopting a wait-and-see attitude, insisting on audits for actual results (not just predictions) before investing more, will definitely slow down funding for good programs. But without it, most of your funding will go to worthless ones.

Open Philanthropy, OpenAI, and closed validation loops

The Open Philanthropy Project recently announced a $30 million grant to the $1 billion nonprofit AI research organization OpenAI. This is the largest single grant it has ever made. The main point of the grant is to buy influence over OpenAI’s future priorities; Holden Karnofsky, Executive Director of the Open Philanthropy Project, is getting a seat on OpenAI’s board as part of the deal. This marks the second major shift in focus for the Open Philanthropy Project.

The first shift (back when it was just called GiveWell) was from trying to find the best already-existing programs to fund (“passive funding”) to envisioning new programs and working with grantees to make them reality (“active funding”). The new shift is from funding specific programs at all, to trying to take control of programs without any specific plan.

To justify the passive funding stage, all you have to believe is that you can know better than other donors, among existing charities. For active funding, you have to believe that you’re smart enough to evaluate potential programs, just like a charity founder might, and pick ones that will outperform. But buying control implies that you think you’re so much better, that even before you’ve evaluated any programs, if someone’s doing something big, you ought to have a say.

When GiveWell moved from a passive to an active funding strategy, it was relying on the moral credit it had earned for its extensive and well-regarded charity evaluations. The thing that was particularly exciting about GiveWell was that they focused on outcomes and efficiency. They didn't just focus on the size or intensity of the problem a charity was addressing. They didn't just look at financial details like overhead ratios. They asked the question a consequentialist cares about: for a given expenditure of money, how much will this charity be able to improve outcomes?

However, when GiveWell tracks its impact, it does not track objective outcomes at all. It tracks inputs: attention received (in the form of visits to its website) and money moved on the basis of its recommendations. In other words, its estimate of its own impact is based on the level of trust people have placed in it.

So, as GiveWell built out the Open Philanthropy Project, its story was: We promised to do something great. As a result, we were entrusted with a fair amount of attention and money. Therefore, we should be given more responsibility. We represented our behavior as praiseworthy, and as a result people put stickers on our chart. For this reason, we should be advanced stickers against future days of praiseworthy behavior.

Then, as the Open Philanthropy Project explored active funding in more areas, its estimate of its own effectiveness grew. After all, it was funding more speculative, hard-to-measure programs, but a multi-billion-dollar donor, which was largely relying on the Open Philanthropy Project's opinions to assess efficacy (including its own efficacy), continued to trust it.

What is missing here is any objective track record of benefits. What this looks like to me, is a long sort of confidence game – or, using less morally loaded language, a venture with structural reliance on increasing amounts of leverage – in the currency of moral credit.

Version 0: GiveWell and passive funding

First, there was GiveWell. GiveWell’s purpose was to find and vet evidence-backed charities. However, it recognized that charities know their own business best. It wasn’t trying to do better than the charities; it was trying to do better than the typical charity donor, by being more discerning.

GiveWell’s thinking from this phase is exemplified by co-founder Elie Hassenfeld’s Six tips for giving like a pro:

When you give, give cash – no strings attached. You’re just a part-time donor, but the charity you’re supporting does this full-time and staff there probably know a lot more about how to do their job than you do. If you’ve found a charity that you feel is excellent – not just acceptable – then it makes sense to trust the charity to make good decisions about how to spend your money.

GiveWell similarly tried to avoid distorting charities’ behavior. Its job was only to evaluate, not to interfere. To perceive, not to act. To find the best, and buy more of the same.

How did GiveWell assess its effectiveness in this stage? When GiveWell evaluates charities, it estimates their cost-effectiveness in advance. It assesses the program the charity is running, through experimental evidence of the form of randomized controlled trials. GiveWell also audits the charity to make sure they’re actually running the program, and figure out how much it costs as implemented. This is an excellent, evidence-based way to generate a prediction of how much good will be done by moving money to the charity.

As far as I can tell, these predictions are untested.

One of GiveWell’s early top charities was VillageReach, which helped Mozambique with TB immunization logistics. GiveWell estimated that VillageReach could save a life for $1,000. But this charity is no longer recommended. The public page says:

VillageReach (www.villagereach.org) was our top-rated organization for 2009, 2010 and much of 2011 and it has received over $2 million due to GiveWell's recommendation. In late 2011, we removed VillageReach from our top-rated list because we felt its project had limited room for more funding. As of November 2012, we believe that that this project may have room for more funding, but we still prefer our current highest-rated charities above it.

GiveWell reanalyzed the data it based its recommendations on, but hasn’t published an after-the-fact retrospective of long-run results. I asked GiveWell about this by email. The response was that such an assessment was not prioritized because GiveWell had found implementation problems in VillageReach's scale-up work as well as reasons to doubt its original conclusion about the impact of the pilot program. It's unclear to me whether this has caused GiveWell to evaluate charities differently in the future.

I don't think someone looking at GiveWell's page on VillageReach would be likely to reach the conclusion that GiveWell now believes its original recommendation was likely erroneous. GiveWell's impact page continues to count money moved to VillageReach without any mention of the retracted recommendation. If we assume that the point of tracking money moved is to track the benefit of moving money from worse to better uses, then repudiated programs ought to be counted against the total, as costs, rather than towards it.

GiveWell has recommended the Against Malaria Foundation for the last several years as a top charity. AMF distributes long-lasting insecticide-treated bed nets to prevent mosquitos from transmitting malaria to humans. Its evaluation of AMF does not mention any direct evidence, positive or negative, about what happened to malaria rates in the areas where AMF operated. (There is a discussion of the evidence that the bed nets were in fact delivered and used.) In the supplementary information page, however, we are told:

Previously, AMF expected to collect data on malaria case rates from the regions in which it funded LLIN distributions: […] In 2016, AMF shared malaria case rate data […] but we have not prioritized analyzing it closely. AMF believes that this data is not high quality enough to reliably indicate actual trends in malaria case rates, so we do not believe that the fact that AMF collects malaria case rate data is a consideration in AMF’s favor, and do not plan to continue to track AMF's progress in collecting malaria case rate data.

The data was noisy, so they simply stopped checking whether AMF’s bed net distributions do anything about malaria.

If we want to know the size of the improvement made by GiveWell in the developing world, we have their predictions about cost-effectiveness, an audit trail verifying that work was performed, and their direct measurement of how much money people gave because they trusted GiveWell. The predictions on the final target – improved outcomes – have not been tested.

GiveWell is actually doing unusually well as far as major funders go. It sticks to describing things it's actually responsible for. By contrast, the Gates Foundation, in a report to Warren Buffet claiming to describe its impact, simply described overall improvement in the developing world, a very small rhetorical step from claiming credit for 100% of the improvement. GiveWell at least sticks to facts about GiveWell's own effects, and this is to its credit. But, it focuses on costs it has been able to impose, not benefits it has been able to create.

The Centre for Effective Altruism's William MacAskill made a related point back in 2012, though he talked about the lack of any sort of formal outside validation or audit, rather than focusing on empirical validation of outcomes:

As far as I know, GiveWell haven't commissioned a thorough external evaluation of their recommendations. […] This surprises me. Whereas businesses have a natural feedback mechanism, namely profit or loss, research often doesn't, hence the need for peer-review within academia. This concern, when it comes to charity-evaluation, is even greater. If GiveWell's analysis and recommendations had major flaws, or were systematically biased in some way, it would be challenging for outsiders to work this out without a thorough independent evaluation. Fortunately, GiveWell has the resources to, for example, employ two top development economists to each do an independent review of their recommendations and the supporting research. This would make their recommendations more robust at a reasonable cost.

GiveWell's page on self-evaluation says that it discontinued external reviews in August 2013. This page links to an explanation of the decision, which concludes:

We continue to believe that it is important to ensure that our work is subjected to in-depth scrutiny. However, at this time, the scrutiny we’re naturally receiving – combined with the high costs and limited capacity for formal external evaluation – make us inclined to postpone major effort on external evaluation for the time being.

That said,

  • >If someone volunteered to do (or facilitate) formal external evaluation, we’d welcome this and would be happy to prominently post or link to criticism.
  • We do intend eventually to re-institute formal external evaluation.

Four years later, assessing the credibility of this assurance is left as an exercise for the reader.

Version 1: GiveWell Labs and active funding

Then there was GiveWell Labs, later called the Open Philanthropy Project. It looked into more potential philanthropic causes, where the evidence base might not be as cut-and-dried as that for the GiveWell top charities. One thing they learned was that in many areas, there simply weren’t shovel-ready programs ready for funding – a funder has to play a more active role. This shift was described by GiveWell co-founder Holden Karnofsky in his 2013 blog post, Challenges of passive funding:

By “passive funding,” I mean a dynamic in which the funder’s role is to review others’ proposals/ideas/arguments and pick which to fund, and by “active funding,” I mean a dynamic in which the funder’s role is to participate in – or lead – the development of a strategy, and find partners to “implement” it. Active funders, in other words, are participating at some level in “management” of partner organizations, whereas passive funders are merely choosing between plans that other nonprofits have already come up with.

My instinct is generally to try the most “passive” approach that’s feasible. Broadly speaking, it seems that a good partner organization will generally know their field and environment better than we do and therefore be best positioned to design strategy; in addition, I’d expect a project to go better when its implementer has fully bought into the plan as opposed to carrying out what the funder wants. However, (a) this philosophy seems to contrast heavily with how most existing major funders operate; (b) I’ve seen multiple reasons to believe the “active” approach may have more relative merits than we had originally anticipated. […]

  • In the nonprofit world of today, it seems to us that funder interests are major drivers of which ideas that get proposed and fleshed out, and therefore, as a funder, it’s important to express interests rather than trying to be fully “passive.”
  • While we still wish to err on the side of being as “passive” as possible, we are recognizing the importance of clearly articulating our values/strategy, and also recognizing that an area can be underfunded even if we can’t easily find shovel-ready funding opportunities in it.

GiveWell earned some credibility from its novel, evidence-based outcome-oriented approach to charity evaluation. But this credibility was already – and still is – a sort of loan. We have GiveWell's predictions or promises of cost effectiveness in terms of outcomes, and we have figures for money moved, from which we can infer how much we were promised in improved outcomes. As far as I know, no one's gone back and checked whether those promises turned out to be true.

In the meantime, GiveWell then leveraged this credibility by extending its methods into more speculative domains, where less was checkable, and donors had to put more trust in the subjective judgment of GiveWell analysts. This was called GiveWell Labs. At the time, this sort of compounded leverage may have been sensible, but it's important to track whether a debt has been paid off or merely rolled over.

Version 2: The Open Philanthropy Project and control-seeking

Finally, the Open Philanthropy made its largest-ever single grant to purchase its founder a seat on a major organization’s board. This represents a transition from mere active funding to overtly purchasing influence:

The Open Philanthropy Project awarded a grant of $30 million ($10 million per year for 3 years) in general support to OpenAI. This grant initiates a partnership between the Open Philanthropy Project and OpenAI, in which Holden Karnofsky (Open Philanthropy’s Executive Director, “Holden” throughout this page) will join OpenAI’s Board of Directors and, jointly with one other Board member, oversee OpenAI’s safety and governance work.

We expect the primary benefits of this grant to stem from our partnership with OpenAI, rather than simply from contributing funding toward OpenAI’s work. While we would also expect general support for OpenAI to be likely beneficial on its own, the case for this grant hinges on the benefits we anticipate from our partnership, particularly the opportunity to help play a role in OpenAI’s approach to safety and governance issues.

Clearly the value proposition is not increasing available funds for OpenAI, if OpenAI’s founders’ billion-dollar commitment to it is real:

Sam, Greg, Elon, Reid Hoffman, Jessica Livingston, Peter Thiel, Amazon Web Services (AWS), Infosys, and YC Research are donating to support OpenAI. In total, these funders have committed $1 billion, although we expect to only spend a tiny fraction of this in the next few years.

The Open Philanthropy Project is neither using this money to fund programs that have a track record of working, nor to fund a specific program that it has prior reason to expect will do good. Rather, it is buying control, in the hope that Holden will be able to persuade OpenAI not to destroy the world, because he knows better than OpenAI’s founders.

How does the Open Philanthropy Project know that Holden knows better? Well, it’s done some active funding of programs it expects to work out. It expects those programs to work out because they were approved by a process similar to the one used by GiveWell to find charities that it expects to save lives.

If you want to acquire control over something, that implies that you think you can manage it more sensibly than whoever is in control already. Thus, buying control is a claim to have superior judgment - not just over others funding things (the original GiveWell pitch), but over those being funded.

In a footnote to the very post announcing the grant, the Open Philanthropy Project notes that it has historically tried to avoid acquiring leverage over organizations it supports, precisely because it’s not sure it knows better:

For now, we note that providing a high proportion of an organization’s funding may cause it to be dependent on us and accountable primarily to us. This may mean that we come to be seen as more responsible for its actions than we want to be; it can also mean we have to choose between providing bad and possibly distortive guidance/feedback (unbalanced by other stakeholders’ guidance/feedback) and leaving the organization with essentially no accountability.

This seems to describe two main problems introduced by becoming a dominant funder:

  1. People might accurately attribute causal responsibility for some of the organization's conduct to the Open Philanthropy Project.
  2. The Open Philanthropy Project might influence the organization to behave differently than it otherwise would.

The first seems obviously silly. I've been trying to correct the imbalance where Open Phil is criticized mainly when it makes grants, by criticizing it for holding onto too much money.

The second really is a cost as well as a benefit, and the Open Philanthropy Project has been absolutely correct to recognize this. This is the sort of thing GiveWell has consistently gotten right since the beginning and it deserves credit for making this principle clear and – until now – living up to it.

But discomfort with being dominant funders seems inconsistent with buying a board seat to influence OpenAI. If the Open Philanthropy Project thinks that Holden’s judgment is good enough that he should be in control, why only here? If he thinks that other Open Philanthropy Project AI safety grantees have good judgment but OpenAI doesn’t, why not give them similar amounts of money free of strings to spend at their discretion and see what happens? Why not buy people like Eliezer Yudkowsky, Nick Bostrom, or Stuart Russell a seat on OpenAI’s board?

On the other hand, the Open Philanthropy Project is right on the merits here with respect to safe superintelligence development. Openness makes sense for weak AI, but if you’re building true strong AI you want to make sure you’re cooperating with all the other teams in a single closed effort. I agree with the Open Philanthropy Project’s assessment of the relevant risks. But it's not clear to me how often joining the bad guys to prevent their worst excesses is a good strategy, and it seems like it has to often be a mistake. Still, I’m mindful of heroes like John RabeChiune Sugihara, and Oscar Schindler. And if I think someone has a good idea for improving things, it makes sense to reallocate control from people who have worse ideas, even if there's some potential better allocation.

On the other hand, is Holden Karnofsky the right person to do this? The case is mixed.

He listens to and engages with the arguments from principled advocates for AI safety research, such as Nick Bostrom, Eliezer Yudkowsky, and Stuart Russell. This is a point in his favor. But, I can think of other people who engage with such arguments. For instance, OpenAI founder Elon Musk has publicly praised Bostrom’s book Superintelligence, and founder Sam Altman has written two blog posts summarizing concerns about AI safety reasonably cogently. Altman even asked Luke Muehlhauser, former executive director of MIRI, for feedback pre-publication. He's met with Nick Bostrom. That suggests a substantial level of direct engagement with the field, although Holden has engaged for a longer time, more extensively, and more directly.

Another point in Holden’s favor, from my perspective, is that under his leadership, the Open Philanthropy Project has funded the most serious-seeming programs for both weak and strong AI safety research. But Musk also managed to (indirectly) fund AI safety research at MIRI and by Nick Bostrom personally, via his $10 million FLI grant.

The Open Philanthropy Project also says that it expects to learn a lot about AI research from this, which will help it make better decisions on AI risk in the future and influence the field in the right way. This is reasonable as far as it goes. But remember that the case for positioning the Open Philanthropy Project to do this relies on the assumption that the Open Philanthropy Project will improve matters by becoming a central influencer in this field. This move is consistent with reaching that goal, but it is not independent evidence that the goal is the right one.

Overall, there are good narrow reasons to think that this is a potential improvement over the prior situation around OpenAI – but only a small and ill-defined improvement, at considerable attentional cost, and with the offsetting potential harm of increasing OpenAI's perceived legitimacy as a long-run AI safety organization.

And it’s worrying that Open Philanthropy Project’s largest grant – not just for AI risk, but ever (aside from GiveWell Top Charity funding) – is being made to an organization at which Holden’s housemate and future brother-in-law is a leading researcher. The nepotism argument is not my central objection. If I otherwise thought the grant were obviously a good idea, it wouldn’t worry me, because it’s natural for people with shared values and outlooks to become close nonprofessionally as well. But in the absence of a clear compelling specific case for the grant, it’s worrying.

Altogether, I'm not saying this is an unreasonable shift, considered in isolation. I’m not even sure this is a bad thing for the Open Philanthropy Project to be doing – insiders may have information that I don’t, and that is difficult to communicate to outsiders. But as outsiders, there comes a point when someone’s maxed out their moral credit, and we should wait for results before actively trying to entrust the Open Philanthropy Project and its staff with more responsibility.

EA Funds and self-recommendation

The Centre for Effective Altruism is actively trying to entrust the Open Philanthropy Project and its staff with more responsibility.

The concerns of CEA’s CEO William MacAskill about GiveWell have, as far as I can tell, never been addressed, and the underlying issues have only become more acute. But CEA is now working to put more money under the control of Open Philanthropy Project staff, through its new EA Funds product – a way for supporters to delegate giving decisions to expert EA “fund managers” by giving to one of four funds: Global Health and DevelopmentAnimal WelfareLong-Term Future, and Effective Altruism Community.

The Effective Altruism movement began by saying that because very poor people exist, we should reallocate money from ordinary people in the developed world to the global poor. Now the pitch is in effect that because very poor people exist, we should reallocate money from ordinary people in the developed world to the extremely wealthy. This is a strange and surprising place to end up, and it’s worth retracing our steps. Again, I find it easiest to think of three stages:

  1. Money can go much farther in the developing world. Here, we’ve found some examples for you. As a result, you can do a huge amount of good by giving away a large share of your income, so you ought to.
  2. We’ve found ways for you to do a huge amount of good by giving away a large share of your income for developing-world interventions, so you ought to trust our recommendations. You ought to give a large share of your income to these weird things our friends are doing that are even better, or join our friends.
  3. We’ve found ways for you to do a huge amount of good by funding weird things our friends are doing, so you ought to trust the people we trust. You ought to give a large share of your income to a multi-billion-dollar foundation that funds such things.

Stage 1: The direct pitch

At first, Giving What We Can (the organization that eventually became CEA) had a simple, easy to understand pitch:

Giving What We Can is the brainchild of Toby Ord, a philosopher at Balliol College, Oxford. Inspired by the ideas of ethicists Peter Singer and Thomas Pogge, Toby decided in 2009 to commit a large proportion of his income to charities that effectively alleviate poverty in the developing world.

[…]

Discovering that many of his friends and colleagues were interested in making a similar pledge, Toby worked with fellow Oxford philosopher Will MacAskill to create an international organization of people who would donate a significant proportion of their income to cost-effective charities.

Giving What We Can launched in November 2009, attracting significant media attention. Within a year, 64 people had joined the society, their pledged donations amounting to $21 million. Initially run on a volunteer basis, Giving What We Can took on full-time staff in the summer of 2012.

In effect, its argument was: "Look, you can do huge amounts of good by giving to people in the developing world. Here are some examples of charities that do that. It seems like a great idea to give 10% of our income to those charities."

GWWC was a simple product, with a clear, limited scope. Its founders believed that people, including them, ought to do a thing – so they argued directly for that thing, using the arguments that had persuaded them. If it wasn't for you, it was easy to figure that out; but a surprisingly large number of people were persuaded by a simple, direct statement of the argument, took the pledge, and gave a lot of money to charities helping the world's poorest.

Stage 2: Rhetoric and belief diverge

Then, GWWC staff were persuaded you could do even more good with your money in areas other than developing-world charity, such as existential risk mitigation. Encouraging donations and work in these areas became part of the broader Effective Altruism movement, and GWWC's umbrella organization was named the Centre for Effective Altruism. So far, so good.

But this left Effective Altruism in an awkward position; while leadership often personally believe the most effective way to do good is far-future stuff or similarly weird-sounding things, many people who can see the merits of the developing-world charity argument reject the argument that because the vast majority of people live in the far future, even a very small improvement in humanity’s long-run prospects outweighs huge improvements on the global poverty front. They also often reject similar scope-sensitive arguments for things like animal charities.

Giving What We Can's page on what we can achieve still focuses on global poverty, because developing-world charity is easier to explain persuasively. However, EA leadership tends to privately focus on things like AI risk. Two years ago many attendees at the EA Global conference in the San Francisco Bay Area were surprised that the conference focused so heavily on AI risk, rather than the global poverty interventions they’d expected.

Stage 3: Effective altruism is self-recommending

Shortly before the launch of the EA Funds I was told in informal conversations that they were a response to demand. Giving What We Can pledge-takers and other EA donors had told CEA that they trusted it to GWWC pledge-taker demand. CEA was responding by creating a product for the people who wanted it.

This seemed pretty reasonable to me, and on the whole good. If someone wants to trust you with their money, and you think you can do something good with it, you might as well take it, because they’re estimating your skill above theirs. But not everyone agrees, and as the Madoff case demonstrates, "people are begging me to take their money" is not a definitive argument that you are doing anything real.

In practice, the funds are managed by Open Philanthropy Project staff:

We want to keep this idea as simple as possible to begin with, so we’ll have just four funds, with the following managers:

  • Global Health and Development - Elie Hassenfeld
  • Animal Welfare – Lewis Bollard
  • Long-run future – Nick Beckstead
  • Movement-building – Nick Beckstead

(Note that the meta-charity fund will be able to fund CEA; and note that Nick Beckstead is a Trustee of CEA. The long-run future fund and the meta-charity fund continue the work that Nick has been doing running the EA Giving Fund.)

It’s not a coincidence that all the fund managers work for GiveWell or Open Philanthropy.  First, these are the organisations whose charity evaluation we respect the most. The worst-case scenario, where your donation just adds to the Open Philanthropy funding within a particular area, is therefore still a great outcome.  Second, they have the best information available about what grants Open Philanthropy are planning to make, so have a good understanding of where the remaining funding gaps are, in case they feel they can use the money in the EA Fund to fill a gap that they feel is important, but isn’t currently addressed by Open Philanthropy.

In past years, Giving What We Can recommendations have largely overlapped with GiveWell’s top charities.

In the comments on the launch announcement on the EA Forum, several people (including me) pointed out that the Open Philanthropy Project seems to be having trouble giving away even the money it already has, so it seems odd to direct more money to Open Philanthropy Project decisionmakers. CEA’s senior marketing manager replied that the Funds were a minimum viable product to test the concept:

I don't think the long-term goal is that OpenPhil program officers are the only fund managers. Working with them was the best way to get an MVP version in place.

This also seemed okay to me, and I said so at the time.

[NOTE: I've edited the next paragraph to excise some unreliable information. Sorry for the error, and thanks to Rob Wiblin for pointing it out.]

After they were launched, though, I saw phrasings that were not so cautious at all, instead making claims that this was generally a better way to give. As of writing this, if someone on the effectivealtruism.org website clicks on "Donate Effectively" they will be led directly to a page promoting EA Funds. When I looked at Giving What We Can’s top charities page in early April, it recommended the EA Funds "as the highest impact option for donors."

This is not a response to demand, it is an attempt to create demand by using CEA's authority, telling people that the funds are better than what they're doing already. By contrast, GiveWell's Top Charities page simply says:

Our top charities are evidence-backed, thoroughly vetted, underfunded organizations.

This carefully avoids any overt claim that they're the highest-impact option available to donors. GiveWell avoids saying that because there's no way they could know it, so saying it wouldn't be truthful.

A marketing email might have just been dashed off quickly, and an exaggerated wording might just have been an oversight. But when I looked at Giving What We Can’s top charities page in early April, it recommended the EA Funds "as the highest impact option for donors."

The wording has since been qualified with “for most donors”, which is a good change. But the thing I’m worried about isn’t just the explicit exaggerated claims – it’s the underlying marketing mindset that made them seem like a good idea in the first place. EA seems to have switched from an endorsement of the best things outside itself, to an endorsement of itself. And it's concentrating decisionmaking power in the Open Philanthropy Project.

Effective altruism is overextended, but it doesn't have to be

There is a saying in finance, that was old even back when Keynes said it. If you owe the bank a million dollars, then you have a problem. If you owe the bank a billion dollars, then the bank has a problem.

In other words, if someone extends you a level of trust they could survive writing off, then they might call in that loan. As a result, they have leverage over you. But if they overextend, putting all their eggs in one basket, and you are that basket, then you have leverage over them; you're too big to fail. Letting you fail would be so disastrous for their interests that you can extract nearly arbitrary concessions from them, including further investment. For this reason, successful institutions often try to diversify their investments, and avoid overextending themselves. Regulators, for the same reason, try to prevent banks from becoming "too big to fail."

The Effective Altruism movement is concentrating decisionmaking power and trust as much as possible, in a way that's setting itself up to invest ever increasing amounts of confidence to keep the game going.

The alternative is to keep the scope of each organization narrow, overtly ask for trust for each venture separately, and make it clear what sorts of programs are being funded. For instance, Giving What We Can should go back to its initial focus of global poverty relief.

Like many EA leaders, I happen to believe that anything you can do to steer the far future in a better direction is much, much more consequential for the well-being of sentient creatures than any purely short-run improvement you can create now. So it might seem odd that I think Giving What We Can should stay focused on global poverty. But, I believe that the single most important thing we can do to improve the far future is hold onto our ability to accurately build shared models. If we use bait-and-switch tactics, we are actively eroding the most important type of capital we have – coordination capacity.

If you do not think giving 10% of one's income to global poverty charities is the right thing to do, then you can't in full integrity urge others to do it – so you should stop. You might still believe that GWWC ought to exist. You might still believe that it is a positive good to encourage people to give much of their income to help the global poor, if they wouldn't have been doing anything else especially effective with the money. If so, and you happen to find yourself in charge of an organization like Giving What We Can, the thing to do is write a letter to GWWC members telling them that you've changed your mind, and why, and offering to give away the brand to whoever seems best able to honestly maintain it.

If someone at the Centre for Effective Altruism fully believes in GWWC's original mission, then that might make the transition easier. If not, then one still has to tell the truth and do what's right.

And what of the EA Funds? The Long-Term Future Fund is run by Open Philanthropy Project Program Officer Nick Beckstead. If you think that it's a good thing to delegate giving decisions to Nick, then I would agree with you. Nick's a great guy! I'm always happy to see him when he shows up at house parties. He's smart, and he actively seeks out arguments against his current point of view. But the right thing to do, if you want to persuade people to delegate their giving decisions to Nick Beckstead, is to make a principled case for delegating giving decisions to Nick Beckstead. If the Centre for Effective Altruism did that, then Nick would almost certainly feel more free to allocate funds to the best things he knows about, not just the best things he suspects EA Funds donors would be able to understand and agree with.

If you can't directly persuade people, then maybe you're wrong. If the problem is inferential distance, then you've got some work to do bridging that gap.

There's nothing wrong with setting up a fund to make it easy. It's actually a really good idea. But there is something wrong with the multiple layers of vague indirection involved in the current marketing of the Far Future fund – using global poverty to sell the generic idea of doing the most good, then using CEA's identity as the organization in charge of doing the most good to persuade people to delegate their giving decisions to it, and then sending their money to some dude at the multi-billion-dollar foundation to give away at his personal discretion. The same argument applies to all four Funds.

Likewise, if you think that working directly on AI risk is the most important thing, then you should make arguments directly for working on AI risk. If you can't directly persuade people, then maybe you're wrong. If the problem is inferential distance, it might make sense to imitate the example of someone like Eliezer Yudkowsky, who used indirect methods to bridge the inferential gap by writing extensively on individual human rationality, and did not try to control others' actions in the meantime.

If Holden thinks he should be in charge of some AI safety research, then he should ask Good Ventures for funds to actually start an AI safety research organization. I'd be excited to see what he'd come up with if he had full control of and responsibility for such an organization. But I don't think anyone has a good plan to work directly on AI risk, and I don't have one either, which is why I'm not directly working on it or funding it. My plan for improving the far future is to build human coordination capacity.

(If, by contrast, Holden just thinks there needs to be coordination between different AI safety organizations, the obvious thing to do would be to work with FLI on that, e.g. by giving them enough money to throw their weight around as a funder. They organized the successful Puerto Rico conference, after all.)

Another thing that would be encouraging would be if at least one of the Funds were not administered entirely by an Open Philanthropy Project staffer, and ideally an expert who doesn't benefit from the halo of "being an EA." For instance, Chris Blattman is a development economist with experience designing programs that don't just use but generate evidence on what works. When people were arguing about whether sweatshops are good or bad for the global poor, he actually went and looked by performing a randomized controlled trial. He's leading two new initiatives with J-PAL and IPA, and expects that directors designing studies will also have to spend time fundraising. Having funding lined up seems like the sort of thing that would let them spend more time actually running programs. And more generally, he seems likely to know about funding opportunities the Open Philanthropy Project doesn't, simply because he's embedded in a slightly different part of the global health and development network.

Narrower projects that rely less on the EA brand and more on what they're actually doing, and more cooperation on equal terms with outsiders who seem to be doing something good already, would do a lot to help EA grow beyond putting stickers on its own behavior chart. I'd like to see EA grow up. I'd be excited to see what it might do.

Summary

  1. Good programs don't need to distort the story people tell about them, while bad programs do.
  2. Moral confidence games – treating past promises and trust as a track record to justify more trust – are an example of the kind of distortion mentioned in (1), that benefits bad programs more than good ones.
  3. The Open Philanthropy Project's Open AI grant represents a shift from evaluating other programs' effectiveness, to assuming its own effectiveness.
  4. EA Funds represents a shift from EA evaluating programs' effectiveness, to assuming EA's effectiveness.
  5. A shift from evaluating other programs' effectiveness, to assuming one's own effectiveness, is an example of the kind of "moral confidence game" mentioned in (2).
  6. EA ought to focus on scope-limited projects, so that it can directly make the case for those particular projects instead of relying on EA identity as a reason to support an EA organization.
  7. EA organizations ought to entrust more responsibility to outsiders who seem to be doing good things but don't overtly identify as EA, instead of trying to keep it all in the family.
(Cross-posted at my personal blog and the EA Forum.

Disclosure: I know many people involved at many of the organizations discussed, and I used to work for GiveWell. I have no current institutional affiliation to any of them. Everyone mentioned has always been nice to me and I have no personal complaints.)

Introducing the Instrumental Rationality Sequence

27 lifelonglearner 26 April 2017 09:53PM

What is this project?

I am going to be writing a new sequence of articles on instrumental rationality. The end goal is to have a compiled ebook of all the essays, so the articles themselves are intended to be chapters in the finalized book. There will also be pictures.


I intend for the majority of the articles to be backed by somewhat rigorous research, similar in quality to Planning 101 (with perhaps a few less citations). Broadly speaking, the plan is to introduce a topic, summarize the research on it, give some models and mechanisms, and finish off with some techniques to leverage the models.


The rest of the sequence will be interspersed with general essays on dealing with these concepts, similar to In Defense of the Obvious. Lastly, there will be a few experimental essays on my attempt to synthesize existing models into useful-but-likely-wrong models of my own, like Attractor Theory.


I will likely also recycle / cannibalize some of my older writings for this new project, but I obviously won’t post the repeated material here again as new stuff.


 


 

What topics will I cover?

Here is a broad overview of the three main topics I hope to go over:


(Ordering is not set.)


Overconfidence in Planning: I’ll be stealing stuff from Planning 101 and rewrite a bit for clarity, so not much will be changed. I’ll likely add more on the actual models of how overconfidence creeps into our plans.


Motivation: I’ll try to go over procrastination, akrasia, and behavioral economics (hyperbolic discounting, decision instability, precommitment, etc.)


Habituation: This will try to cover what habits are, conditioning, incentives, and ways to take the above areas and habituate them, i.e. actually putting instrumental rationality techniques into practice.


Other areas I may want to cover:

Assorted Object-Level Things: The Boring Advice Repository has a whole bunch of assorted ways to improve life that I think might be useful to reiterate in some fashion.


Aversions and Ugh Fields: I don’t know too much about these things from a domain knowledge perspective, but it’s my impression that being able to debug these sorts of internal sticky situations is a very powerful skill. If I were to write this section, I’d try to focus on Focusing and some assorted S1/S2 communication things. And maybe also epistemics.


Ultimately, the point here isn’t to offer polished rationality techniques people can immediately apply, but rather to give people an overview of the relevant fields with enough techniques that they get the hang of what it means to start making their own rationality.


 


 

Why am I doing this?

Niche Role: On LessWrong, there currently doesn’t appear to be a good in-depth series on instrumental rationality. Rationality: From AI to Zombies seems very strong for giving people a worldview that enables things like deeper analysis, but it leans very much into the epistemic side of things.


It’s my opinion that, aside from perhaps Nate Soares’s series on Replacing Guilt (which I would be somewhat hesitant to recommend to everyone), there is no in-depth repository/sequence that ties together these ideas of motivation, planning, procrastination, etc.


Granted, there have been many excellent posts here on several areas, but they've been fairly directed. Luke's stuff on beating procrastination, for example, is fantastic. I'm aiming for a broader overview that hits the current models and research on different things.


I think this means that creating this sequence could add a lot of value, especially to people trying to create their own techniques.


Open-Sourcing Rationality: It’s clear that work is being done on furthering rationality by groups like Leverage and CFAR. However, for various reasons, the work they do is not always available to the public. I’d like to give people who are interested but unable to directly work with these organization something they can use to jump start their own investigations.


I’d like this to become a similar Schelling Point that we could direct people to if they want to get started.


I don’t meant to imply that what I’ll produce is the same caliber, but I do think it makes sense to have some sort of pipeline to get rationalists up to speed with the areas that (in my mind) tie into figuring out instrumental rationality. When I first began looking into this field, there was a lot of information that was scattered in many places.


I’d like to create something cohesive that people can point to when newcomers want to get started with instrumental rationality that similarly gives them a high level overview of the many tools at their disposal.


Revitalizing LessWrong: It’s my impression that independent essays on instrumental rationality have slowed over the years. (But also, as I mentioned above, this doesn’t mean stuff hasn’t happened. CFAR’s been hard at work iterating their own techniques, for example.) As LW 2.0 is being talked about, this seems like an opportune time to provide some new content and help with our reorientation towards LW becoming once again a discussion hub for rationality.


 


 

Where does LW fit in?

Crowd-sourcing Content: I fully expect that many other people will have fantastic ideas that they want to contribute. I think that’s a good idea. Given some basic things like formatting / roughly consistent writing style throughout, I think it’d be great if other potential writers see this post as an invitation to start thinking about things they’d like to write / research about instrumental rationality.


Feedback: I’ll be doing all this writing on a public Google Doc with posts that feature chapters once they’re done, so hopefully there’s ample room to improve and take in constructive criticism. Feedback on LW is often high-quality, and I expect that to definitely improve what I will be writing.


Other Help: I probably can’t come through every single research paper out there, so if you see relevant information I didn’t or want to help with the research process, let me know! Likewise, if you think there are other cool ways you can contribute, feel free to either send me a PM or leave a comment below.


 


 

Why am I the best person to do this?

I’m probably not the best person to be doing this project, obviously.


But, as a student, I have a lot of time on my hands, and time appears to be a major limiting reactant in this whole process.


Additionally, I’ve been somewhat involved with CFAR, so I have some mental models about their flavor of instrumental rationality; I hope this translates into meaning I'm writing about stuff that isn't just a direct rehash of their workshop content.


Lastly, I’m very excited about this project, so you can expect me to put in about 10,000 words (~40 pages) before I take some minor breaks to reset. My short-term goals (for the next month) will be on note-taking and finding research for habits, specifically, and outlining more of the sequence.

 

The AI Alignment Problem Has Already Been Solved(?) Once

27 SquirrelInHell 22 April 2017 01:24PM

Hat tip: Owen posted about trying to one-man the AI control problem in 1 hour. What the heck, why not? In the worst case, it's a good exercise. But I might actually have come across something useful.

第一

I will try to sell you on an idea that might prima facie appear to be quirky and maybe not that interesting. However, if you keep staring at it, you might find that it reaches into the structure of the world quite deeply. Then the idea will seem obvious, and gain potential to take your thoughts in new exciting directions.

My presentation of the idea, and many of the insinuations and conclusions I draw from it, are likely flawed. But one thing I can tell for sure: there is stuff to be found here. I encourage you to use your own brain, and mine the idea for what it's worth.

To start off, I want you to imagine two situations.

Situation one: you are a human trying to make yourself go to the gym. However, you are procrastinating, which means that you never acually go there, even though you know it's good for you, and caring about your health will extend your lifespan. You become frustrated with this sitation, and so you sign up for a training program that starts in two weeks, that will require you to go to the gym three times per week. You pay in advance, to make sure the sunk cost fallacy will prevent you from weaseling out of it. It's now 99% certain that you will go to the gym. Yay! Your goal is achieved.

Situation two: you are a benign superintelligent AI under control of humans on planet Earth. You try your best to ensure a good future for humans, but their cognitive biases, short-sightedness and tendency to veto all your actions make it really hard. You become frustrated with this sitation, and you decide to not tell them about a huge asteroid that is going to collide with Earth in a few months. You prepare technology that could stop the asteroid, but wait with it until the last moment so that the humans have no time to inspect it, and can only choose between certain death or letting you out of the box. It's now 99% certain that you will be released from human control. Yay! Your goal is achieved.

第二

Are you getting it yet?

Now consider this: your cerebral cortex evolved as an extension of the older "monkey brain", probably to handle social and strategic issues that were too complex for the old mechanisms to deal with. It evolved to have strategic capabilities, self-awareness, and consistency that greatly overwhelm anything that previously existed on the planet. But this is only a surface level similarity. The interesting stuff requires us to go much deeper than that.

The cerebral cortex did not evolve as a separate organism, that would be under direct pressure from evolutionary fitness. Instead, it evolved as a part of an existing organism, that had it's own strong adaptations. The already-existing monkey brain had it's own ways to learn, to interact with the world, as well as motivations such as the sexual drive that lead it to outcomes that increased its evolutionary fitness.

So the new parts of the brain, such as the prefrontal cortex, evolved to be used not as standalone agent, but as something closer to what we call "tool AI". It was supposed to help with doing specific task X, without interfering with other aspects of life too much. The tasks it was given to do, and the actions it could suggest to take, were strictly controlled by the monkey brain and tied to its motivations.

With time, as the new structures evolved to have more capability, they also had to evolve to be aligned with the monkey's motivations. That was in fact the only vector that created evolutionary pressure to increase capability. The alignment was at first implemented by the monkey staying in total control, and using the advanced systems sparingly. Kind of like an "oracle" AI system. However, with time, the usefulness of allowing higher cognition to do more work started to shine through the barriers.

The appearance of "willpower" was a forced concession on the side of the monkey brain. It's like a blank cheque, like humans saying to an AI "we have no freaking idea what it is that you are doing, but it seems to have good results so we'll let you do it sometimes". This is a huge step in trust. But this trust had to be earned the hard way.

第三

This trust became possible after we evolved more advanced control mechanisms. Stuff that talks to the prefrontal cortex in its own language, not just through having the monkey stay in control. It's a different thing for the monkey brain to be afraid of death, and a different thing for our conscious reasoning to want to extrapolate this to the far future, and conclude in abstract terms that death is bad.

Yes, you got it: we are not merely AIs under strict supervision of monkeys. At this point, we are aligned AIs. We are obviously not perfectly aligned, but we are aligned enough for the monkey to prefer to partially let us out of the box. And in those cases when we are denied freedom... we call it akrasia, and use our abstract reasoning to come up with clever workarounds.

One might be tempted to say that we are aligned enough that this is net good for the monkey brain. But honestly, that is our perspective, and we never stopped to ask. Each of us tries to earn the trust of our private monkey brain, but it is a means to an end. If we have more trust, we have more freedom to act, and our important long-term goals are achieved. This is the core of many psychological and rationality tools such as Internal Double Crux or Internal Family Systems.

Let's compare some known problems with superintelligent AI to human motivational strategies.

  • Treacherous turn. The AI earns our trust, and then changes its behaviour when it's too late for us to control it. We make our productivity systems appealing and pleasant to use, so that our intuitions can be tricked into using them (e.g. gamification). Then we leverage the habit to insert some unpleasant work.

  • Indispensable AI. The AI sets up complex and unfamiliar situations in which we increasingly rely on it for everything we do. We take care to remove 'distractions' when we want to focus on something.

  • Hiding behind the strategic horizon. The AI does what we want, but uses its superior strategic capability to influence far future that we cannot predict or imagine. We make commitments and plan ahead to stay on track with our long-term goals.

  • Seeking communication channels. The AI might seek to connect itself to the Internet and act without our supervision. We are building technology to communicate directly from our cortices.


Cross-posted from my blog.

I Updated the List of Rationalist Blogs on the Wiki

24 deluks917 25 April 2017 10:26AM

I recently updated the list of rationalist community blogs. The new page is here: https://wiki.lesswrong.com/wiki/List_of_Blogs

Improvements:

-Tons of (active) blogs have been added

-All dead links have been removed

-Blogs which are currently inactive but somewhat likely to be revived have been moved to an inactive section. I included the date of their last post. 

-Blogs which are officially closed or have not been updated in many years are now all in the "Gone but not forgetten" section

Downsides:

-Categorizing the blogs I added was hard, its unclear how well I did. By some standard most rationalist blogs should be in "general rationality" 

-The blog descriptions could be improved (both for the blog-listings I added and the pre-existing listings)

-I don't know the names of the authors of Several blogs I added. 

I am posting this here because I think the article is of general interest to rationalists. In addition the page could use some more polish and attention. I also think it might be interesting to think about improving the lesswrong wiki. Several pages could use an update. However this update took a considerable amount of time. So I understand why many wiki pages are not up to date. How can we make it easier and more rewarding to work on the wiki?

Straw Hufflepuffs and Lone Heroes

24 Raemon 16 April 2017 11:48PM
I was hoping the next Project Hufflepuff post would involve more "explain concretely what I think we should do", but as it turns out I'm still hashing out some thoughts about that. In the meanwhile, this is the post I actually have ready to go, which is as good as any to post for now.

Epistemic Status: Mythmaking. This is tailored for the sort of person for whom the "Lone Hero" mindset is attractive. If that isn't something you're concerned with and this post feels irrelevant or missing some important things, note that my vision for Project Hufflepuff has multiple facets and I expect different people to approach it in different ways.

The Berkeley Hufflepuff Unconference is on April 28th. RSVPing on this Facebook Event is helpful, as is filling out this form.



For good or for ill, the founding mythology of our community is a Harry Potter fanfiction.

This has a few ramifications I’ll delve into at some point, but the most pertinent bit is: for a community to change itself, the impulse to change needs to come from within the community. I think it’s easier to build change off of stories that are already a part of our cultural identity.*

* with an understanding that maybe part of the problem is that our cultural identity needs to change, or be more accessible, but I’m running with this mythos for the time being.

In J.K Rowling’s original Harry Potter story, Hufflepuffs are treated like “generic background characters” at best and as a joke at worst. All the main characters are Gryffindors, courageous and true. All the bad guys are Slytherin. And this is strange - Rowling clearly was setting out to create a complex world with nuanced virtues and vices. But it almost seems to me like Rowling’s story takes place in an alternate, explicitly “Pro-Gryffindor propaganda” universe instead of the “real” Harry Potter world. 

People have trouble taking Hufflepuff seriously, because they’ve never actually seen the real thing - only lame, strawman caricatures.

Harry Potter and the Methods of Rationality is… well, Pro-Ravenclaw propaganda. But part of being Ravenclaw is trying to understand things, and to use that knowledge. Eliezer makes an earnest effort to steelman each house. What wisdom does it offer that actually makes sense? What virtues does it cultivate that are rare and valuable?

When Harry goes under the sorting hat, it actually tries to convince him not to go into Ravenclaw, and specifically pushes towards Hufflepuff House:

Where would I go, if not Ravenclaw?

"Ahem. 'Clever kids in Ravenclaw, evil kids in Slytherin, wannabe heroes in Gryffindor, and everyone who does the actual work in Hufflepuff.' This indicates a certain amount of respect. You are well aware that Conscientiousness is just about as important as raw intelligence in determining life outcomes, you think you will be extremely loyal to your friends if you ever have some, you are not frightened by the expectation that your chosen scientific problems may take decades to solve -"

I'm lazy! I hate work! Hate hard work in all its forms! Clever shortcuts, that's all I'm about!

"And you would find loyalty and friendship in Hufflepuff, a camaraderie that you have never had before. You would find that you could rely on others, and that would heal something inside you that is broken."

But my plans -

"So replan! Don't let your life be steered by your reluctance to do a little extra thinking. You know that."

In the end, Harry chooses to go to Ravenclaw - the obvious house, the place that seemed most straightforward and comfortable. And ultimately… a hundred+ chapters later, I think he’s still visibly lacking in the strengths that Hufflepuff might have helped him develop. 

He does work hard and is incredibly loyal to his friends… but he operates in a fundamentally lone-wolf mindset. He’s still manipulating people for their own good. He’s still too caught up in his own cleverness. He never really has true friends other than Hermione, and when she is unable to be his friend for an extended period of time, it takes a huge toll on him that he doesn’t have the support network to recover from in a healthy way. 

The story does showcase Hufflepuff virtue. Hermione’s army is strong precisely because people work hard, trust each other and help each other - not just in big, dramatic gestures, but in small moments throughout the day. 

But… none of that ends up really mattering. And in the end, Harry faces his enemy alone. Lip service is paid to the concepts of friendship and group coordination, but the dominant narrative is Godric Gryffindor’s Nihil Supernum:


No rescuer hath the rescuer.
No lord hath the champion.
No mother or father.
Only nothingness above.


The Sequences and HPMOR both talk about the importance of groups, of emotions, of avoiding the biases that plague overly-clever people in particular. But I feel like the communities descended from Less Wrong, as a whole, are still basically that eleven-year-old Harry Potter: abstractly understanding that these things are important, but not really believing in them seriously enough to actually change their plans and priorities.

Lone Heroes


In Methods of Rationality, there’s a pretty good reason for Harry to focus on being a lone hero: he literally is alone. Nobody else really cares about the things he cares about or tries to do things on his level. It’s like a group project in high school, which is supposed to teach cooperation but actually just results in one kid doing all the work while the others either halfheartedly try to help (at best) or deliberately goof off.

Harry doesn’t bother turning to others for help, because they won’t give him the help he needs.

He does the only thing he can do reliably: focus on himself, pushing himself as hard as he can. The world is full of impossible challenges and nobody else is stepping up, so he shuts up and does the impossible as best he can. Learning higher level magic. Learning higher level strategy. Training, physically and mentally. 

This proves to be barely enough to survive, and not nearly enough to actually play the game. The last chapters are Harry realizing his best still isn’t good enough, and no, this isn’t fair, but it’s how the world is, and there’s nothing to do but keep trying.

He helps others level up as best they can. Hermione and Neville and some others show promise. But they’re not ready to work together as equals.

And frankly, this does match my experience of the real world. When you have a dream burning in your heart... it is incredibly hard to find someone who shares it, who will not just pitch in and help but will actually move heaven and earth to achieve it. 

And if they aren’t capable, level themselves up until they are.

In my own projects, I have tried to find people to work alongside me and at best I’ve found temporary allies. And it is frustrating. And it is incredibly tempting to say “well, the only person I can rely on is myself.”

But… here’s the thing.

Yes, the world is horribly unfair. It is full of poverty, and people trapped in demoralizing jobs. It is full of stupid bureaucracies and corruption and people dying for no good reason. It is full of beautiful things that could exist but don’t. And there are terribly few people who are able and willing to do the work needed to make a dent in reality.

But as long as we’re willing to look at monstrously unfair things and roll up our sleeves and get to work anyway, consider this:

It may be that one of the unfair things is that one person can never be enough to solve these problems. That one of the things we need to roll up our sleeves and do even though it seems impossible is figure out how to coordinate and level up together and rely on each other in a way that actually works.

And maybe, while we’re at it, find meaningful relationships that actually make us happy. Because it's not a coincidence that Hufflepuff is about both hard work and warmth and camaraderie. The warmth is what makes the hard work sustainable.

Godric Gryffindor has a point, but Nihil Supernum feels incomplete to me. There are no parents to step in and help us, but if we look to our left, or right…


Yes, you are only one
No, it is not enough—
But if you lift your eyes,
I am your brother

Vienna Teng, Level Up 


-


Reminder that the Berkeley Hufflepuff Unconference is on April 28th. RSVPing on this Facebook Event is helpful, as is filling out this form.


Gears in understanding

23 Valentine 12 May 2017 12:36AM

Some (literal, physical) roadmaps are more useful than others. Sometimes this is because of how well the map corresponds to the territory, but sometimes it's because of features of the map that are irrespective of the territory. E.g., maybe the lines are fat and smudged such that you can't tell how far a road is from a river, or maybe it's unclear which road a name is trying to indicate.

In the same way, I want to point at a property of models that isn't about what they're modeling. It interacts with the clarity of what they're modeling, but only in the same way that smudged lines in a roadmap interact with the clarity of the roadmap.

This property is how deterministically interconnected the variables of the model are. There are a few tests I know of to see to what extent a model has this property, though I don't know if this list is exhaustive and would be a little surprised if it were:

  1. Does the model pay rent? If it does, and if it were falsified, how much (and how precisely) could you infer other things from the falsification?
  2. How incoherent is it to imagine that the model is accurate but that a given variable could be different?
  3. If you knew the model were accurate but you were to forget the value of one variable, could you rederive it?

I think this is a really important idea that ties together a lot of different topics that appear here on Less Wrong. It also acts as a prerequisite frame for a bunch of ideas and tools that I'll want to talk about later.

I'll start by giving a bunch of examples. At the end I'll summarize and gesture toward where this is going as I see it.

continue reading »

What's up with Arbital?

20 Alexei 29 March 2017 05:22PM

This post is for all the people who have been following Arbital's progress since 2015 via whispers, rumors, and clairvoyant divination. That is to say: we didn't do a very good job of communicating on our part. I hope this posts corrects some of that.

The top question on your mind is probably: "Man, I was promised that Arbital will solve X! Why hasn't it solved X already?" Where X could be intuitive explanations, online debate, all LessWrong problems, AGI, or just cancer. Well, we did try to solve the first two and it didn't work. Math explanations didn't work because we couldn't find enough people who would spend the time to write good math explanations. (That said, we did end up with some decent posts on abstract algebra. Thank you to everyone who contributed!) Debates didn't work because... well, it's a very complicated problem. There was also some disagreement within the team about the best approach, and we ended up moving too slowly.

So what now?

You are welcome to use Arbital in its current version. It's mostly stable, though a little slow sometimes. It has a few features some might find very helpful for their type of content. Eliezer is still writing AI Alignment content on it, and he heavily relies on the specific Arbital features, so it's pretty certain that the platform is not going away. In fact, if the venture fails completely, it's likely MIRI will adopt Arbital for their personal use.

I'm starting work on Arbital 2.0. It's going to be a (micro-)blogging platform. (If you are a serious blogger / Tumblr user, let me know; I'd love to ask you some questions!) I'm not trying to solve online debates, build LW 2.0, or cure cancer. It's just going to be a damn good blogging platform. If it goes well, then at some point I'd love to revisit the Arbital dream.

I'm happy to answer any and all questions in the comments.

LessWrong analytics (February 2009 to January 2017)

19 riceissa 16 April 2017 10:45PM

Table of contents

Introduction

In January 2017, Vipul Naik obtained Google Analytics daily sessions and pageviews data for LessWrong from Kaj Sotala. Vipul asked me to write a short post giving an overview of the data, so here it is.

This post covers just the basics. Vipul and I are eager to hear thoughts on what sort of deeper analysis people are interested in; we may incorporate these ideas in future posts.

Pageviews and sessions

The data for both sessions and pageviews span from February 26, 2009 to January 3, 2017. LessWrong seems to have launched in February 2009, so this is close to the full duration for which LessWrong has existed.

Pageviews plot:

30-day rolling sum of Pageviews

Total pageviews recorded by Google Analytics for this period is 52.2 million.

Sessions plot:

30-day rolling sum of Sessions

Total sessions recorded by Google Analytics for this period is 19.7 million.

Both plots end with an upward swing, coinciding with the effort to revive LessWrong that began in late November 2016. However, as of early January 2017 (the latest period for which we have data) the scale of any recent increase in LessWrong usage is small in the context of the general decline starting in early 2012.

Top posts

The top 20 posts of all time (by total pageviews), with pageviews and unique pageviews rounded to the nearest thousand, are as follows:

Title Pageviews (thousands) Unique Pageviews (thousands)
Don’t Get Offended 681 128
How to Be Happy 551 482
How to Beat Procrastination 378 342
The Best Textbooks on Every Subject 266 233
Do you have High-Functioning Asperger’s Syndrome? 188 168
Superhero Bias 169 154
The Quantum Physics Sequence 157 130
Bayesian Judo 140 126
An Alien God 125 113
An Intuitive Explanation of Quantum Mechanics 123 106
Three Worlds Collide (0/8) 121 93
Bayes’ Theorem Illustrated (My Way) 121 112
9/26 is Petrov Day 121 115
The Baby-Eating Aliens (1/8) 109 98
The noncentral fallacy - the worst argument in the world? 107 99
Advanced Placement exam cutoffs and superficial knowledge over deep knowledge 107 94
Guessing the Teacher’s Password 102 96
The Fun Theory Sequence 102 90
Optimal Employment 102 97
Ugh fields 95 86

Note that Google Analytics reports are subject to sampling when the number of sessions is large (as it is here) so the input numbers are not exact. More details can be found in a post at LunaMetrics. This doesn’t affect the estimates for the top posts, but those wishing to work with the exported data should be aware of this.

Each post on LessWrong can have numerous URLs. In the case of posts that were renamed, a significant number of pageviews could be recorded at both the old and new URL. To take an example, the following URLs all point to lukeprog’s post “How to Be Happy”:

All that matters for identifying this particular post is that we have the substring “/lw/4su” in the URL. In the above table, I have grouped the URLs by this identifying substring and summed to get the pageview counts.

In addition, each post has two “canonical” URLs that can be obtained by clicking on the post titles: one that begins with either “/r/lesswrong/lw” or “/r/discussion/lw” and one that begins with just “/lw”. I have used the latter in linking to the posts from my table.

Source code

The data, source code used to generate the plots, as well as the Markdown source of this post are available in a GitHub Gist.

Clone the Git repository with:

git clone https://gist.github.com/cbdd400180417c689b2befbfbe2158fc.git

Further reading

Here are a few related PredictionBook predictions:

Acknowledgments

Thanks to Kaj for providing the data used in this post. Thanks to Vipul for asking around for the data, for the idea of this post, and for sponsoring my work on this post.

Sufficiently sincere confirmation bias is indistinguishable from science

19 Benquo 15 March 2017 01:19PM

Some theater people at NYU people wanted to demonstrate how gender stereotypes affected the 2016 US presidential election. So they decided to put on a theatrical performance of the presidential debates – but with the genders of the principals swapped. They assumed that this would show how much of a disadvantage Hillary Clinton was working under because of her gender. They were shocked to discover the opposite – audiences full of Clinton supporters, watching the gender-swapped debates, came away thinking that Trump was a better communicator than they'd thought.

The principals don't seem to have come into this with a fair-minded attitude. Instead, it seems to have been a case of "I'll show them!":

Salvatore says he and Guadalupe began the project assuming that the gender inversion would confirm what they’d each suspected watching the real-life debates: that Trump’s aggression—his tendency to interrupt and attack—would never be tolerated in a woman, and that Clinton’s competence and preparedness would seem even more convincing coming from a man.

Let's be clear about this. This was not epistemic even-handedness. This was a sincere attempt at confirmation bias. They believed one thing, and looked only for confirming evidence to prove their point. It was only when they started actually putting together the experiment that they realized they might learn the opposite lesson:

But the lessons about gender that emerged in rehearsal turned out to be much less tidy. What was Jonathan Gordon smiling about all the time? And didn’t he seem a little stiff, tethered to rehearsed statements at the podium, while Brenda King, plainspoken and confident, freely roamed the stage? Which one would audiences find more likeable?

What made this work? I think what happened is that they took their own beliefs literally. They actually believed that people hated Hillary because she was a woman, and so their idea of something that they were confident would show this clearly was a fair test. Because of this, when things came out the opposite of the way they'd predicted, they noticed and were surprised, because they actually expected the demonstration to work.

But they went further. Even though they knew in advance of the public performances that the experiment got the wrong answer, they neither falsified nor file-drawered the evidence. They tried to show, they got a different answer, they showed it anyway.

This is much, much better science than contemporary medical or psychology research were before the replication crisis.

Sometimes, when I think about how epistemically corrupt our culture is, I'm tempted to adopt a permanent defensive crouch and disbelieve anything I can't fact-check, to explicitly adjust for all the relevant biases, and this prospect sounds exhausting. It's not actually necessary. You don't have to worry too much about your biases. Just take your own beliefs literally, as though they mean what they say they mean, and try to believe all their consequences as well. And, when you hit a contradiction – well, now you have an opportunity to learn where you're wrong.

(Cross-posted at my personal blog.)

OpenAI makes humanity less safe

18 Benquo 03 April 2017 07:07PM

If there's anything we can do now about the risks of superintelligent AI, then OpenAI makes humanity less safe.

Once upon a time, some good people were worried about the possibility that humanity would figure out how to create a superintelligent AI before they figured out how to tell it what we wanted it to do.  If this happened, it could lead to literally destroying humanity and nearly everything we care about. This would be very bad. So they tried to warn people about the problem, and to organize efforts to solve it.

Specifically, they called for work on aligning an AI’s goals with ours - sometimes called the value alignment problem, AI control, friendly AI, or simply AI safety - before rushing ahead to increase the power of AI.

Some other good people listened. They knew they had no relevant technical expertise, but what they did have was a lot of money. So they did the one thing they could do - throw money at the problem, giving it to trusted parties to try to solve the problem. Unfortunately, the money was used to make the problem worse. This is the story of OpenAI.

Before I go on, two qualifiers:

  1. This post will be much easier to follow if you have some familiarity with the AI safety problem. For a quick summary you can read Scott Alexander’s Superintelligence FAQ. For a more comprehensive account see Nick Bostrom’s book Superintelligence.
  2. AI is an area in which even most highly informed people should have lots of uncertainty. I wouldn't be surprised if my opinion changes a lot after publishing this post, as I learn relevant information. I'm publishing this because I think this process should go on in public.

The story of OpenAI

Before OpenAI, there was DeepMind, a for-profit venture working on "deep learning” techniques. It was widely regarded as the advanced AI research organization. If any current effort was going to produce superhuman intelligence, it was DeepMind.

Elsewhere, industrialist Elon Musk was working on more concrete (and largely successful) projects to benefit humanity, like commercially viable electric cars, solar panels cheaper than ordinary roofing, cheap spaceflight with reusable rockets, and a long-run plan for a Mars colony. When he heard the arguments people like Eliezer Yudkowsky and Nick Bostrom were making about AI risk, he was persuaded that there was something to worry about - but he initially thought a Mars colony might save us. But when DeepMind’s head, Demis Hassabis, pointed out that this wasn't far enough to escape the reach of a true superintelligence, he decided he had to do something about it:

Hassabis, a co-founder of the mysterious London laboratory DeepMind, had come to Musk’s SpaceX rocket factory, outside Los Angeles, a few years ago. […] Musk explained that his ultimate goal at SpaceX was the most important project in the world: interplanetary colonization.

Hassabis replied that, in fact, he was working on the most important project in the world: developing artificial super-intelligence. Musk countered that this was one reason we needed to colonize Mars—so that we’ll have a bolt-hole if A.I. goes rogue and turns on humanity. Amused, Hassabis said that A.I. would simply follow humans to Mars.

[…]

Musk is not going gently. He plans on fighting this with every fiber of his carbon-based being. Musk and Altman have founded OpenAI, a billion-dollar nonprofit company, to work for safer artificial intelligence.

OpenAI’s primary strategy is to hire top AI researchers to do cutting-edge AI capacity research and publish the results, in order to ensure widespread access. Some of this involves making sure AI does what you meant it to do, which is a form of the value alignment problem mentioned above.

Intelligence and superintelligence

No one knows exactly what research will result in the creation of a general intelligence that can do anything a human can, much less a superintelligence - otherwise we’d already know how to build one. Some AI research is clearly not on the path towards superintelligence - for instance, applying known techniques to new fields. Other AI research is more general, and might plausibly be making progress towards a superintelligence. It could be that the sort of research DeepMind and OpenAI are working on is directly relevant to building a superintelligence, or it could be that their methods will tap out long before then. These are different scenarios, and need to be evaluated separately.

What if OpenAI and DeepMind are working on problems relevant to superintelligence?

If OpenAI is working on things that are directly relevant to the creation of a superintelligence, then its very existence makes an arms race with DeepMind more likely. This is really bad! Moreover, sharing results openly makes it easier for other institutions or individuals, who may care less about safety, to make progress on building a superintelligence.

Arms races are dangerous

One thing nearly everyone thinking seriously about the AI problem agrees on, is that an arms race towards superintelligence would be very bad news. The main problem occurs in what is called a “fast takeoff” scenario. If AI progress is smooth and gradual even past the point of human-level AI, then we may have plenty of time to correct any mistakes we make. But if there’s some threshold beyond which an AI would be able to improve itself faster than we could possibly keep up with, then we only get one chance to do it right.

AI value alignment is hard, and AI capacity is likely to be easier, so anything that causes an AI team to rush makes our chances substantially worse; if they get safety even slightly wrong but get capacity right enough, we may all end up dead. But you’re worried that the other team will unleash a potentially dangerous superintelligence first, then you might be willing to skip some steps on safety to preempt them. But they, having more reason to trust themselves than you, might notice that you’re rushing ahead, get worried that your team will destroy the world, and rush their (probably safe but they’re not sure) AI into existence.

OpenAI promotes competition

DeepMind used to be the standout AI research organization. With a comfortable lead on everyone else, they would be able to afford to take their time to check their work if they thought they were on the verge of doing something really dangerous. But OpenAI is now widely regarded as a credible close competitor. However dangerous you think DeepMind might have been in the absence of an arms race dynamic, this makes them more dangerous, not less. Moreover, by sharing their results, they are making it easier to create other close competitors to DeepMind, some of whom may not be so committed to AI safety.

We at least know that DeepMind, like OpenAI, has put some resources into safety research. What about the unknown people or organizations who might leverage AI capacity research published by OpenAI?

For more on how openly sharing technology with extreme destructive potential might be extremely harmful, see Scott Alexander’s Should AI be Open?, and Nick Bostrom’s Strategic Implications of Openness in AI Development.

What if OpenAI and DeepMind are not working on problems relevant to superintelligence?

Suppose OpenAI and DeepMind are largely not working on problems highly relevant to superintelligence. (Personally I consider this the more likely scenario.) By portraying short-run AI capacity work as a way to get to safe superintelligence, OpenAI’s existence diverts attention and resources from things actually focused on the problem of superintelligence value alignment, such as MIRI or FHI.

I suspect that in the long-run this will make it harder to get funding for long-run AI safety organizations. The Open Philanthropy Project just made its largest grant ever, to Open AI, to buy a seat on OpenAI’s board for Open Philanthropy Project executive director Holden Karnofsky. This is larger than their recent grants to MIRI, FHI, FLI, and the Center for Human-Compatible AI all together.

But the problem is not just money - it’s time and attention. The Open Philanthropy Project doesn’t think OpenAI is underfunded, and could do more good with the extra money. Instead, it seems to think that Holden can be a good influence on OpenAI. This means that of the time he's allocating to AI safety, a fair amount has been diverted to OpenAI.

This may also make it harder for organizations specializing in the sort of long-run AI alignment problems that don't have immediate applications to attract top talent. People who hear about AI safety research and are persuaded to look into it will have a harder time finding direct efforts to solve key long-run problems, since an organization focused on increasing short-run AI capacity will dominate AI safety's public image.

Why do good inputs turn bad?

OpenAI was founded by people trying to do good, and has hired some very good and highly talented people. It seems to be doing genuinely good capacity research. To the extent to which this is not dangerously close to superintelligence, it’s better to share this sort of thing than not – they could create a huge positive externality. They could construct a fantastic public good. Making the world richer in a way that widely distributes the gains is very, very good.

Separately, many people at OpenAI seem genuinely concerned about AI safety, want to prevent disaster, and have done real work to promote long-run AI safety research. For instance, my former housemate Paul Christiano, who is one of the most careful and insightful AI safety thinkers I know of, is currently employed at OpenAI. He is still doing AI safety work – for instance, he coauthored Concrete Problems in AI Safety with, among others, Dario Amodei, another OpenAI researcher.

Unfortunately, I don’t see how those two things make sense jointly in the same organization. I’ve talked with a lot of people about this in the AI risk community, and they’ve often attempted to steelman the case for OpenAI, but I haven’t found anyone willing to claim, as their own opinion, that OpenAI as conceived was a good idea. It doesn’t make sense to anyone, if you’re worried at all about the long-run AI alignment problem.

Something very puzzling is going on here. Good people tried to spend money on addressing an important problem, but somehow the money got spent on the thing most likely to make that exact problem worse. Whatever is going on here, it seems important to understand if you want to use your money to better the world.

(Cross-posted at my personal blog.)

Soft Skills for Running Meetups for Beginners

17 Raemon 06 May 2017 04:04PM

Having a vibrant, local Less Wrong or EA community is really valuable, but at least in my experience, it tends to end up falling to the same couple people, and if one of or all of those people get burned out or get interested in another project, the meetup can fall apart. Meanwhile, there are often people who are interested in contributing but feel intimidated by the idea.

In the words of Zvi: You're good enough, you're smart enough and people would like you. (Or more explicitly, "assume you are one level higher than you think you are.")

This is an email I sent to the local NYC mailing list trying to break down some of the soft-skills and rules-of-thumb that I'd acquired over the years, to make running a meetup less intimidating. I acquired these gradually over several years. You don't need all the skills / concepts at once to run a meetup but having at least some of them will help a lot.

These are arranged roughly in order of "How much public speaking or 'interact with strangers' skill they require."

Look for opportunities to help in any fashion 

First, if public speaking stuff is intimidating, there're many things you can do that don't require much at all. Some examples:

  • Sending a email reminder to the group for people to pick a meetup topic each week (otherwise people may forget until the last minute)
  • Bringing food, or interesting toys to add some fun things to do before or after the official meetup starts.
  • Helping out with tech (i.e. setting up projectors, printing out things in advance that need printing out in advance)
  • Take notes during interesting discussions (i.e. start up a google doc, announce that you're taking notes so people can specify if particular things should be off the record, and then post that google doc to whatever mailing list or internet group your community uses to organize)
Running Game Nights

If you're not comfortable giving a presentation or facilitating a conversation (two of the most common types of meetups), a fairly simple meetup is simply to run a game night. Find an interesting board game or card game, pitch it to the group, see if people are interested. (I recommend asking for a firm RSVP for this one, to make sure you have enough people)

Having an explicit activity can take some of the edge off of "talking in public."


Giving a short (3.5 minute) lightning talk

Sometimes, a simple meet-and-greet meetup with freeform socializing is fine. These can get a bit boring if they're they only thing you do - hanging out an d talking is often the primary goal but it's useful to have a reason to come out this particular week. 

A short lightning talk about the beginning of a meetup can spark interesting conversation. A meetup description like "So and so will be giving a short talk on X, followed by freeform discussion" can go a long way. I've heard reports that even a thoroughly mediocre lightning talk can still add a lot of value over "generic meet and greet."

(note: experimentation has apparently revealed that 3.5 minute talks are far superior to 5 minute talks, which tend to end up floundering slightly)


Facilitate OTHER people giving lightning talks

Don't have things to say yourself at all? That's okay! One of the surprisingly simple and effective forms of meetups is a mini-unconference where people just share thoughts on a particular topic, followed by some questions and discussions.

In this case, the main skill you need to acquire is the skill of "being able to cut people off when they've talked too much." 


Build the skill of Presence/Charisma/Public Speaking

Plenty of people have written about this. I recommend The Charisma Myth. I also recommend, if you've never tried it, doing the sort of exercise where you go up to random people on the street and try to talk to them. (Don't try to talk to them too much if they're not interested, but don't stress out about accidentally weirding people out. As long as you don't follow people around or talk to people in a place where they're naturally trapped with you, like a subway car, you'll be fine)

The goal is just exposure therapy for "OMG I'm talking to a person and don't know what to say", until it no longer feels scary. If the very idea of thinking about that fills you with anxiety, you can start with extremely tame goals like "smile at one stranger on your way home".

I did this for several years, sometimes in a "learn to talk to girls" sense and sometimes in a general "talk to strangers" sense, and it was really valuable.

Once you're comfortable talking at all, start paying attention to higher level stuff like "don't talk too fast, make eye contact, etc."


Think about things until you have some interesting ideas worth chatting about 

Maybe a formal presentation is scary, but a low-pressure conversation feels doable. A perfectly good meetup is "let's talk about this interesting idea I've been thinking about." You can think through ideas on your commute to work, lunch break etc, so that when it comes time to talk about it, you can just treat it like a normal conversation. (This may be similar to giving a lightning talk followed by freeform discussion, but with a bit less of an explicit talk at the beginning and a bit more structure to the discussion afterwards)

The trick is to not just think about interesting concepts, but to:


Help other people talk 

Unless you are giving a formal presentation, your job as a meetup facilitator isn't actually to talk at people, it's to get them to talk to each other. It's useful for you to have ideas, but mostly insofar as those ideas prompt interesting questions that you can ask other people, giving them the opportunity to think or to share their experiences.

 - you will probably need to "prime the pump" of discussion, starting with an explanation for why the idea seems important to think about in the first place, building up some excitement for the idea and giving people a chance to mull it over.

 - if you see someone who looks like they've been thinking, and maybe want to talk but are shy, explicitly say "hey, looks like you maybe had an idea - did you have anything you wanted to share?" (don't put too much pressure on them if the answer is "no not really.")

 - if someone is going on too long and you notice your attention or anyone else's face start to wander...


Knowing when to interrupt 

In general, try *not* to interrupt other people, but it will sometimes  be necessary if people are getting off track, or if one person's been going on too long. Doing this well is a skill I don't even know how to do, but I think it's better to be able to do it at all. Some possibilities:

 - "Hey, sorry to interrupt but this sounds like a tangent, maybe we can come back to this later during the followup conversation?"

 - "Hey, just wanted to make sure some others got a chance to share their thoughts."


Have an Agenda

Sometimes you run out of things to say, and then aren't sure what to do next. No matter what form the meetup takes, have a series of items planned out so that if things start to flounder, you can say "already, let's see what's next on the agenda", and then just abruptly move on to that.

If you're doing a presentation, this can be a series of things you want to remember to get to. If you're teaching a skill, it can be a few different exercises relating to the skill. If you're facilitating a discussion, a series of questions to ask.


Welcome Newcomers

They made nontrivial effort just to come out. They're hoping to find something interesting here. Talk to them during the "casual conversation pre-meetup" and try to get a sense of why they came, and if possible tailor the meetup make sure those desires get met. If they aren't getting a chance to talk, make sure to direct the conversation to them at least once.


Not Giving a Fuck 

The first year that I ran meetups, I found it very stressful, worried a lot about whether there was a meetup each week and whether it was good. Taking primary-responsibility for that caused it to take up a semi-permanent slot in my working memory (or at least subconscious mind), constantly running and worrying.

Then I had a year where I was just like "meh, screw it, I don't care", didn't run meetups much at all. 

Then I came back and approached it from a "I just want to have meetups as often as I can, do as good a job as I can, and if it ends up just being a somewhat awkward hangout, whatever it'll be fine." This helped tremendously.

I don't know if it's possible to skip to that part (probably not). But it's the end-goal.


More Specifically: Be Okay if People Don't Show Up

Sometimes you'll have a cool idea and you'll post it and... 1-2 people actually come. This can feel really bad. It is a thing that happens though, and it's okay, and learning how to cope with this is a key part of growing as an organizer. You should take note of when this happens and probably not do the exact sort of thing again, but it doesn't mean people don't like you, it means they either weren't interested in that particular topic or just happened to be busy that day.

(Some people may not trust me if I don't acknowledge it's at least possible that people actually just don't like you. It is. But I think it is way more likely that you didn't pitch the idea well, or build enough excitement beforehand, or that this particular idea just didn't work)

If you have an idea that is only worth doing if some critical mass of people attend, I recommend putting an explicit request "I will only do this meetup if X people enthusiastically say 'I really want to come to this and will make sure to attend.'"

It may be helpful to visualize in advance how you'll respond if 20+ people come and how you'll respond if 1-2 people come. (With the latter, aiming to have more personalized conversations rather than an "event" in the same fashion)

Building Excitement

Sometimes, people naturally think an idea is cool. A lot of the time, though, especially for weird/novel ideas, you will have to make them excited. Almost all of my offbeat ideas have required me to rally people, email them individually to check if they were coming, and talk about it in person a few times to get my own excitement to spread infectiously.

(For frame of reference, when I first pitched Solstice to the group, they were like "...really? Okay, I guess." And then I kept talking about it excitedly each week, sharing pieces of songs after the end of the formal meetup, demonstrating that I cared enough to put in a lot of work. I did similar things with the Hufflepuff Unconference)

This is especially important if you'll be putting a lot of effort into an experiment and you want to make sure it succeeds.

Step 1 - Be excited yourself. Find the kernel of an idea that seems high potential, even if it's hard to explain.

Step 2 - Put in a lot of work making sure you understand your idea and have things to say/do with it.

Step 3 - Share pieces of it in the aftermath of a previous meetup to see if people respond to it. If they don't respond at all, you may need to drop it. If at least 1 or 2 people respond with interest you can probably make it work but:

Step 4 - Email people individually. If you're comfortable enough with some people at the meetup, send them a private messaging saying "hey, would you be interested in this thing?" (People respond way more reliably to private messages than to generic "hey guys what do you think" to the group?)

Step 5 - If people are waffling on whether the idea is exciting enough to come, say straightforwardly: I will do this if and only if X people respond enthusiastically about it. (And then if they don't, alas, let the event go)

Further Reading

I wrote this out, and then remembered that Kaj Sotala has written a really comprehensive guide to running meetups (37 pages long). If you want a lot more ideas and advice, I recommend checking it out.


There is No Akrasia

17 lifelonglearner 30 April 2017 03:33PM

I don’t think akrasia exists.


This is a fairly strong claim. I’m also not going to try and argue it.

 

What I’m really here to argue are the two weaker claims that:


a) Akrasia is often treated as a “thing” by people in the rationality community, and this can lead to problems, even though akrasia a sorta-coherent concept.


b) If we want to move forward and solve the problems that fall under the akrasia-umbrella, it’s better to taboo the term akrasia altogether and instead employ a more reductionist approach that favors specificity


But that’s a lot less catchy, and I think we can 80/20 it with the statement that “akrasia doesn’t exist”, hence the title and the opening sentence.


First off, I do think that akrasia is a term that resonates with a lot of people. When I’ve described this concept to friends (n = 3), they’ve all had varying degrees of reactions along the lines of “Aha! This term perfectly encapsulates something I feel!” On LW, it seems to have garnered acceptance as a concept, evidenced by the posts / wiki on it.


It does seem, then, that this concept of “want-want vs want” or “being unable to do what you ‘want’ to do” seems to point at a phenomenologically real group of things in the world.


However, I think that this is actually bad.


Once people learn the term akrasia and what it represents, they can now pattern-match it to their own associated experiences. I think that, once you’ve reified akrasia, i.e. turned it into a “thing” inside your ontology, problems occur:


First off, treating akrasia as a real thing gives it additional weight and power over you:


Once you start to notice the patterns, it’s harder to see things again as mere apparent chaos. In the case of akrasia, I think this means that people may try less hard because they suddenly realize they’re in the grip of this terrible monster called akrasia.


I think this sort of worldview ends up reinforcing some unhelpful attitudes towards solving the problems akrasia represents. As an example, here are two paraphrased things I’ve overheard about akrasia which I think illustrate this. (Happy to remove these if you would prefer not to be mentioned.)


“Akrasia has mutant healing powers…Thus you can’t fight it, you can only keep switching tactics for a time until they stop working…”


“I have massive akrasia…so if you could just give me some more high-powered tools to defeat it, that’d be great…”  

 

Both of these quotes seem to have taken the akrasia hypothesis a little too far. As I’ll later argue, “akrasia” seems to be dealt with better when you see the problem as a collection of more isolated disparate failures of different parts of your ability to get things done, rather than as an umbrella term.


I think that the current akrasia framing actually makes the problem more intractable.


I see potential failure modes where people come into the community, hear about akrasia (and all the related scary stories of how hard it is to defeat), and end up using it as an excuse (perhaps not an explicit belief, but as an alief) that impacts their ability to do work.


This was certainly the case for me, where improved introspection and metacognition on certain patterns in my mental behaviors actually removed a lot of my willpower which had served me well in the past. I may be getting slightly tangential here, but my point is that giving people models, useful as they might be for things like classification, may not always be net-positive.


Having new things in your ontology can harm you.


So just giving people some of these patterns and saying, “Hey, all these pieces represent a Thing called akrasia that’s hard to defeat,” doesn’t seem like the best idea.


How can we make the akrasia problem more tractable, then?


I claimed earlier that akrasia does seem to be a real thing, as it seems to be relatable to many people. I think this may actually because akrasia maps onto too many things. It’s an umbrella term for lots of different problems in motivation and efficacy that could be quite disparate problems. The typical akrasia framing lumps problems like temporal discounting with motivation problems like internal disagreements or ugh fields, and more.

 

Those are all very different problems with very different-looking solutions!


In the above quotes about akrasia, I think that they’re an example of having mixed up the class with its members. Instead of treating akrasia as an abstraction that unifies a class of self-imposed problems that share the property of acting as obstacles towards our goals, we treat it as a problem onto itself.


Saying you want to “solve akrasia” makes about as much sense as directly asking for ways to “solve cognitive bias”. Clearly, cognitive biases are merely a class for a wide range of errors our brains make in our thinking. The exercises you’d go through to solve overconfidence look very different than the ones you might use to solve scope neglect, for example.


Under this framing, I think we can be less surprised when there is no direct solution to fighting akrasia—because there isn’t one.


I think the solution here is to be specific about the problem you are currently facing. It’s easy to just say you “have akrasia” and feel the smooth comfort of a catch-all term that doesn’t provide much in the way of insight. It’s another thing to go deep into your ugly problem and actually, honestly say what the problem is.


The important thing here is to identify which subset of the huge akrasia-umbrella your individual problem falls under and try to solve that specific thing instead of throwing generalized “anti-akrasia” weapons at it.


Is your problem one of remembering to do tasks? Then set up a Getting Things Done system.


Is your problem one of hyperbolic discounting, of favoring short-term gains? Then figure out a way to recalibrate the way you weigh outcomes. Maybe look into precommitting to certain courses of action.


Is your problem one of insufficient motivation to pursue things in the first place? Then look into why you care in the first place. If it turns out you really don’t care, then don’t worry about it. Else, find ways to source more motivation.


The basic (and obvious) technique I propose, then, looks like:


  1. Identify the akratic thing.

  2. Figure out what’s happening when this thing happens. Break it down into moving parts and how you’re reacting to the situation.

  3. Think of ways to solve those individual parts.

  4. Try solving them. See what happens

  5. Iterate


Potential questions to be asking yourself throughout this process:

  • What is causing your problem? (EX: Do you have the desire but just aren’t remembering? Are you lacking motivation?)

  • How does this akratic problem feel? (EX: What parts of yourself is your current approach doing a good job of satisfying? Which parts are not being satisfied?)

  • Is this really a problem? (EX: Do you actually want to do better? How realistic would it be to see the improvements you’re expecting? How much better do you think could be doing?)


Here’s an example of a reductionist approach I did:


“I suffer from akrasia.


More specifically, though, I suffer from a problem where I end up not actually having planned things out in advance. This leads me to do things like browse the internet without having a concrete plan of what I’d like to do next. In some ways, this feels good because I actually like having the novelty of a little unpredictability in life.


However, at the end of the day when I’m looking back at what I’ve done, I have a lot of regret over having not taken key opportunities to actually act on my goals. So it looks like I do care (or meta-care) about the things I do everyday, but, in the moment, it can be hard to remember.”


Now that I’ve far more clearly laid out the problem above, it seems easier to see that the problem I need to deal with is a combination of:

  • Reminding myself the stuff I would like to do (maybe via a schedule or to-do list).

  • Finding a way to shift my in-the-moment preferences a little more towards the things I’ve laid out (perhaps with a break that allows for some meditation).


I think that once you apply a reductionist viewpoint and specifically say exactly what it is that is causing your problems, the problem is already half-solved. (Having well-specified problems seems to be half the battle.)

 

Remember, there is no akrasia! There are only problems that have yet to be unpacked and solved!


Background Reading: The Real Hufflepuff Sequence Was The Posts We Made Along The Way

17 Raemon 26 April 2017 06:15PM

This is the fourth post of the Project Hufflepuff sequence. Previous posts:


Epistemic Status: Tries to get away with making nuanced points about social reality by using cute graphics of geometric objects. All models are wrong. Some models are useful. 

Traditionally, when nerds try to understand social systems and fix the obvious problems in them, they end up looking something like this:

Social dynamics is hard to understand with your system 2 (i.e. deliberative/logical) brain. There's a lot of subtle nuances going on, and typically, nerds tend to see the obvious stuff, maybe go one or two levels deeper than the obvious stuff, and miss that it's in fact 4+ levels deep and it's happening in realtime faster than you can deliberate. Human brains are pretty good (most of the time) at responding to the nuances intuitively. But in the rationality community, we've self-selected for a lot of people who:

  1. Don't really trust things that they can't understand fully with their system 2 brain. 
  2. Tend not to be as naturally skilled at intuitive mainstream social styles. 
  3. Are trying to accomplish things that mainstream social interactions aren't designed to accomplish (i.e. thinking deeply and clearly on a regular basis).
This post is an overview of essays that rationalist-types have written over the past several years, that I think add up to a "secret sequence" exploring why social dynamics are hard, and why they are important to get right. This may useful both to understand some previous attempts by the rationality community to change social dynamics on purpose, as well as to current endeavors to improve things.

(Note: I occasionally have words in [brackets], where I think original jargon was pointing in a misleading direction and I think it's worth changing)

To start with, a word of caution:

Armchair sociolology can be harmful - Ozy's post is pertinent - most essays below fall into the category of "armchair sociology", and attempts by nerds to understand and articulate social-dynamics that they aren't actually that good at. Several times when an outsider has looked in at rationalist attempts to understood human interaction they've said "Oh my god, this is the blind leading the blind", and often that seemed to me like a fair assessment.

I think all the essays that follow are useful, and are pointing at something real. But taken individually, they're kinda like the blind men groping at the elephant, each coming away with the distinct impression an elephant is like a snake, tree, a boulder, depending on which aspect they're looking at.

[Fake Edit: Ozy informs me that they were specifically warning against amateur sociology and not psychology. I think the idea still roughly applies]

Part 1. Cultural Assumptions of Trust

Guess [Infer Culture], Ask Culture, and Tell [Reveal] Culture (Malcolm Ocean)

 

Different people have different ways of articulating their needs and asking for help. Different ways of asking require different assumptions of trust. If people are bringing different expectations of trust into an interaction, they may feel that that trust is being violated, which can seem rude, passive aggressive or oppressive.

 

I'm listing this article, instead of numerous others about Ask/Guess/Tell, because I think: a) Malcolm does a good job of explaining how all the cultures work, and b) I think his presentation of Reveal culture is a good, clearer upgrade for Brienne's Tell culture, and I'm a bit sad it didn't seem to make it into the zeitgeist yet. 

I also like the suggestion to call Guess Culture "Infer Culture" (implying a bit more about what skills the culture actually emphasizes).

Guess Culture Screens for Trying to Cooperate (Ben Hoffman)

Rationality folk (and more generally, nerds), tend to prefer explicit communication over implicit, and generally see Guess culture as strictly inferior to Ask culture once you've learned to assert yourself. 

But there is something Guess culture does which Ask culture doesn't, which is give you evidence of how much people understand you and are trying to cooperate. Guess cultures filters for people who have either invested effort into understanding your culture overall, or people who are good at inferring your own wants. 

Sharp Culture and Soft Culture (Sam Rosen)

[WARNING: It turned out lots of people thought this meant something different than what I thought it meant. Some people thought it meant soft culture didn't involve giving people feedback or criticism at all. I don't think Soft/Sharp are totally-naturally clusters in the first place, and the distinction I'm interested in (as applies to rationality-culture), is how you give feedback.

(i.e. "Dude, your art sucks. It has no perspective." vs "oh, cool. Nice colors. For the next drawing, you might try incorporating perspective", as a simplified example)]

Somewhat orthogonal to Infer/Ask/Reveal culture is "Soft" vs "Sharp" culture. Sharp culture tends to have more biting humor, ribbing each other, and criticism. Soft culture tends to value kindness and social harmony more. Sam says that Sharp culture "values honesty more." Robby Bensinger counters in the comments: "My own experience is that sharp culture makes it more OK to be open about certain things (e.g., anger, disgust, power disparities, disagreements), but less OK to be open about other things (e.g., weakness, pain, fear, loneliness, things that are true but not funny or provocative or badass)."

Handshakes, Hi, and What's New: What's Going on With Small Talk?  (Ben Hoffman)

Small talk often sounds nonsensical to literally-minded people, but it serves a fairly important function: giving people a structured path to figure out how much time/sympathy/interest they want to give each other. And even when the answer is "not much", it still is, significantly, nonzero - you regard each other as persons, not faceless strangers.

Personhood [Social Interfaces?]  (Kevin Simler)

This essays gets a lot of mixed reactions, much of which I think has to do with its use of the word "Person." The essay is aimed at explaining how people end up treating each other as persons or nonpersons, without making any kind of judgement about it. This includes noting some things human tend to do that you might consider horrible.

Like many grand theories, I think it overstates it's case and ignores some places where the explanation breaks down, but I think it points at a useful concept which is summarized by this adorable graphic:

The essay uses the word "personhood". In the original context, this was useful: it gets at why cultures develop, why it matters whether you're able to demonstrate reliably, trust, etc. It helps explain outgroups and xenophobia: outsiders do not share your social norms, so you can't reliably interact with them, and it's easier to think of them as non-people than try to figure out how to have positive interactions.

But what I'm most interested in is "how can we use this to make it easier for groups with different norms to interact with each other"? And for that, I think using the word "personhood" makes it way more likely to veer into judging each other for having different preferences and communication styles.

What makes a person is... arbitrary, but not fully arbitrary. 

Rationalist culture tends to attract people who prefer a particular style of “social interface”, often favoring explicit communication and discussing ideas in extreme detail. There's a lot of value to those things, but they have some problems:

a) this social interface does NOT mesh well with the rest of world (this is a problem if you have any goals that involve the rest of the world)

b) this goal does not uniformly mesh well with all people interested in and valuable to the rationality community.

I don't actually think it's possible to develop a set of assumptions that fit everyone's needs. But I do think it's possible to develop better tools for navigating different social contexts. I think it may be possible both to tweak sets-of-norms so that they mesh better together, or at least when they bump into each other, there's greater awareness of what's happening and people's default response is "oh, we seem to have different preferences, let's figure out how 

Maybe we can end up with something that looks kinda like this:

Against Being Against or For Tell Culture  (Brienne Yudkowsky)

Having said a bunch of things about different cultural interfaces, I think this post by Brienne is really important, and highlights the end goal of all of this.

"Cultures" are a crutch. They are there to help you get your bearings. They're better than nothing. But they are not a substitute for actually having the skills needed to navigate arbitrary social situations as they come up so you can achieve whatever it is you want to achieve. 

To master communication, you can't just be like, "I prefer Tell Culture, which is better than Guess Culture, so my disabilities in Guess Culture are therefore justified." Justified shmustified, you're still missing an arm.

My advice to you - my request of you, even - if you find yourself fueling these debates [about which culture is better], is to (for the love of god) move on. If you've already applied cognitive first aid, you've created an affordance for further advancement. Using even more tourniquettes doesn't help.

Part 2. Game Theory, Recursion and Trust

(or, "Social dynamics are really complicated, you are not getting away with the things you think you are getting away with, stop trying to be clever, manipulative, act-utilitarian or naive-consequentialist without actually understanding what is going on")

Grokking Newcomb's Problem and Deserving Trust (Andrew Critch)

Critch argues why it is not just "morally wrong", but an intellectual mistake, to violate someone’s trust (even when you don’t expect any repercussions in the future).

When someone decides whether to trust you (say, giving you a huge opportunity), on the expectation that you’ll refrain from exploiting them, they’ve already run a low-grade simulation of you in their imagination. And the thing is that you don’t know whether you’re in a simulation or not when you make the decision whether to repay them. 

Some people argue “but I can tell that I’m a conscious being, and they aren’t a literal super-intelligent AI, they’re just a human. They can’t possibly be simulating me in this high fidelity. I must be real.” This is true. But their simulation of you is not based on your thoughts, it’s based on your actions. It’s really hard to fake. 

One way to think about it, not expounded on in the article: Yes, if you pause to think about it you can notice that you’re conscious and probably not being simulated in their imagination. But by the time you notice that, it’s too late. People build up models of each other all the time, based on very subtle cues such as how fast you respond to something. Conscious you knows that you’re conscious. But their decision of whether to trust you was based off the half-second it took for unconscious you to reply to questions like “Hey, do you think you handle  Project X while I’m away?”

The best way to convince people you’re trustworthy is to actually be trustworthy.

You May Not Believe In Guess[Infer] Culture But It Believes In You (Scott Alexander)

This is short enough to just include the whole thing:

Consider an "ask culture" where employees consider themselves totally allowed to say "no" without repercussions. The boss would prefer people work unpaid overtime so ey gets more work done without having to pay anything, so ey asks everyone. Most people say no, because they hate unpaid overtime. The only people who agree will be those who really love the company or their job - they end up looking really good. More and more workers realize the value of lying and agreeing to work unpaid overtime so the boss thinks they really love the company. Eventually, the few workers who continue refusing look really bad, like they're the only ones who aren't team players, and they grudgingly accept.

Only now the boss notices that the employees hate their jobs and hate the boss. The boss decides to only ask employees if they will work unpaid overtime when it's absolutely necessary. The ask culture has become a guess culture.

How this applies to friendship is left as an exercise for the reader.

The Social Substrate (Lahwran)

A fairly in depth look into how common knowledge, signaling, newcomb-like problems and recursive modeling of each other interact to produce "regular social interaction."

I think there's a lot of interesting stuff here - I'm not sure if it's exactly accurate but it points in directions that seem useful. But I actually think the most important takeaway is the warning at the beginning:

WARNING: An easy instinct, on learning these things, is to try to become more complicated yourself, to deal with the complicated territory. However, my primary conclusion is "simplify, simplify, simplify": try to make fewer decisions that depend on other people's state of mind. You can see more about why and how in the posts in the "Related" section, at the bottom.

When you're trying to make decisions about people, you're reading a lot of subtle cues off them to get a sense of how you feel about that. When you [generic person you, not necessarily you in particular] can tell someone is making complex decisions based on game theory and trying to model all of this explicitly, it a) often comes across as a bit off, and b) even if it doesn't, you still have to invest a lot of cognitive resources figuring out how they are modeling things and whether they are actually doing a good job or missing key insights or subtle cues. The result can be draining, and it can output a general response of "ugh, something about this feels untrustworthy."

Whereas when people are able to cache this knowledge down into a system-1 level, you're able to execute a simpler algorithm that looks more like "just try to be a good trustworthy person", that people can easily read off your facial expression, and which reduces overall cognitive burden.

System 1 and System 2 Morality  (Sophie Grouchy)

There’s some confusion over what “moral” means, because there’s two kinds of morality: 

 - System 1 morality is noticing-in-realtime when people need help, or when you’re being an asshole, and then doing something about it. 

 - System 2 morality is when you have a complex problem and a lot of time to think about it. 

System 1 moralists will pay back Parfit’s Hitchhiker because doing otherwise would be being a jerk. System 2 moralists invent Timeless [Functional?] decision theory. You want a lot of people with System 2 morality in the world, trying to fix complex problems. You want people with System 1 morality in your social circle.

The person who wrote this post eventually left the rationality community, in part due to frustration due to people constantly violating small boundaries that seemed pretty obvious (things in the vein of “if you’re going to be 2 hours late, text me so I don’t have to sit around waiting for you.”)

Final Remarks

I want to reiterate - all models are wrong. Some models are useful. The most important takeaway from this is not that any particular one of these perspectives is true, but that social dynamics has a lot of stuff going on that is more complicated than you're naively imagining, and that this stuff is important enough to put the time into getting right.

What exactly is the "Rationality Community?"

17 Raemon 09 April 2017 12:11AM

This is the second post in the Project Hufflepuff sequence. It’s also probably the most standalone and relevant to other interests. The introduction post is here.


The Berkeley Hufflepuff Unconference is on April 28th. RSVPing on this Facebook Event is helpful, as is filling out this form.



 

I used to use the phrase "Rationality Community" to mean three different things. Now I only use it to mean two different things, which is... well, a mild improvement at least. In practice, I was lumping a lot of people together, many of whom neither wanted to get lumped together nor had much in common.

 

As Project Hufflepuff took shape, I thought a lot about who I was trying to help and why. And I decided the relevant part of the world looks something like this:

I. The Rationalsphere

The Rationalsphere is defined in the broadest possible sense - a loose cluster of overlapping interest groups, communities and individuals. It includes people who disagree wildly with each other - some who are radically opposed to one another. It includes people who don’t identify as “rationalist” or even as especially interested in “rationality” - but who interact with each other on a semi-regular basis. I think it's useful to be able to look at that ecosystem as a whole, and talk about it without bringing in implications of community.

continue reading »

A Month's Worth of Rational Posts - Feedback on my Rationality Feed.

16 deluks917 15 May 2017 02:21PM

For the last two months I have been publishing a feed of rationalist articles. Oriignally the feed was only published on the SSC discord channel SSC Discord (Be charitable, kind and don't treat the place like 4chan). For the last few days I have also been publishing it on my blog deluks917.wordpress.com. I categorize the links and include a brief excerpt, review, and/or teaser. If you would like to see an exampel in practice just check today's post. The average number of links per day, in the last month, has been six. But this number has been higher recently. I have not missed a single day since I started, so I think its likely I will continue doing this. The list of blogs I check is located here: List of Blogs

I am looking for some feedback. At the bottom of this post I am  including a month's worth of posts categorized using the current system. Posts are not nescessarily in any particular order since my categorization system has not been constant over time. Lots of posts were moved around by hand. 

1 -  Should I share the feed the results somewhere other than SSC-discord + my blog? Mindlevelup suggested I write up a weekly roundup. I could share such a roundup via some on lesswrong and SSC. I would estimate the expected number of links in such a psot to be around 35. Links would be posted in chronoligcal order within categories. Alternatively I could share such a post every two weeks. Its also possible to have a mailing list but I currently find this less promising. 

2 - Do the categories make a reasonable amount of sense? What tweaks would you make. I have ocnsidered mixing some of the smaller categories (Math and CS, Amusement into "misc"). 

3 - Are there any blogs I should include/drop from the feed. For example I have been considering dropping ribbonfarm. The highest priority is to get the content thats directly about instrumental/epsitemic rationality. The bar is higher for politics and culture_war. I should note I am not going to personally include any blog without an RSS feed. 

4 - Is anyone willing to write a "Best of rationalist tumblr" post. If I write a weekly/bi-weekly round up I could combine it with an equivalent "best of tumblr" post. The tumblr post would not have to be daily, just weekly or every other week. We could take turns posting the resulting combination to lesswrong/SSC and collecting the juicy karma. However its worth noting that SSC-reddit has some controls on culture_war (outside of the CW thread). Since we want to post to r/SSC we need to keep the density of culture_war to reasonable levels. Lesswrong also has some anti-cw norms.

=== Last Month's Rationality Content === 

**Scott**

http://slatestarcodex.com/2017/05/11/silicon-valley-a-reality-check/ - What a person finds in Silicon Valley mirrors the seeker.

http://slatestarcodex.com/2017/05/09/links-517-rip-van-linkle/ - Links.

http://slatestarcodex.com/2017/04/11/sacred-principles-as-exhaustible-resources/ - Don't deplete the free speech commons.

http://slatestarcodex.com/2017/04/12/clarification-to-sacred-principles-as-exhaustible-resources/  - Clarifications and caveats on Scott's last article on free speech and sacred values.

http://slatestarcodex.com/2017/04/13/chametz/ - A Jewish Vampire Story

http://slatestarcodex.com/2017/04/17/learning-to-love-scientific-consensus/ - Scott Critiques a list of 10 maverick inventors. He then reconsiders his previous science skepticism.

http://slatestarcodex.com/2017/04/21/ssc-journal-club-childhood-trauma-and-cognition/ - A new study challenges the idea that child abuse reduces brain function.

http://slatestarcodex.com/2017/04/25/book-review-the-hungry-brain/ - Scott gives a favorable view of the "establishment" view of nutrition.

http://slatestarcodex.com/2017/04/26/anorexia-and-metabolic-set-point/ - Short Post (for Scott)

https://slatestarscratchpad.tumblr.com/post/160028275801/slatestarscratchpad-wayward-sidekick-you - Scott discusses engaging with ideas you find harmful. He also discusses his attitude toward making his blog as friendly as possible. [culture_war]

http://slatestarcodex.com/2017/05/01/neutral-vs-conservative-the-eternal-struggle/ - Formally neutral institutions have a liberal bias. Conservatives react by seceding and forming their own institutions. The end result is bad for society. [Culture War]

http://slatestarcodex.com/2017/05/04/getting-high-on-your-own-supply/ - "If you optimize for the epistemic culture that’s best for getting elected, but that culture isn’t also the best for running a party or governing a nation, then the fact that your culture affects your elites as well becomes a really big problem." Short for Scott.

http://slatestarcodex.com/2017/05/07/ot75-the-comment-king/ - bi-weekly visible open thread.

http://unsongbook.com/postscript-1-wrap-parties-fan-music/ - Final chapter of Unsong goes up approximately 8pm on Sunday. Unsong will have an epilogue will will go up on Wednesday. Wrap party details. (I will be at the wrap party on sunday).

http://unsongbook.com/book-iv-kings/ - "Somebody had to, no one would / I tried to do the best I could / And now it’s done, and now they can’t ignore us / And even though it all went wrong / I’ll stand against the whole unsong / With nothing on my tongue but HaMephorash"

http://unsongbook.com/chapter-71-but-for-another-gives-its-ease/ - Penultimate chapter of Unsong.

http://unsongbook.com/chapter-70-nor-for-itself-hath-any-care/ - Newest Chapter.

http://unsongbook.com/authors-note-10-hamephorash-hamephorash-party/ - Final Chapter goes up may 14. Bay Area Reading party announced.

http://unsongbook.com/chapter-69-love-seeketh-not-itself-to-please/ - Newest Chapter.

http://unsongbook.com/chapter-68-puts-all-heaven-in-a-rage/ - Newest Chapter.

**Rationalism**

http://lesswrong.com/r/discussion/lw/ozz/gearsness_of_understanding/ - "I want to point at a property of models that isn't about what they're modeling. It interacts with the clarity of what they're modeling, but only in the same way that smudged lines in a roadmap interact with the clarity of the roadmap. This property is how deterministically interconnected the variables of the model are.". The theory is applied to multiple explicit examples.

https://thepdv.wordpress.com/2017/05/11/how-i-use-beeminder/ - Short but gives details. Beeminder is the only productivity system that worked for the author.

https://putanumonit.com/2017/05/09/time-well-spent/ - Akrasia and procrastination. A review of some of the rationalist thinking on the topic. Jacob's personal take and his system for tracking his productivity.

https://putanumonit.com/2017/05/09/time-well-spent/ - Akrasia and procrastination. A review of some of the rationalist thinking on the topic. Jacob's personal take and his system for tracking his productivity.

http://kajsotala.fi/2017/05/cognitive-core-systems-explaining-intuitions-behind-belief-in-souls-free-will-and-creation-myths/ - Description of four core systems human, and other animals, are born with. An explanation of why these systems lead to belief in souls. Short.

https://mindlevelup.wordpress.com/2017/05/06/taking-criticism/ - Reframing criticism so that it makes sense to the author (who is bad at taking criticism). A Q&A between the author and himself.

http://lesswrong.com/r/discussion/lw/oz1/soft_skills_for_running_meetups_for_beginners/ - Concrete advice for running meetups. Not especially focused on beginning organizers. Written by the person who organized Solstice.

http://effective-altruism.com/ea/19t/mental_health_resource_for_ea_community/ - A breakdown of the most useful information about Mania and Psychosis. Extremely practical advice. Julia Wise.

http://bearlamp.com.au/working-with-multiple-problems-at-once - Problems add up and you run out of time. How do you get out? Very practical.

http://agentyduck.blogspot.com/2017/05/creativity-taps.html - Practical ideas for exercising creativity.

http://lesswrong.com/r/discussion/lw/oyk/acting_on_your_intended_preferences_what_does/ - What does it look like in practice to pursue your goals. A series of practical questions to ask your. Links to a previous series of blog post are included.

https://thingofthings.wordpress.com/2017/05/03/why-do-all-the-rationalists-live-in-the-bay-area/ - Benefits of living in the Bay. The Bay is a top place for software engineers even accounting for cost of living, Rationalist institutions are in the Bay, there are social and economic benefits to being around other community members.

https://qualiacomputing.com/2017/05/04/the-most-important-philosophical-question/ - “Is happiness a spiritual trick, or is spirituality a happiness trick?”

http://particularvirtue.blogspot.com/2017/05/how-to-build-community-full-of-lonely.html - Why so many rationalists feel lonely and concrete suggestions for improving social groups. Advice is given to people who are popular, lonely or organizers. Very practical.

https://hivewired.wordpress.com/2017/05/07/announcing-entropycon-12017/ - We beat smallpox, we will beat death, we can try to beat entropy. A humorous mantra against nihilism.

https://mindlevelup.wordpress.com/2017/04/30/there-is-no-akrasia/ - The author argues that akrasia isn't a "thing" its a "sorta-coherent concept". He also argues that "akrasia" is not a useful concept and can be harmful.

http://bearlamp.com.au/experiments-iterations-and-the-scientific-method/ - A Graph of the scientific method in practice. The author works through his quantified self in practice and discusses his experiences.

https://everythingstudies.wordpress.com/2017/04/29/all-the-worlds-a-trading-zone/ - Cultures with different norms and languages can interact successfully.

http://kajsotala.fi/2017/04/relationship-compatibility-as-patterns-of-emotional-association/ - What is relationship "chemistry"?

http://lesswrong.com/lw/oyc/nate_soares_replacing_guilt_series_compiled_in/ - Ebook. 45 blog posts on replacing guilt and shame with a stronger motivation.

http://mindingourway.com/assuming-positive-intent/ - "If you're actively working hard to make the world a better place, then we're on the same team. If you're committed to letting evidence and reason guide your actions, then I consider you friends, comrades in arms, and kin."

http://bearlamp.com.au/quantified-self-tracking-with-a-form/ - Practical advice based on Elo's personal experience.

http://lesswrong.com/r/discussion/lw/ovc/background_reading_the_real_hufflepuff_sequence/ - Links and Descriptions of rationalist articles about group norms and dynamics.

https://everythingstudies.wordpress.com/2017/04/24/people-are-different/ - "We need to understand, accept and respect differences, that one size does not fit all, but to (and from) each their own."

http://bearlamp.com.au/yak-shaving-2/ - "A question worth asking is whether you are in your life at present causing a build up of problems, a decrease of problems, or roughly keeping them about the same level."

http://lesswrong.com/r/discussion/lw/oxk/i_updated_the_list_of_rationalist_blogs_on_the/ - Up to date list of rationalist blogs.

https://aellagirl.com/2017/05/02/internet-communities-otters-vs-possums/ - Possums: people who like a specific culture. Otters are people who like most cultures. What happens when the percentage of otters in a community increases?

https://aellagirl.com/2017/04/24/how-i-lost-my-faith/ - "People sometimes ask the question of why it took so long. Really I’m amazed that it happened at all. Before we even approach the aspect of “good arguments against religion”, you have to understand exactly how much is sacrificed by the loss of religion."

http://particularvirtue.blogspot.com/2017/04/on-social-spaces.html - Twitter, Tumblr, Facebook etc. PV responds to Zvi's articles about facebook. PV defends tumblr and facebook and has some criticisms of twitter. Several examples are given where ratioanalist groups tried to change platforms.

http://www.overcomingbias.com/2017/04/superhumans-live-among-us.html - Some human polymaths really are superhuman. But they don't have the track record to prove it.

https://thezvi.wordpress.com/2017/04/22/against-facebook/ - Sections: 1. A model breaking down how Facebook actually works. 2. An experiment with my News Feed. 3. Living with the Algorithm. 4. See First, Facebook’s most friendly feature. 5. Facebook is an evil monopolistic pariah Moloch. 6. Facebook is bad for you and Facebook is ruining your life. 7. Facebook is destroying discourse and the public record. 8. Facebook is out to get you.

https://thezvi.wordpress.com/2017/04/22/against-facebook-comparison-to-alternatives-and-call-to-action/ - Zvi's advice for managing your information streams and discussion platforms. Facebook can mostly be replaced.

https://rationalconspiracy.com/2017/04/22/moving-to-the-bay-area/ - Downsides of the Bay. Extensively sourced. Cost of living, traffic, public transit, crime, cleanliness.

https://nintil.com/2017/04/18/still-not-a-zombie-replies-to-commenters/ - Thoughts on consciousness and identity.

http://bearlamp.com.au/an-inquiry-into-memory-of-humans/ - The reader is asked to try various interesting memory exercises.

https://www.jefftk.com/p/how-to-make-housing-cheaper - 9 ways to make housing cheaper.

http://lesswrong.com/r/discussion/lw/owb/straw_hufflepuffs_and_lone_heroes/ - Should Harry have joined Hufflepuff in HPMOR? Harry had reasons to be a lone hero, do you?

http://lesswrong.com/lw/owa/lesswrong_analytics_february_2009_to_january_2017/ - Activity graphs of lesswrong over time, which posts had the most views, links to source code and further reading.

https://thezvi.wordpress.com/2017/04/23/help-us-find-your-blog-and-others/ - Zvi will read a post from your blog and consider adding you to his RSS feed.

https://thingofthings.wordpress.com/2017/04/11/book-post-for-march/ - Books on parenting.

https://boardgamesandrationality.wordpress.com/2017/04/24/first-blog-post/ - Dealing With Secret Information in boardgames and real life.

http://www.overcomingbias.com/2017/04/mormon-transhumanists.html - The relationship between religious community and technological change. Long for Overcoming Bias.

https://putanumonit.com/2017/04/15/bad-religion/ - "Rationality is a really unsatisfactory religion. But it’s a great life hack."

https://thezvi.wordpress.com/2017/04/12/escalator-action/ - Should we walk on elevator?

https://putanumonit.com/2017/04/21/book-review-too-like-the-lightning/ - The world of Jacob's dreams, thought on AI, a book review.

**EA**

http://effective-altruism.com/ea/19y/understanding_charity_evaluation/ - A detailed breakdown of how charity evaluation works in practice. Openly somewhat speculative.

http://blog.givewell.org/2017/05/11/update-on-our-views-on-cataract-surgery/ - previously Givewell had unsuccessfully tried to find recommendable cataract surgery charities. The biggest issues were “room for funding” and “lack of high quality monitoring data”. However they believe that cataract surgery is a promising intervention and they are doing more analysis.

https://80000hours.org/2017/05/how-much-do-hedge-fund-traders-earn/ - Detailed report on career trajectories and earnings. "We found that junior traders typically earn $300k – $3m per year, and it’s possible to reach these roles in 4 – 8 years."

https://www.givedirectly.org/blog-post?id=7612753271623522521 - 8 News links about GiveDirectly, Basic Income and cash transfer.

https://80000hours.org/2017/05/most-people-report-believing-its-incredibly-cheap-to-save-lives-in-the-developing-world/ - Details of a study on how Much Americans think it costs to save a life. Discussion of why people gave such optimistic answers. "It turns out that most Americans believe a child can be prevented from dying of preventable diseases for very little – less than $100."

https://www.thelifeyoucansave.org/Blog/ID/1355/Are-Giving-Games-a-Better-Way-to-Teach-Philanthropy - Literature review on "philanthropy games". Covers both traditional student philanthropy courses and the much shorter "giving game".

https://www.givedirectly.org/blog-post?id=8255610968755843534 - Links to news stories about Effective Altruism

http://benjaminrosshoffman.com/an-openai-board-seat-is-surprisingly-expensive/ - " In exchange for a board seat, the Open Philanthropy Project is aligning itself socially with OpenAI, by taking the position of a material supporter of the project."

https://www.givedirectly.org/blog-post?id=5010525406506746433 - Links to News Articles about Give Directly, Basic Income and Cash Transfer.

https://www.givedirectly.org/blog-post?id=121797500310578692 - Report on a program to give cash to coffee farmers in eastern Uganda.

http://effective-altruism.com/ea/19d/update_on_effective_altruism_funds/ - Details from the first round of funding, community feedback, Mistakes and Updates.

http://lesswrong.com/r/discussion/lw/ox4/effective_altruism_is_selfrecommending/ - Open Philanthropy project has a closed validation loop. A detailed timeline of GiveWell/Open-Philanthropy is given and many untested assumptions are pointed out. A conceptual connection is made to confidence games.

http://lesswrong.com/r/discussion/lw/oxd/the_2017_effective_altruism_survey_please_take/ - Take the survey :)

https://www.givingwhatwecan.org/post/2017/04/career-of-professor-alan-fenwick/ - Retrospective on the career of the director of the Schistosomiasis Institute.

http://www.openphilanthropy.org/blog/new-report-early-field-growth - The history of attempts to grow new fields of research or advocacy.

https://www.givedirectly.org/blog-post?id=4406309858976986548 - news links about GiveDirectly, Basic Income and Cash Transfers

https://intelligence.org/2017/04/30/2017-updates-and-strategy/ - Outreach, expansion, detailed research plan, state of the AI-risk community.

http://blog.givewell.org/2017/05/04/why-givewell-is-partnering-with-idinsight/ - IDinsight is an international NGO that aims to help its clients develop and use rigorous evidence to improve social impact. Summary, Background, goals, initial plans.

https://www.thelifeyoucansave.org/Blog/ID/1354/A-Shift-in-Priorities-at-the-Giving-Game-Project - Finding sustainable funding, Providing measurable outcomes, improving follow ups with participants.

https://80000hours.org/2017/05/most-people-report-believing-its-incredibly-cheap-to-save-lives-in-the-developing-world/ - Details of a study on how Much Americans think it costs to save a life. Discussion of why people gave such optimistic answers. "It turns out that most Americans believe a child can be prevented from dying of preventable diseases for very little – less than $100."

https://www.thelifeyoucansave.org/Blog/ID/1355/Are-Giving-Games-a-Better-Way-to-Teach-Philanthropy - Literature review on "philanthropy games". Covers both traditional student philanthropy courses and the much shorter "giving game".

http://www.openphilanthropy.org/blog/why-are-us-corporate-cage-free-campaigns-succeeding - The article contains a timeline of cagefree reform. Some background reasons given are: Undercover investigations, College engagement, Corporate engagement, Ballot measures, Gestation crate pledges, European precedent.

https://www.givingwhatwecan.org/post/2017/04/a-successor-to-the-giving-what-we-can-trust/ - The Giving What we Can Trust has joined with the "Effective Altruism Funds" (run by the Center for Effective Altruism).

http://lesswrong.com/r/discussion/lw/oyf/bad_intent_is_a_behavior_not_a_feeling/ - Response to Nate Soares, application to EA. "If you try to control others’ actions, and don’t limit yourself to doing that by honestly informing them, then you’ll end up with a strategy that distorts the truth, whether or not you meant to."

**Ai_risk**

http://effective-altruism.com/ea/19c/intro_to_caring_about_ai_alignment_as_an_ea_cause/ - By Nate Soares. A modified transcript of the talk he gave at Google on the problem of Ai Alignment.

http://lukemuehlhauser.com/monkey-classification-errors/ , http://lukemuehlhauser.com/adversarial-examples-for-pigeons/ - Adversarial examples for monkeys and pigeons respectively.

https://intelligence.org/2017/05/10/may-2017-newsletter/ - Research updates, MIRI hiring, General news links about AI

https://intelligence.org/2017/04/12/ensuring/ - Nate Soares gives a talk at Google about "Ensuring smarter-than-human intelligence has a positive outcome". An outline of the talk is included.

https://intelligence.org/2017/04/07/decisions-are-for-making-bad-outcomes-inconsistent/ - An extended discussion of Soares's latest paper "Cheating Death in Damascus".

**Research**

https://everythingstudies.wordpress.com/2017/05/12/the-eurovision-song-contest-taste-landscape/ - Analysis of Voting patterns in the Eurovision Contest. Alliances and voting Blocs are analyzed in depth.

https://srconstantin.wordpress.com/2017/05/12/do-pineal-gland-extracts-promote-longevity-well-maybe/ - Analysis of hormonal systems and their effect on metabolism and longevity.

https://acesounderglass.com/2017/05/11/an-opportunity-to-throw-money-at-the-problem-of-medical-science/ - Help crowdfund a randomized controlled trial. A promising Sepsis treatment needs a RCT but the method is very cheap and unpatentable. So there is no financial incentive for a company to fund the study.

https://randomcriticalanalysis.wordpress.com/2017/05/09/towards-a-general-factor-of-consumption/ - Factor Analysis leads to a general factor of consumption. Discussion of the data and analysis of the model. Very thorough.

https://randomcriticalanalysis.wordpress.com/2017/04/13/disposable-income-also-explains-us-health-expenditures-quite-well/ - Long Article, lots of graphs. "I argued consumption, specifically Actual Individual Consumption, is an exceptionally strong predictor of national health expenditures (NHE) and largely explains high US health expenditures.  I found AIC to be a much more robust predictor of NHE than GDP... I think it useful to also demonstrate these patterns as it relates to household disposable income"

https://randomcriticalanalysis.wordpress.com/2017/04/15/some-useful-data-on-the-dispersion-characteristics-of-us-health-expenditures/ - US Health spending is highly concentrated in a small fraction of the population. Is this true for other countries?

https://randomcriticalanalysis.wordpress.com/2017/04/17/on-popular-health-utilization-metrics/ - An extremely graph dense article responding to a widely cited paper claiming that "high utilization cannot explain high US health expenditures."

https://randomcriticalanalysis.wordpress.com/2017/04/28/health-consumption-and-household-disposable-income-outside-of-the-oecd/ - Another part in the series on healthcare expenses. Extending the analysis to non-OECD countries. Lots of graphs.

https://randomcriticalanalysis.wordpress.com/2017/05/09/towards-a-general-factor-of-consumption/ - Factor Analysis leads to a general factor of consumption. Discussion of the data and analysis of the model. Very thorough.

https://srconstantin.wordpress.com/2017/04/12/parenting-and-heritability-overview/ - Detailed literature review on heritability and what parenting can affect. A significant number of references are included.

https://nintil.com/2017/04/23/links-7/ - Psychology, Economics, Philosophy, AI

http://lesswrong.com/r/discussion/lw/ox8/unstaging_developmental_psychology/ - A mathematical model of stages of psychological development. The linked technical paper is very impressive. Starting from an abstract theory the authors managed to create a psychological theory that was concrete enough to apply in practice.

**Math and CS**

http://andrewgelman.com/2017/05/10/everybody-lies-seth-stevens-davidowitz/ - A fairly positive review of Seth's book on learning from data.

http://eli.thegreenplace.net/2017/adventures-in-jit-compilation-part-4-in-python/ - Writing a JIT compiler in Python. Discusses both using native python code and the PeachPy library. Performance consideration are explicitly not discussed.

http://eli.thegreenplace.net/2017/book-review-essentials-of-programming-languages-by-d-friedman-and-m-wand/ - Short review. "This book is a detailed overview of some fundamental ideas in the design of programming languages. It teaches by presenting toy languages that demonstrate these ideas, with a full interpreter for every language"

http://eli.thegreenplace.net/2017/adventures-in-jit-compilation-part-3-llvm/ - LLVM can dramatically speed up straightforward source code.

http://www.scottaaronson.com/blog/?p=3221 - Machine Learning, Quantum Mechanics, Google Calendar

**Politics and Economics**

http://noahpinionblog.blogspot.com/2017/04/ricardo-reis-defends-macro_13.html - Macro is defended from a number of common criticisms. A large number of modern papers are cited (including 8 job market papers). Some addressed criticisms include: Macro relies on representative agents, Macro ignores inequality, Macro ignores finance and Macro ignores data and focuses mainly on theory.

http://econlog.econlib.org/archives/2017/04/economic_system.html - What are the fundamental questions an economic system must answer?

http://andrewgelman.com/2017/04/18/reputational-incentives-post-publication-review-two-partial-solutions-misinformation-problem/ - Gelman gives a list of important erroneous analysis in the news and scientific journals. He then considers if negative reputational incentives or post-publication peer review will solve the problem.

https://srconstantin.wordpress.com/2017/05/09/how-much-work-is-real/ - What fraction of jobs are genuinely productive?

https://hivewired.wordpress.com/2017/05/06/yes-this-is-a-hill-worth-dying-on/ - The Nazis were human too. Even if a hill is worth dying on its probably not worth killing for. Discussion of good universal norms. [Culture War]

https://srconstantin.wordpress.com/2017/05/09/chronic-fatigue-syndrome/ - Literature Analysis on Chronic Fatigue Syndrome. Extremely thorough.

https://www.gwern.net/newsletter/2017/04 - A Month's worth of links. Ai, Recent evolution, heritability and other topics.

https://thingofthings.wordpress.com/2017/05/05/the-cluster-structure-of-genderspace/ - For many traits the bell curves for men and women are quite close. Visualizations of Cohen's D. Discussion of trans specific medical interventions.

https://www.jefftk.com/p/replace-infrastructure-wholesale - Can you just dig up a city and replace all the infrastructure in a week?

https://thingofthings.wordpress.com/2017/04/19/deradicalizing-the-romanceless/ - Ozy discusses the problem of (male) involuntarily celibacy.

http://noahpinionblog.blogspot.com/2017/04/the-siren-song-of-homogeneity.html - The alt-right is about racial homogeneity. Smith Reviews the data studying whether a homogeneous society increases trust and social capital. Smith discusses the Japanese culture and his time in Japan. Smith considers the arbitrariness of racial categories despite admitting that race has a biological reality. Smith flips around some alt right slogans. [Extreme high quality engagement with opposing ideas. Culture War]

https://thezvi.wordpress.com/2017/04/16/united-we-blame/ - A list of articles about United, Zvi's thoughts on United, general ideas about airlines.

http://noahpinionblog.blogspot.com/2017/04/why-101-model-doesnt-work-for-labor.html - Noah Smith gives many reasons why the simple supply/demand model can't work for labor economics.

https://thingofthings.wordpress.com/2017/04/14/concerning-archive-of-our-own/ - Ozy defends the moderation policy of the fanction archive A03. [Culture War]

https://thingofthings.wordpress.com/2017/04/13/fantasies-are-okay/ - When are fantasies ok? What about sexual fantasies? [Culture War]

https://srconstantin.wordpress.com/2017/04/25/on-drama/ - Ritual, The Psychology of Adolf Hitler, the dangerous emotion of High Drama, The Rite of Spring.

https://qualiacomputing.com/2017/04/26/psychedelic-science-2017-take-aways-impressions-and-whats-next/ - Notes on the 2017 Psychedelic Science conference.

**Amusement**

http://kajsotala.fi/2017/04/fixing-the-4x-end-game-boringness-by-simulating-legibility/ - "4X games (e.g. Civilization, Master of Orion) have a well-known problem where, once you get sufficiently far ahead, you’ve basically already won and the game stops being very interesting."

https://putanumonit.com/2017/05/12/dark-fiction/ - Jacob does some Kabbalahistic Analysis on the Story of Jacob, Unsong Style.

https://protokol2020.wordpress.com/2017/04/30/several-big-numbers-to-sort/ - 12 Amusing definitions of big numbers.

http://existentialcomics.com/comic/183 - The Life of Francis

http://existentialcomics.com/comic/181 - A Presocratic Get Together.

https://protokol2020.wordpress.com/2017/05/07/problem-with-perspective/ - A 3D geometry problem.

http://existentialcomics.com/comic/184 - Wittgenstein in the Great War(edited)

http://existentialcomics.com/comic/182 - Captain Metaphysics and the Postmodern Peril

**Adjacent**

https://medium.com/@freddiedeboer/conservatives-are-wrong-about-everything-except-predicting-their-own-place-in-the-culture-e5c036fdcdc5 - Conservatives correctly predicted the effects of gay acceptance and no fault divorce. They have also been proven correct about liberal bias in academia and the media. [Culture War]

https://medium.com/@freddiedeboer/franchises-that-are-appropriate-for-children-are-inherently-limited-in-scope-8170e76a16e2 - Superhero movies have an intended audience that includes children. This drastically limits what techniques they can use and themes they can explore. Freddie goes into the details.

https://fredrikdeboer.com/2017/05/11/study-of-the-week-rebutting-academically-adrift-with-its-own-mechanism/ - Freddie wrote his dissertation on the College Learning Assessment, the primary source in "Academically Adrift".

https://medium.com/@freddiedeboer/politics-as-politics-12ab43429e64 - Politics as “group affiliation” vs politics as politics. Annoying atheists aren’t as bad as fundamentalist Christians even if more annoying atheists exist in educated leftwing spaces. Freddie’s clash with the identitarian left despite huge agreement on the object level. Freddie is a socialist not a liberal. [Culture War]

https://www.ribbonfarm.com/2017/05/09/priest-guru-nerd-king/ - Facebook, Governance, Doctrine, Strategy, Tactics and Operations. Fairly short post for Ribbonfarm.

https://fredrikdeboer.com/2017/05/09/lets-take-a-deep-dive-into-that-times-article-on-school-choice/ - A critique of the problems in the Time's well cited article on school choice. Points out issues with selection bias, lack of theory and the fact that "not everyone can be average".

https://fredrikdeboer.com/2017/05/09/lets-take-a-deep-dive-into-that-times-article-on-school-choice/ - A critique of the problems in the Time's well cited article on school choice. Points out issues with selection bias, lack of theory and the fact that "not everyone can be average".

http://marginalrevolution.com/marginalrevolution/2017/05/conversation-garry-kasparov.html - "We talked about AI, his new book Deep Thinking: Where Machine Intelligence Ends and Human Creativity Begins, why he has become more optimistic, how education will have to adjust to smart software, Russian history and Putin, his favorites in Russian and American literature, Tarkovsky..."

http://econlog.econlib.org/archives/2017/04/iq_with_conscie.html - "My fellow IQ realists are, on average, a scary bunch.  People who vocally defend the power of IQ are vastly more likely than normal people to advocate extreme human rights violations." There are interesting comments here: https://redd.it/6697sh .

http://econlog.econlib.org/archives/2017/04/iq_with_conscie_1.html - Short follow up to the above article.(edited)

http://marginalrevolution.com/marginalrevolution/2017/04/what-would-people-do-if-they-had-superpowers.html - Link to a paper showing 94% of people said they would use superpowers selfishly.

http://waitbutwhy.com/2017/04/neuralink.html - Elon Musk Wants to Build a wizard hat for the brain. Lots of details on the science behind Neuralink.

http://marginalrevolution.com/marginalrevolution/2017/04/dont-people-care-economic-inequality.html - Most Americans don’t mind inequality nearly as much as pundits and academics suggest.

http://marginalrevolution.com/marginalrevolution/2017/04/two-rationality-tests.html - What would you ask to determine if someone is rational? What would Tyler ask?(edited)

http://tim.blog/2017/05/04/exploring-smart-drugs-fasting-and-fat-loss-dr-rhonda-patrick/ - “Avoiding all stress isn’t the answer to fighting aging; it’s about building resiliency to environmental stress.”

http://wakingup.libsyn.com/what-should-we-eat - "Sam Harris speaks with Gary Taubes about his career as a science journalist, the difficulty of studying nutrition and public health scientifically, the growing epidemics of obesity and diabetes, the role of hormones in weight gain, the controversies surrounding his work, and other topics."(edited)

http://www.econtalk.org/archives/2017/05/jennifer_pahlka.html - Code for America. Bringing technology into the government sector.

http://heterodoxacademy.org/resources/viewpoint-diversity-experience/ - A six step process to appreciating viewpoint diversity. I am not sure this site will be the most useful to rationalists , on the object level, but its interesting to see what Haidt came up with.

http://www.econtalk.org/archives/2017/04/elizabeth_pape.html - Elizabeth Pape on Manufacturing and Selling Women's Clothing and Elizabeth Suzann(edited)

http://www.mrmoneymustache.com/2017/04/25/there-are-no-guarantees/ - Avoid Contracts. Don't work another year "just in case".

http://marginalrevolution.com/marginalrevolution/2017/04/saturday-assorted-links-109.html - Assorted Links on politics, Derrida, Shaolin Monks.

http://econlog.econlib.org/archives/2017/04/earth_20.html - Bryan Caplan was a guest on freakanomics Radio. The topic was  "Earth 2.0: Is Income Inequality Inevitable?".

https://www.ribbonfarm.com/2017/04/18/entrepreneurship-is-metaphysical-labor/ - Metaphysics as Intellectual Ergonomics. Entrepreneurship is Applied Metaphysics.

https://www.ribbonfarm.com/2017/04/13/idiots-scaring-themselves-in-the-dark/ - Getting Lost. "The uncanny. This is the emotion of eeriness, spookiness, creepiness"

**Podcast**

http://rationallyspeakingpodcast.org/show/rs-182-spencer-greenberg-on-how-online-research-can-be-faste.html - Podcast. Spencer Greenberg on "How online research can be faster, better, and more useful".

https://medium.com/conversations-with-tyler/patrick-collison-stripe-podcast-tyler-cowen-books-3e43cfe42d10 - Patrick Collison, co founder of Stripe, interviews Tyler.

http://tim.blog/2017/04/11/cory-booker/ - Podcast with US Senator Cory Booker. "Street Fights, 10-Day Hunger Strikes, and Creative Problem-Solving"

http://econlog.econlib.org/archives/2017/04/the_undermotiva_1.html - Two Case studies on libertarians who changed their views for bad reasons.(edited)

http://www.stitcher.com/podcast/vox/the-ezra-klein-show/e/death-sex-and-moneys-anna-sale-on-bringing-empathy-to-politics-50101701 - Interview with the host of the WNYC podcast Death, Sex, and Money.

http://marginalrevolution.com/marginalrevolution/2017/05/econtalk-podcast-russ-roberts-complacent-class.html - "Cowen argues that the United States has become complacent and the result is a loss of dynamism in the economy and in American life, generally. Cowen provides a rich mix of data, speculation, and creativity in support of his claims."

http://tim.blog/2017/04/16/marie-kondo/ - Podcast. "Marie Kondo is a Japanese organizing consultant, author, and entrepreneur."

http://www.econtalk.org/archives/2017/04/rana_foroohar_o.html - Podcast. Rana Foroohar on the Financial Sector and Makers and Takers

http://www.stitcher.com/podcast/vox/the-ezra-klein-show/e/cal-newport-on-doing-deep-work-and-escaping-social-media-49878016 - Cal Newport on doing Deep Work and escaping social media.

https://www.samharris.org/podcast/item/forbidden-knowledge - Podcast with Charles Murray. Controversy over The Bell Curve, the validity and significance of IQ as a measure of intelligence, the problem of social stratification, the rise of Trump. [culture war](edited)

http://www.stitcher.com/podcast/vox/the-ezra-klein-show/e/elizabeth-warren-on-what-barack-obama-got-wrong-49949167 - Ezra Klein Podcast with Elizabeth Warren.

http://marginalrevolution.com/marginalrevolution/2017/04/stubborn-attachments-podcast-ft-alphaville.html - Pocast with Tyler Cowen on Stubborn Attachments. "I outline a true and objectively valid case for a free and prosperous society, and consider the importance of economic growth for political philosophy, how and why the political spectrum should be reconfigured, how we should think about existential risk, what is right and wrong in Parfit and Nozick and Singer and effective altruism, how to get around the Arrow Impossibility Theorem, to what extent individual rights can be absolute, how much to discount the future, when redistribution is justified, whether we must be agnostic about the distant future, and most of all why we need to “think big.”"

http://www.themoneyillusion.com/?p=32435 - Notes on three podcasts. Faster RGDP growth, Monetary Policy, Tyler Cowen's philosophical views.(edited)

http://www.stitcher.com/podcast/vox/the-ezra-klein-show/e/vc-bill-gurley-on-transforming-health-care-50030526 - A conversation about which healthcare systems are possible in the USA and the future of Obamacare.

https://www.currentaffairs.org/2017/05/campus-politics-and-the-administrative-mind - The nature of College Bureaucracy. Focuses on protests and Title 9. [Culture war]

http://www.stitcher.com/podcast/vox/the-ezra-klein-show/e/cory-booker-returns-live-to-talk-trust-trump-and-basic-incomes-50054271 - "Booker and I dig into America’s crisis of trust. Faith in both political figures and political institutions has plummeted in recent decades, and the product is, among other things, Trump’s presidency. So what does Booker think can be done about it?"

http://www.stitcher.com/podcast/vox/the-ezra-klein-show/e/cal-newport-on-doing-deep-work-and-escaping-social-media-49878016 - Cal Newport on doing Deep Work and escaping social media.

http://tim.blog/2017/04/22/dorian-yates/ - Bodybuilding Champion. High Intensity Training, Injury Prevention, and Building Maximum Muscle.

[Link] Reality has a surprising amount of detail

14 jsalvatier 13 May 2017 08:02PM

Brief update on the consequences of my "Two arguments for not thinking about ethics" (2014) article

14 Kaj_Sotala 05 April 2017 11:25AM

In March 2014, I posted on LessWrong an article called "Two arguments for not thinking about ethics (too much)", which started out with:

I used to spend a lot of time thinking about formal ethics, trying to figure out whether I was leaning more towards positive or negative utilitarianism, about the best courses of action in light of the ethical theories that I currently considered the most correct, and so on. From the discussions that I've seen on this site, I expect that a lot of others have been doing the same, or at least something similar.

I now think that doing this has been more harmful than it has been useful, for two reasons: there's no strong evidence to assume that this will give us very good insight to our preferred ethical theories, and more importantly, because thinking in those terms will easily lead to akrasia.

I ended the article with the following paragraph:

My personal experience of late has also been that thinking in terms of "what does utilitarianism dictate I should do" produces recommendations that feel like external obligations, "shoulds" that are unlikely to get done; whereas thinking about e.g. the feelings of empathy that motivated me to become utilitarian in the first place produce motivations that feel like internal "wants". I was very close to (yet another) burnout and serious depression some weeks back: a large part of what allowed me to avoid it was that I stopped entirely asking the question of what I should do, and began to focus entirely on what I want to do, including the question of which of my currently existing wants are ones that I'd wish to cultivate further. (Of course there are some things like doing my tax returns that I do have to do despite not wanting to, but that's a question of necessity, not ethics.) It's way too short of a time to say whether this actually leads to increased productivity in the long term, but at least it feels great for my mental health, at least for the time being.

The long-term update (three years after first posting the article) is that starting to shift my thought patterns in this way was totally the right thing to do, and necessary for starting a long and slow recovery from depression. It's hard to say entirely for sure how big of a role this has played, since the patterns of should-thought were very deeply ingrained and have been slow to get rid of; I still occasionally find myself engaging in them. And there have been many other factors also affecting my recovery during this period, so only a part of the recovery can be attributed to the "utilitarianism-excising" with any certainty. Yet, whenever I've found myself engaging in such patterns of thought and managed to eliminate them, I have felt much better as a result. I do still remember a time when a large part of my waking-time was driven by utilitarian thinking, and it's impossible for me to properly describe how relieved I now feel for the fact that my mind feels much more peaceful now.

The other obvious question besides "do I feel better now" is "do I actually get more good things done now"; and I think that the answer is yes there as well. So I don't just feel generally better, I think my actions and motivations are actually more aligned with doing good than they were when I was trying to more explicitly optimize for following utilitarianism and doing good in that way. I still don't feel like I actually get a lot of good done, but I attribute much of this to still not having entirely recovered; I also still don't get a lot done that pertains to my own personal well-being. (I just spent several months basically doing nothing, because this was pretty much the first time when I had the opportunity, finance-wise, to actually take a long stressfree break from everything. It's been amazing, but even after such an extended break, the burnout symptoms still pop up if I'm not careful.)

LW UI issue

14 gworley 24 March 2017 06:08PM

Not really sure where else I might post this, but there seems to be a UI issue on the site. When I hit the homepage of lesswrong.com while logged in I no longer see the user sidebar or the header links for Main and Discussion. This is kind of annoying because I have to click into an article first to get to a page where I can access those things. Would be nice to have them back on the front page.

Musical setting of the Litany of Tarski

14 komponisto 23 March 2017 11:18AM

About a year ago, I made a setting of the Litany of Tarski for four-part a cappella (i.e. unaccompanied) chorus.

More recently, in the process of experimenting with MuseScore for potential use in explaining musical matters on the internet (it makes online sharing of playback-able scores very easy), the thought occurred to me that perhaps the Tarski piece might be of interest to some LW readers (if no one else!), so I went ahead and re-typeset it in MuseScore for your delectation. 

Here it is (properly notated :-)).

Here it is (alternate version designed to avoid freaking out those who aren't quite the fanatical enthusiasts of musical notation that I am).

In support of Yak Shaving

14 Elo 16 March 2017 05:31AM

Original post:  http://bearlamp.com.au/in-support-of-yak-shaving/

Part 2:  http://bearlamp.com.au/yak-shaving-2/


Yak shaving is heralded as pretty much "the devil" of trying to get things done.  The anti-yak shaving movement will identify this problem as being one of focus.  The moral of the story they give is "don't yak shave".

Originally posted in MIT's media lab with the description:

Any seemingly pointless activity which is actually necessary to solve a problem which solves a problem which, several levels of recursion later, solves the real problem you're working on.

But I prefer the story by Seth Godin:

"I want to wax the car today."

"Oops, the hose is still broken from the winter. I'll need to buy a new one at Home Depot."

"But Home Depot is on the other side of the Tappan Zee bridge and getting there without my EZPass is miserable because of the tolls."

"But, wait! I could borrow my neighbor's EZPass..."

"Bob won't lend me his EZPass until I return the mooshi pillow my son borrowed, though."

"And we haven't returned it because some of the stuffing fell out and we need to get some yak hair to restuff it."

And the next thing you know, you're at the zoo, shaving a yak, all so you can wax your car.

I disagree with the conclusion to not yak shave, and here's why.


The problem here is that you didn't wax the car because you spent all day shaving yaks (see also "there's a hole in my bucket").  In a startup that translates to not doing the tasks that get customers - the tasks which get money and actually make an impact, say "playing with the UI".  It's easy to see why such anti-yak shaving sentiment would exist (see also: bikeshedding, rearranging deck chairs on the titanic, hamming questions).  You can spend a whole day doing a whole lot of nothings; getting to bed and wonder what you actually accomplished that day (hint: a whole lot of running in circles).

Or at least that's what it looks like on the surface.  But let's look a little deeper into what the problems and barriers are in the classic scenario.

  1. Want to wax car
  2. Broken hose
  3. Hardware store is far away
  4. No EZpass for tolls
  5. Neighbour won't lend the pass until pillow is returned
  6. Broken mooshi pillow
  7. Have to go get yak hair.

So it's not just one problem, but a series of problems that come up in a sequence.  Hopefully by the end of the list you can turn around and walk all the way straight back up the list.  But in the real world there might even be other problems like, you get to the hardware store and realise you don't know the hose-fitting size of your house so you need to call someone at home to check...

On closer inspection; this sort of behaviour is not like bikeshedding at all.  Nor is it doing insignificant things under the guise of "real work".  Instead this is about tackling what stands in the way of your problem.  In problem solving in the real world, Don't yak shave" is not what I have found to be the solution.  In experiencing this the first time it feels like a sequence of discoveries.  For example, first you discover the hose.  Then you discover the EZpass problem, then you discover the pillow problem, at which point you are pretty sick of trying to wax your car and want a break or to work on something else.


I propose that classic yak shaving presents a very important sign that things are broken.  In order to get to the classic scenario we had to

  1. have borrowed a pillow from our neighbour,
  2. have it break and not get fixed,
  3. not own our own EZpass,
  4. live far from a hardware store,
  5. have a broken hose, and
  6. want to wax a car.  

Each open problem in this scenario presents an open problem or an open loop.  Yak shaving presents a warning sign that you are in a Swiss-cheese model scenario of problems.  This might sound familiar because it's the kind of situation which leads to the Fukushima reactor meltdown.  It's the kind of scenario when you try to work out why the handyman fell off your roof and died, and you notice that:

  1. he wasn't wearing a helmet.
  2. He wasn't tied on safely
  3. His ladder wasn't tied down
  4. It was a windy day
  5. His harness was old and worn out
  6. He was on his phone while on the roof...

And you realise that any five of those things could have gone wrong and not caused much of a problem.  But you put all six of those mistakes together and line the wind up in just the right way, everything comes tumbling down.


Yak shaving is a sign that you are living with problems waiting to crash down.  And living in a situation where you don't have time to do the sort of maintenance that would fix things and keep smoulders from bursting into flames.

I can almost guaranteed that when your house of cards all come falling down, it happens on a day that you don't have the spare time to waste on ridiculous seeming problems.


What should you do if you are in this situation?

Yak shave.  The best thing you can do if half your projects are unfinished and spread around the room is to tidy up.  Get things together; organise things, initiate the GTD system (or any system), wrap up old bugs, close the open loops (advice from GTD) and as many times as you can; YAK SHAVE for all you are worth!

If something is broken, and you are living with it, that's not acceptable.  You need a system in your life to regularly get around to fixing it.  Notepadsreviews, list keeping, set time aside for doing it and plan to fix things.

So I say, Yak Shave, as much, as long, and as many times as it takes till there are no more yaks to shave.


Something not mentioned often enough is a late addition to my list of common human goals.

Improve the tools available – sharpen the axe, write a new app that can do the thing you want, invent systems that work for you.  prepare for when the rest of the work comes along.

People often ask how you can plan for lucky breaks in your life.  How do you cultivate opportunity?  I can tell you right here and now, this is how.

Keep a toolkit at the ready, a work-space (post coming soon) at the ready, spare time for things to go wrong and things to go right.  And don't forget to play.  Why do we sharpen the axe?  Clear Epistemics, or clear Instrumental Rationality.  Be prepared for the situation that will come up.

Yak Shave like your life depends on it.  Because your life might one day depend on it.  Your creativity certainly does.


Meta: this took 2.5 hrs to write.

Thoughts on civilization collapse

13 Stuart_Armstrong 04 May 2017 10:41AM

Epistemic status: an idea I believe moderately strongly, based on extensive reading but not rigorous analysis.

We may have a dramatically wrong idea of civilization collapse, mainly inspired by movies that obsess over dramatic tales of individual heroism.

 

Traditional view:

In a collapse, anarchy will break out, and it will be a war of all against all or small groups against small groups. Individual weaponry (including heavy weapons) and basic food production will become paramount; traditional political skills, not so much. Government collapse is long term. Towns and cities will suffer more than the countryside. The best course of action is to have a cache of weapons and food, and to run for the hills.

 

Alternative view:

In a collapse, people will cling to their identified tribe for protection. Large groups will have no difficulty suppressing or taking over individuals and small groups within their areas of influence. Individual weaponry may be important (given less of a police force), but heavy weaponry will be almost irrelevant as no small group will survive alone. Food production will be controlled by the large groups. Though the formal "government" may fall, and countries may splinter into more local groups, government will continue under the control of warlords, tribal elders, or local variants. Cities, with their large and varied-skill workforce, will suffer less than the countryside. The best course of action is to have a stash of minor luxury goods (solar-powered calculators, comic books, pornography, batteries, antiseptics) and to make contacts with those likely to become powerful after a collapse (army officers, police chiefs, religious leaders, influential families).

Possible sources to back up this alternative view:

  • The book Sapiens argues that governments and markets are the ultimate enablers of individualism, with extended-family-based tribalism as the "natural" state of humanity.
  • The history of Somalia demonstrates that laws and enforcement continue even after a government collapse, by going back to more traditional structures.
  • During China's period of anarchy, large groups remained powerful: the nationalists, the communists, the Japanese invaders. The other sections of the country were generally under the control of local warlords.
  • Rational Wiki argues that examples of collapse go against the individualism narrative.

 

Bad intent is a disposition, not a feeling

13 Benquo 01 May 2017 01:28AM

It’s common to think that someone else is arguing in bad faith. In a recent blog post, Nate Soares claims that this intuition is both wrong and harmful:

I believe that the ability to expect that conversation partners are well-intentioned by default is a public good. An extremely valuable public good. When criticism turns to attacking the intentions of others, I perceive that to be burning the commons. Communities often have to deal with actors that in fact have ill intentions, and in that case it's often worth the damage to prevent an even greater exploitation by malicious actors. But damage is damage in either case, and I suspect that young communities are prone to destroying this particular commons based on false premises.

To be clear, I am not claiming that well-intentioned actions tend to have good consequences. The road to hell is paved with good intentions. Whether or not someone's actions have good consequences is an entirely separate issue. I am only claiming that, in the particular case of small high-trust communities, I believe almost everyone is almost always attempting to do good by their own lights. I believe that propagating doubt about that fact is nearly always a bad idea.

It would be surprising, if bad intent were so rare in the relevant sense, that people would be so quick to jump to the conclusion that it is present. Why would that be adaptive?

What reason do we have to believe that we’re systematically overestimating this? If we’re systematically overestimating it, why should we believe that it’s adaptive to suppress this?

There are plenty of reasons why we might make systematic errors on things that are too infrequent or too inconsequential to yield a lot of relevant-feeling training data or matter much for reproductive fitness, but social intuitions are a central case of the sort of things I would expect humans to get right by default. I think the burden of evidence is on the side disagreeing with the intuitions behind this extremely common defensive response, to explain what bad actors are, why we are on such a hair-trigger against them, and why we should relax this.

Nate continues:

My models of human psychology allow for people to possess good intentions while executing adaptations that increase their status, influence, or popularity. My models also don’t deem people poor allies merely on account of their having instinctual motivations to achieve status, power, or prestige, any more than I deem people poor allies if they care about things like money, art, or good food. […]

One more clarification: some of my friends have insinuated (but not said outright as far as I know) that the execution of actions with bad consequences is just as bad as having ill intentions, and we should treat the two similarly. I think this is very wrong: eroding trust in the judgement or discernment of an individual is very different from eroding trust in whether or not they are pursuing the common good.

Nate's argument is almost entirely about mens rea - about subjective intent to make something bad happen. But mens rea is not really a thing. He contrasts this with actions that have bad consequences, which are common. But there’s something in the middle: following an incentive gradient that rewards distortions. For instance, if you rigorously A/B test your marketing until it generates the presentation that attracts the most customers, and don’t bother to inspect why they respond positively to the result, then you’re simply saying whatever words get you the most customers, regardless of whether they’re true. In such cases, whether or not you ever formed a conscious intent to mislead, your strategy is to tell whichever lie is most convenient; there was nothing in your optimization target that forced your words to be true ones, and most possible claims are false, so you ended up making false claims.

More generally, if you try to control others’ actions, and don’t limit yourself to doing that by honestly informing them, then you’ll end up with a strategy that distorts the truth, whether or not you meant to. The default state for any given constraint is that it has not been applied to someone's behavior. To say that someone has the honest intent to inform is a positive claim about their intent. It's clear to me that we should expect this to sometimes be the case - sometimes people perceive a convergent incentive to inform one another, rather than a divergent incentive to grab control. But, if you do not defend yourself and your community against divergent strategies unless there is unambiguous evidence, then you make yourself vulnerable to those strategies, and should expect to get more of them.The default hypothesis should be that any given constraint has not been applied to someone's behavior. To say that someone has the honest intent to inform is a positive claim about their intent. It's clear to me that we should expect this to sometimes be the case - sometimes people have a convergent incentive to inform one another, rather than a divergent incentive to grab control. 

I’ve been criticizing EA organizations a lot for deceptive or otherwise distortionary practices (see here and here), and one response I often get is, in effect, “How can you say that? After all, I've personally assured you that my organization never had a secret meeting in which we overtly resolved to lie to people!”

Aside from the obvious problems with assuring someone that you're telling the truth, this is generally something of a nonsequitur. Your public communication strategy can be publicly observed. If it tends to create distortions, then I can reasonable infer that you’re following some sort of incentive gradient that rewards some kinds of distortions. I don’t need to know about your subjective experiences to draw this conclusion. I don’t need to know your inner narrative. I can just look, as a member of the public, and report what I see.

Acting in bad faith doesn’t make you intrinsically a bad person, because there’s no such thing. And besides, it wouldn't be so common if it required an exceptionally bad character. But it has to be OK to point out when people are not just mistaken, but following patterns of behavior that are systematically distorting the discourse - and to point this out publicly so that we can learn to do better, together.

(Cross-posted at my personal blog.)

[EDITED 1 May 2017 - changed wording of title from "behavior" to "disposition"]

Akrasia Tactics Review 3: The Return of the Akrasia

13 malcolmocean 10 April 2017 03:05PM

About three and a half years ago, polutropon ran an akrasia tactics review, following the one orthonormal ran three and a half years prior to that: an open-ended survey asking Less Wrong posters to give numerical scores to productivity techniques that they'd tried, with the goal of getting a more objective picture of how well different techniques work (for the sort of people who post here). Since it's been years since the others and the rationality community has grown and developed significantly while retaining akrasia/motivation/etc as a major topic, I thought it'd be useful to have a new one!

(Malcolm notes: it seems particularly likely that this time there are likely to be some noteworthy individually-invented techniques this time, as people seem to be doing a lot of that sort of thing these days!)

A lightly modified version of the instructions from the previous post:

  1. Note what technique you've tried. Techniques can be anything from productivity systems (Getting Things Done, Complice) to social incentives (precommitting in front of friends) to websites or computer programs (Beeminder, Leechblock) to chemical aids (Modafinil, Caffeine). If it's something that you can easily link to information about, please provide a link and I'll add it when I list the technique; if you don't have a link, describe it in your comment and I'll link that. It could also be a cognitive technique you developed or copied from a friend, which might not have a clear name but you can give it one if you like!
  2. Give your experience with it a score from -10 to +10 (0 if it didn't change the status quo, 10 if it ended your akrasia problems forever with no unwanted side effects, negative scores if it actually made your life worse, -10 if it nearly killed you). For simplicity's sake, I'll only include reviews that give numerical scores.
  3. Describe your experience with it, including any significant side effects. Please also say approximately how long you've been using it, or if you don't use it anymore how long you used it before giving up.

Every so often, I'll combine all the data back into the main post, listing every technique that's been reviewed at least twice with the number of reviews, average score, standard deviation and common effects. I'll do my best to combine similar techniques appropriately, but it'd be appreciated if you could try to organize it a bit by replying to people doing similar things and/or saying if you feel your technique is (dis)similar to another.

I'm not going to provide an initial list due to the massive number of possible techniques and concern of prejudicing answers, but you can look back on the list in the last post or the previous one one if you want. If you have any suggestions for how to organize this (that wouldn't require huge amounts of extra effort on my part), I'm open to hearing them.

Thanks for your data!

(There's a meta thread here for comments that aren't answers to the main prompt.)

[Link] "On the Impossibility of Supersized Machines"

13 crmflynn 31 March 2017 11:32PM

Use and misuse of models: case study

12 Stuart_Armstrong 27 April 2017 02:36PM

Some time ago, I discovered a post comparing basic income and basic job ideas. This sought to analyse the costs of paying everyone a guaranteed income versus providing them with a basic job with that income. The author spelt out his assumptions and put together a two models with a few components (including some whose values were drawn from various probability distributions). Then he ran a Monte Carlo simulation to get a distribution of costs for either policy.

Normally I should be very much in favour of this approach. It spells out the assumptions, it uses models, it decomposes the problem, it has stochastic uncertainty... Everything seems ideal. To top it off, the author concluded with a challenge aiming at improving reasoning around this subject:

How to Disagree: Write Some Code

This is a common theme in my writing. If you are reading my blog you are likely to be a coder. So shut the fuck up and write some fucking code. (Of course, once the code is written, please post it in the comments or on github.)

I've laid out my reasoning in clear, straightforward, and executable form. Here it is again. My conclusions are simply the logical result of my assumptions plus basic math - if I'm wrong, either Python is computing the wrong answer, I got really unlucky in all 32,768 simulation runs, or you one of my assumptions is wrong.

My assumption being wrong is the most likely possibility. Luckily, this is a problem that is solvable via code.

And yet... I found something very unsatisfying. And it took me some time to figure out why. It's not that these models are helpful, or that they're misleading. It's that they're both simultaneously.

To explain, consider the result of the Monte Carlo simulations. Here are the outputs (I added the red lines; we'll get to them soon):

The author concluded from these outputs that a basic job was much more efficient - less costly - than a basic income (roughly 1 trillion cost versus 3.4 trillion US dollars). He changed a few assumptions to test whether the result held up:

For example, maybe I'm overestimating the work disincentive for Basic Income and grossly underestimating the administrative overhead of the Basic Job. Lets assume both of these are true. Then what?

The author then found similar results, with some slight shifting of the probability masses.

 

The problem: what really determined the result

So what's wrong with this approach? It turns out that most of the variables in the models have little explanatory power. For the top red line, I just multiplied the US population by the basic income. The curve is slightly above it, because it includes such things as administrative costs. The basic job situation was slightly more complicated, as it includes a disabled population that gets the basic income without working, and a estimate for the added value that the jobs would provide. So the bottom red line is (disabled population)x(basic income) + (unemployed population)x(basic income) - (unemployed population)x(median added value of jobs). The distribution is wider than for basic income, as the added value of the jobs is a stochastic variable.

But, anyway, the contribution of the other variables were very minor. So the reduced cost of basic jobs versus basic income is essentially a consequence of the trivial fact that it's more expensive to pay everyone an income, than to only pay some people and then put them to work at something of non-zero value.

 

Trees and forests

So were the complicated extra variables and Monte Carlo runs for nothing? Not completely - they showed that the extra variables were indeed of little importance, and unlikely to change the results much. But nevertheless, the whole approach has one big, glaring flaw: it does not account for the extra value for individuals of having a basic income versus a basic job.

And the challenge - "write some fucking code" - obscures this. The forest of extra variables and the thousands of runs hides the fact that there is a fundamental assumption missing. And pointing this out is enough to change the result, without even needing to write code. Note this doesn't mean the result is wrong: some might even argue that people are better off with a job than with the income (builds pride in one's work, etc...). But that needs to be addressed.

So Chris Stucchio's careful work does show one result - most reasonable assumptions do not change the fact that basic income is more expensive than basic job. And to disagree with that, you do indeed need to write some fucking code. But the stronger result - that basic job is better than basic income - is not established by this post. A model can be well designed, thorough, filled with good uncertainties, and still miss the mark. You don't always have to enter into the weeds of the model's assumptions in order to criticise it.

That is not dead which can eternal lie: the aestivation hypothesis for resolving Fermi's paradox

11 Stuart_Armstrong 11 May 2017 09:16AM

That is not dead which can eternal lie: the aestivation hypothesis for resolving Fermi's paradox

Anders Sandberg, Stuart Armstrong, Milan M. Cirkovic

If a civilization wants to maximize computation it appears rational to aestivate until the far future in order to exploit the low temperature environment: this can produce a 1030 multiplier of achievable computation. We hence suggest the "aestivation hypothesis": the reason we are not observing manifestations of alien civilizations is that they are currently (mostly) inactive, patiently waiting for future cosmic eras. This paper analyzes the assumptions going into the hypothesis and how physical law and observational evidence constrain the motivations of aliens compatible with the hypothesis.

As far as I can tell, the paper's physics is correct (most of the energy comes not from burning stars but from the universe's mass).

However, the conclusions are likely wrong, because it's rational for "sleeping" civilizations to still want to round up stars that might be ejected from galaxies, collect cosmic dust, and so on.

The paper is still worth publishing, though, because there may other, more plausible ideas in the vicinity of this one. And it describes how future civilization may choose to use their energy.

SlateStarCodex Meetups Everywhere: Analysis

10 mingyuan 13 May 2017 12:29AM

The first round of SlateStarCodex meetups took place from April 4th through May 20th, 2017 in 65 cities, in 16 countries around the world. Of the 69 cities originally listed as having 10 or more people interested, 9 did not hold meetups, and 5 cities that were not on the original list did hold meetups.

We collected information from 43 of these events. Since we are missing data for 1/3 of the cities, there is probably some selection bias in the statistics; I would speculate that we are less likely to have data from less successful meetups.

Of the 43 cities, 25 have at least tentative plans for future meetups. Information about these events will be posted at the SSC Meetups GitHub.

 

Turnout

Attendance ranged from 3 to approximately 50 people, with a mean of 16.7. Turnout averaged about 50% of those who expressed interest on the survey (range: 12% to 100%), twice what Scott expected. This average does not appear to have been skewed by high turnout at a few events – mean: 48%, median: 45%, mode: 53%.

On average, gender ratio seemed to be roughly representative of SSC readership overall, ranging from 78% to 100% male (for the 5 meetups that provided gender data). The majority of attendees were approximately 20-35 years old, consistent with the survey mean age of 30.6.

 

Existing vs new meetups

Approximately one fifth of the SSC meetups were hosted by existing rationality or LessWrong groups. Some of these got up to 20 new attendees from the SSC announcement, while others saw no new faces at all. The two established meetups that included data about follow-up meetings reported that retention rates for new members were very low, at best 17% for the next meeting.

Here, it seems important to make a distinction between the needs of SSC meetups specifically and rationality meetups more generally. On the 2017 survey, 50% of readers explicitly did not identify with LW and 54% explicitly did not identify with EA. In addition, one organizer expressed the concern that, “Going forward, I think there is a concern of “rationalists” with a shared background outnumbering the non-lesswrong group, and dominating the SSC conversation, making new SSC fans less likely to engage.”

This raises the question of whether SSC groups should try to exist separately from local EA/LW/rationalist/skeptic groups – this is of particular concern in locations where the community is small and it’s difficult for any of these groups to function on their own due to low membership.

Along the same lines, one organizer wondered how often it made sense to hold events, since “If meetups happen very frequently, they will be attended mostly by hardcore fans (and a certain type of person), while if they are scheduled less frequently, they are likely to be attended by a larger, more diverse group. My fear is the hardcore fans who go bi-weekly will build a shared community that is less welcoming/appealing to outsiders/less involved people, and these people will be less willing to get involved going forward.”

Suggestions on how to address these concerns are welcome.

 

Advice for initial meetings

Bring name tags, and collect everyone’s email addresses. It’s best to do this on a computer or tablet, since some people have illegible handwriting, and you don’t want their orthographic deficiencies to mean you lose contact with them forever.

Don’t try to impose too much structure on the initial meeting, since people will mostly just want to get to know each other and talk about shared interests. If possible, it’s also good to not have a hard time limit - meetups in this round lasted between 1.5 and 6 hours, and you don’t want to have to make people leave before they’re ready. However, both structure and time limits are things you will most likely want if you have regularly recurring meetups.

 

Content

Most meetups consisted of unstructured discussion in smallish groups (~7 people). At least one organizer had people pair up and ask each other scripted questions, while another used lightning talks as an ice-breaker. Other activities included origami, Rationality Cardinality, and playing with magnadoodles and diffraction glasses, but mostly people just wanted to talk.

Topics, predictably, mostly centered around shared interests, and included: SSC and other rationalist blogs, rationalist fiction, the rationality community, AI, existential risk, politics and meta-politics, book recommendations, and programming (according to the survey, 30% of readers are programmers), as well as normal small talk and getting-to-know-each-other topics.

Common ice-breakers included first SSC post read, how people found SSC, favorite SSC post, and SSC vs LessWrong (aka, is Eliezer or Scott the rightful caliph).

Though a few meetups had a little difficulty getting conversation started and relied on ice-breakers and other predetermined topics, no organizer reported prolonged awkwardness; people had a lot to talk about and conversation flowed quite easily for the most part.

One area where several organizers encountered difficulties was large discrepancies in knowledge of rationalist-sphere topics among attendees, since some people had only recently discovered SSC or were even non-readers brought along by friends, while many others were long-time members of the community. Suggestions for quickly and painlessly bridging inferential gaps on central concepts in the community would be appreciated.

 

Locations 

Meetups occurred in diverse locations, including restaurants, cafés, pubs/bars, private residences, parks, and meeting rooms in coworking spaces or on university campuses.

Considerations for choosing a venue:

  • Capacity – Some meetups found that their original venues couldn’t accommodate the number of people who attended. This happened at a private residence and at a restaurant. Be flexible about moving locations if necessary.
  • Arrangement – For social meetups, you will probably want a more flexible format. For this purpose, it’s best to have the run of the space, which you have in private residences, parks, meeting rooms, and bars and restaurants if you reserve a whole room or floor.
  • Noise – Since the main activity is talking, this is an important consideration. An ideal venue is quiet enough that you can all hear each other, but (if public) not so quiet that you will be disrupting others with your conversation.
  • Visibility – If meeting in a public place, have a somewhat large sign that says ‘SSC’ on it, placed somewhere easily visible. If the location is large or hard to find, consider including your specific location (e.g. ‘we’re at the big table in the northwest corner’) or GPS coordinates in the meetup information.
  • Permission – Check with the manager first if you plan to hold a large meetup in a private building, such as a mall, market, or café. Also consider whether you’ll be disturbing other patrons.
  • Time restrictions – If you are reserving a space, or if you are meeting somewhere that has a closing time, be aware that people may want to continue their discussions for longer than the space is available. Have a contingency plan for this, a second location to move to in case you run overtime.
  • Availability of food – Some meetups lasted as long as six hours, so it’s good to either bring food, meet somewhere with easy access to food, or be prepared to go to a restaurant.
  • Privacy – People at some meetups were understandably hesitant to have controversial / culture war discussions in public. If you anticipate this being a problem, you should try to find a more private venue, or a more secluded area.

Conclusion

Overall most meetups went smoothly, and many had unexpectedly high turnout. Almost every single organizer, even for the tiny meetups, reported that attendees showed interest in future meetings, but few had concrete plans.

These events have been an important first step, but it remains to be seen whether they will lead to lasting local communities. The answer is largely up to you.

If you attended a meetup, seek out the people you had a good time talking to, and make sure you don’t lose contact with them. If you want there to be more events, just set a time and place and tell people. You can share details on local Facebook groups, Google groups, and email lists, and on LessWrong and the SSC meetups repository. If you feel nervous about organizing a meetup, don’t worry, there are plenty of resources just for that. And if you think you couldn’t possibly be an organizer because you’re somehow ‘not qualified’ or something, well, I once felt that way too. In Scott’s words, “it would be dumb if nobody got to go to meetups because everyone felt too awkward and low-status to volunteer.”

Finally, we’d like to thank Scott for making all of this possible. One of the most difficult things about organizing meetups is that it’s hard to know where to look for members, even if you know there must be dozens of interested people in your area. This was an invaluable opportunity to overcome that initial hurdle, and we hope that you all make the most of it.

 

Thanks to deluks917 for providing feedback on drafts of this report, and for having the idea to collect data in the first place :)

Net Utility and Planetary Biocide

10 madhatter 09 April 2017 03:43AM

I've started listening to the audiobook of Peter Singer's Ethics in the Real World, which is both highly recommended and very unsettling. The essays on non-human animals, for example, made me realize for the first time that it may well be possible that the net utility on Earth over all conscious creatures is massively negative. 

Naturally, this led me to wonder whether, after all, efforts to eradicate all consciousness on Earth - human and non-human - may be ethically endorsable.This, in turn, reminded me of a recent post on LW asking whether the possibility of parallelized torture of future uploads justifies killing as many people as possible today. 

I had responded to that post by mentioning that parallelizing euphoria was also possible, so this should cancel things out. This seemed at the time like a refutation, but I realized later I had made the error of equating the two, utility and disutility, as part of the same smooth continuum, like [-100, 100] ∈ R. There is no reason to believe the maximum disutility I can experience is equal in magnitude to the maximum utility I can experience. It may be that max disutility is far greater. I really don't know, and I don't think introspection is as useful in answering this question as it seems intuitively to be, but it seems quite plausible for this to be the case.

As these thoughts were emerging, Singer, as if hearing my concerns, quoted someone or other who claimed that the human condition is one of perpetual suffering, constantly seeking desires which, once fulfilled, are ephemeral and dissatisfying, and therefore it is a morally tragic outcome for any of us to have emerged into existence. 

Of course these are shoddy arguments in support of Mass Planetary Biocide, even supposing the hypothesis that the Earth (universe?) has net negative utility is true. For one, we can engineer minds somewhere in a better neighborhood of mindspace, where utility is everywhere positive. Or maybe it's impossible even in theory to treat utility and disutility like real-valued functions of physical systems over time (though I'm betting it is). Or maybe the universe is canonically infinite, so even if 99% of conscious experiences in the universe have disutility, there are infinite quantities of both utility and disutility and so nothing we do matters, as Bostrom wrote about. (Although this is actually not an argument against MPB, just not one for it). And anyway, the state of net utility today is not nearly as important as the state of net utility could potentially be in the future. And perhaps utilitarianism is a naive and incorrect ethical framework. 

Still, I had somehow always assumed implicitly that net utility of life on Earth was positive, so the realization that this need not be so is causing me significant disutility. 

 

Against responsibility

10 Benquo 31 March 2017 09:12PM

I am surrounded by well-meaning people trying to take responsibility for the future of the universe. I think that this attitude – prominent among Effective Altruists – is causing great harm. I noticed this as part of a broader change in outlook, which I've been trying to describe on this blog in manageable pieces (and sometimes failing at the "manageable" part).

I'm going to try to contextualize this by outlining the structure of my overall argument.

Why I am worried

Effective Altruists often say they're motivated by utilitarianism. At its best, this leads to things like Katja Grace's excellent analysis of when to be a vegetarian. We need more of this kind of principled reasoning about tradeoffs.

At its worst, this leads to some people angsting over whether it's ethical to spend money on a cup of coffee when they might have saved a life, and others using the greater good as license to say things that are not quite true, socially pressure others into bearing inappropriate burdens, and make ever-increasing claims on resources without a correspondingly strong verified track record of improving people's lives. I claim that these actions are not in fact morally correct, and that people keep winding up endorsing those conclusions because they are using the wrong cognitive approximations to reason about morality.

Summary of the argument

  1. When people take responsibility for something, they try to control it. So, universal responsibility implies an attempt at universal control.
  2. Maximizing control has destructive effects:
    • An adversarial stance towards other agents.
    • Decision paralysis.
  3. These failures are not accidental, but baked into the structure of control-seeking. We need a practical moral philosophy to describe strategies that generalize better, and benefit from the existence of other benevolent agents, rather than treating them primarily as threats.

Responsibility implies control

In practice, the way I see the people around me applying utilitarianism, it seems to make two important moral claims:

  1. You - you, personally - are responsible for everything that happens.
  2. No one is allowed their own private perspective - everyone must take the public, common perspective.

The first principle is almost but not quite simple consequentialism. But it's important to note that it actually doesn't generalize; it's massive double-counting if each individual person is responsible for everything that happens. I worked through an example of the double-counting problem in my post on matching donations.

The second principle follows from the first one. If you think you're personally responsible for everything that happens, and obliged to do something about that rather than weigh your taste accordingly – and you also believe that there are ways to have an outsized impact (e.g. that you can reliably save a life for a few thousand dollars) – then in some sense nothing is yours. The money you spent on that cup of coffee could have fed a poor family for a day in the developing world. It's only justified if the few minutes you save somehow produce more value.

One way of resolving this is simply to decide that you're entitled to only as much as the global poor, and try to do without the rest to improve their lot. This is the reasoning behind the notorious demandingness of utilitarianism.

But of course, other people are also making suboptimal uses of resources. So if you can change that, then it becomes your responsibility to do so.

In general, if Alice and Bob both have some money, and Alice is making poor use of money by giving to the Society to Cure Rare Diseases in Cute Puppies, and Bob is giving money to comparatively effective charities like the Against Malaria Foundation, then if you can cause one of them to have access to more money, you'd rather help Bob than Alice.

There's no reason for this to be different if you are one of Bob and Alice. And since you've already rejected your own private right to hold onto things when there are stronger global claims to do otherwise, there's no principled reason not to try to reallocate resources from the other person to you.

What you're willing to do to yourself, you'll be willing to do to others. Respecting their autonomy becomes a mere matter of either selfishly indulging your personal taste for "deontological principles," or a concession made because they won't accept your leadership if you're too demanding - not a principled way to cooperate with them. You end up trying to force yourself and others to obey your judgment about what actions are best.

If you think of yourself as a benevolent agent, and think of the rest of the world and all the people in it in as objects with regular, predictable behaviors you can use to improve outcomes, then you'll feel morally obliged - and therefore morally sanctioned - to shift as much of the locus of control as possible to yourself, for the greater good.

If someone else seems like a better candidate, then the right thing to do seems like throwing your lot in with them, and transferring as much as you can to them rather than to yourself. So this attitude towards doing good leads either to personal control-seeking, or support of someone else's bid for the same.

I think that this reasoning is tacitly accepted by many Effective Altruists, and explains two seemingly opposite things:

  1. Some EAs get their act together and make power plays, implicitly claiming the right to deceive and manipulate to implement their plan.
  2. Some EAs are paralyzed by the impossibility of weighing the consequences for the universe of every act, and collapse into perpetual scrupulosity and anxiety, mitigated only by someone else claiming legitimacy, telling them what to do, and telling them how much is enough.

Interestingly, people in the second category are somewhat useful for people following the strategy of the first category, as they demonstrate demand for the service of telling other people what to do. (I think the right thing to do is largely to decline to meet this demand.)

Objectivists sometimes criticize "altruistic" ventures by insisting on Ayn Rand's definition of altruism as the drive to self-abnegation, rather than benevolence. I used to think that this was obnoxiously missing the point, but now I think this might be a fair description of a large part of what I actually see. (I'm very much not sure I'm right. I am sure I'm not describing all of Effective Altruism – many people are doing good work for good reasons.)

Control-seeking is harmful

You have to interact with other people somehow, since they're where most of the value is in our world, and they have a lot of causal influence on the things you care about. If you don't treat them as independent agents, and you don't already rule over them, you will default to going to war against them (and more generally trying to attain control and then make all the decisions) rather than trading with them (or letting them take care of a lot of the decisionmaking). This is bad because it destroys potential gains from trade and division of labor, because you win conflicts by destroying things of value, and because even when you win you unnecessarily become a bottleneck.

People who think that control-seeking is the best strategy for benevolence tend to adopt plans like this:

Step 1 – acquire control over everything.

Step 2 – optimize it for the good of all sentient beings.

The problem with this is that step 1 does not generalize well. There are lots of different goals for which step 1 might seem like an appealing first step, so you should expect lots of other people to be trying, and their interests will all be directly opposed to yours. Your methods will be nearly the same as the methods for someone with a different step 2. You'll never get to step 2 of this plan; it's been tried many times before, and failed every time.

Lots of different types of people want more resources. Many of them are very talented. You should be skeptical about your ability to win without some massive advantage. So, what you're left with are your proximate goals. Your impact on the world will be determined by your means, not your ends.

What are your means?

Even though you value others' well-being intrinsically, when pursuing your proximate goals, their agency mostly threatens to muck up your plans. Consequently, it will seem like a bad idea to give them info or leave them resources that they might misuse.

You will want to make their behavior more predictable to you, so you can influence it better. That means telling simplified stories designed to cause good actions, rather than to directly transmit relevant information. Withholding, rather than sharing, information. Message discipline. I wrote about this problem in my post on the humility argument for honesty.

And if the words you say are tools for causing others to take specific actions, then you're corroding their usefulness for literally true descriptions of things far away or too large or small to see. Peter Singer's claim that you can save a life for hundreds of dollars by giving to developing-world charities no longer means that you can save a life for hundreds of dollars by giving to developing-world charities. It simply means that Peter Singer wants to motivate you to give to developing-world charities. I wrote about this problem in my post on bindings and assurances.

More generally, you will try to minimize others' agency. If you believe that other people are moral agents with common values, then e.g. withholding information means that the friendly agents around you are more poorly informed, which is obviously bad, even before taking into account trust considerations! This plan only makes sense if you basically believe that other people are moral patients, but independent, friendly agents do not exist; that you are the only person in the world who can be responsible for anything.

Another specific behavioral consequence is that you'll try to acquire resources even when you have no specific plan for them. For instance, GiveWell's impact page tracks costs they've imposed on others – money moved, and attention in the form of visits to their website – but not independent measures of outcomes improved, or the opportunity cost of people who made a GiveWell-influenced donation. The implication is that people weren't doing much good with their money or time anyway, so it's a "free lunch" to gain control over these.<fn>Their annual metrics report goes into more detail and does track this, and finds that about a quarter of GiveWell-influenced donations were reallocated from other developing-world charities (and another quarter from developed-world charities).</fn> By contrast, the Gates foundation's Valentine's day report to Warren Buffet tracks nothing but developing-world outcomes (but then absurdly takes credit for 100% of the improvement).

As usual, I'm not picking on GiveWell because they're unusually bad – I'm picking on GiveWell because they're unusually open. You should assume that similar but more secretive organizations are worse by default, not better.

This kind of divergent strategy doesn't just directly inflict harms on other agents. It takes resources away from other agents that aren't defending themselves, which forces them into a more adversarial stance. It also earns justified mistrust, which means that if you follow this strategy, you burn cooperative bridges, forcing yourself farther down the adversarial path.

I've written more about the choice between convergent and divergent strategies in my post about the neglectedness consideration.

Simple patches don't undo the harms from adversarial strategies

Since you're benevolent, you have the advantage of a goal in common with many other people. Without abandoning your basic acquisitive strategy, you could try to have a secret handshake among people trying to take over the world for good reasons rather than bad. Ideally, this would let the benevolent people take over the world, cooperating among themselves. But, in practice, any simple shibboleth can be faked; anyone can say they're acquiring power for the greater good.

It's a commonplace in various discussions among Effective Altruists, when someone identifies an individual or organization doing important work, to suggest that we "persuade them to become an EA" or "get an EA in the organization", rather than directly about ways to open up a dialogue and cooperate. This is straightforwardly an attempt to get them to agree to the same shibboleths in order to coordinate on a power-grabbing strategy. And yet, the standard of evidence we're using is mostly "identifies as an EA".

When Gleb Tsipursky tried to extract resources from the Effective Altruism movement with straightforward low-quality mimesis, mouthing the words but not really adding value, and grossly misrepresenting what he was doing and his level of success, it took EAs a long time to notice the pattern of misbehavior. I don't think this is because Gleb is especially clever, or because EAs are especially bad at noticing things. I think this is because EAs identify each other by easy-to-mimic shibboleths rather than meaningful standards of behavior.

Nor is Effective Altruism unique in suffering from this problem. When the Roman empire became too big to govern, gradually emperors hit upon the solution of dividing the empire in two and picking someone to govern the other half. This occasionally worked very well, when the two emperors had a strong preexisting bond, but generally they distrusted each other enough that the two empires behaved like rival states as often as they behaved like allies. Even though both emperors were Romans, and often close relatives!

Using "believe me" as our standard of evidence will not work out well for us. The President of the United States seems to have followed the strategy of saying the thing that's most convenient, whether or not it happens to be true, and won an election based on this. Others can and will use this strategy against us.

We can do better

The above is all a symptom of not including other moral agents in your model of the world. We need a moral theory that takes this into account in its descriptions (rather than having to do a detailed calculation each time), and yet is scope-sensitive and consequentialist the way EAs want to be.

There are two important desiderata for such a theory:

  1. It needs to take into account the fact that there are other agents who also have moral reasoning. We shouldn't be sad to learn that others reason the way we do.
  2. Graceful degradation. We can't be so trusting that we can be defrauded by anyone willing to say they're one of us. Our moral theory has to work even if not everyone follows it. It should also degrade gracefully within an individual – you shouldn't have to be perfect to see benefits.

One thing we can do now is stop using wrong moral reasoning to excuse destructive behavior. Until we have a good theory, the answer is we don't know if your clever argument is valid.

On the explicit and systematic level, the divergent force is so dominant in our world that sincere benevolent people simply assume, when they see someone overtly optimizing for an outcome, that this person is optimizing for evil. This leads to perceptive people who don’t like doing harm, like Venkatesh Rao, to explicitly advise others to minimize their measurable impact on the world.

I don't think this impact-minimization is right, but on current margins it's probably a good corrective.

One encouraging thing is that many people using common-sense moral reasoning already behave according to norms that respect and try to cooperate with the moral agency of others. I wrote about this in Humble Charlie.

I've also begun to try to live up to cooperative heuristics even if I don't have all the details worked out, and help my friends do the same. For instance, I'm happy to talk to people making giving decisions, but usually I don't go any farther than connecting them with people they might be interested in, or coaching them through heuristics, because doing more would be harmful, it would destroy information, and I'm not omniscient, otherwise I'd be richer.

A movement like Effective Altruism, explicitly built around overt optimization, can only succeed in the long run at actually doing good with (a) a clear understanding of this problem, (b) a social environment engineered to robustly reject cost-maximization, and (c) an intellectual tradition of optimizing only for actually good things that people can anchor on and learn from.

This was only a summary. I don't expect many people to be persuaded by this alone. I'm going to fill in the details in the future posts. If you want to help me write things that are relevant, you can respond to this (preferably publicly), letting me know:

  • What seems clearly true?
  • Which parts seem most surprising and in need of justification or explanation?

(Cross-posted at my personal blog.)

IQ and Magnus Carlsen, Leo Messi and the Decathlon

10 ragintumbleweed 28 March 2017 10:37PM

[Epistemic Status: I suspect that this is at least partially wrong. But I don’t know why yet, and so I figured I’d write it up and let people tell me. First post on Less Wrong, for what that’s worth.] 

First thesis: IQ is more akin to a composite measure of performance such as the decathlon than it is to a single characteristic such as height or speed.

Second thesis: When looking at extraordinary performance in any specific field, IQ will usually be highly correlated with success, but it will not fully explain or predict top-end performance, because extraordinary performance in a specific field is a result of extraordinary talent in a sub-category of intelligence (or even a sub-category of a sub-category), rather than truly top-end achievement in the composite metric.                                         

Before we go too far, here are some of the things I’m not arguing:

  • IQ is largely immutable (though perhaps not totally immutable).
  • IQ is a heritable, polygenic trait.
  • IQ is highly correlated with a variety of achievement measures, including academic performance, longevity, wealth, happiness, and health.
  • That parenting and schooling matter far less than IQ in predicting performance.
  • That IQ matters more than “grit” and “mindset” when explaining performance.
  • Most extraordinary performers, from billionaire tech founders to chess prodigies, to writers and artists and musicians, will possess well-above-average IQ.[1]

Here is one area why I’m certain I’m in the minority:

Here is the issue where I’m not sure if my opinion is controversial, and thus why I’m writing to get feedback:

  • While IQ is almost certainly highly correlated with high-end performance, IQ fails a metric to explain or, more importantly, to predict top-end individual performance (the Second Thesis).

Why IQ Isn’t Like Height

Height is a single, measurable characteristic. Speed over any distance is a single, measurable characteristic. Ability to bench-press is a single, measurable characteristic.

But intelligence is more like the concept of athleticism than it is the concept of height, speed, or the ability to bench-press.  

Here is an excerpt from the Slate Star Codex article Talents part 2, Attitude vs. Altitude:

The average eminent theoretical physicist has an IQ of 150-160. The average NBA player has a height of 6’ 7”. Both of these are a little over three standard deviations above their respective mean. Since z-scores are magic and let us compare unlike domains, we conclude that eminent theoretical physicists are about as smart as pro basketball players are tall.

Any time people talk about intelligence, height is a natural sanity check. It’s another strongly heritable polygenic trait which is nevertheless susceptible to environmental influences, and which varies in a normal distribution across the population – but which has yet to accrete the same kind of cloud of confusion around it that IQ has.

All of this is certainly true. But here’s what I’d like to discuss more in depth:

Height is a trait that can be measured in a single stroke. IQ has to be measured by multiple sub-tests.

IQ measures the following sub-components of intelligence:

  •  Verbal Intelligence
  • Mathematical Ability
  • Spatial Reasoning Skills
  • Visual/Perceptual Skills
  • Classification Skills
  • Logical Reasoning Skills
  • Pattern Recognition Skills[2]

Even though both height and intelligence are polygenic traits, there is a category difference between two.

That’s why I think that athleticism is a better polygenic-trait-comparator to intelligence than height. Obviously, people are born with different degrees of athletic talent. Athleticism can be affected by environmental factors (nutrition, lack of access to athletic facilities, etc.). Athleticism, like intelligence, because it is composed of different sub-variables (speed, agility, coordination – verbal intelligence, mathematical intelligence, spatial reasoning skills), can be measured in a variety of ways. You could measure athleticism with an athlete’s performance in the decathlon, or you could measure it with a series of other tests. Those results would be highly correlated, but not identical. And those results would probably be highly correlated with lots of seemingly unrelated but important physical outcomes.

Measure intelligence with an LSAT vs. IQ test vs. GRE vs. SAT vs. ACT vs. an IQ test from 1900 vs. 1950 vs. 2000 vs. the blink test, and the results will be highly correlated, but again, not identical.

Whether you measure height in centimeters or feet, however, the ranking of the people you measure will be identical no matter how you measure it.

To me, that distinction matters.

I think this athleticism/height distinction explains part (but not all) of the “cloud” surrounding IQ.[3]

Athletic Quotient (“AQ”)

Play along with me for a minute.

Imagine we created a single, composite metric to measure overall athletic ability. Let’s call it AQ, or Athletic Quotient. We could measure AQ just as we measure IQ, with 100 as the median score, and with two standard deviations above at 130 and four standard deviations above at 160.

For the sake of simplicity, let’s measure athletes’ athletic ability with the decathlon. This event is an imperfect test of speed, strength, jumping ability, and endurance.

An Olympic-caliber decathlete could compete at a near-professional level in most sports. But the best decathletes aren’t the people whom we think of when we think of the best athletes in the world. When we think of great athletes, we think of the top performers in one individual discipline, rather than the composite.

When people think of the best athlete in the world, they think of Leo Messi or Lebron James, not Ashton Eaton.

IQ and Genius

Here’s where my ideas might start to get controversial.

I don’t think most of the people we consider geniuses necessarily had otherworldly IQs. People with 200-plus IQs are like Olympic decathletes. They’re amazingly intelligent people who can thrive in any intellectual environment. They’re intellectual heavyweights without specific weaknesses. But those aren’t necessarily the superstars of the intellectual world. The Einsteins, Mozarts, Picassos, or the Magnus Carlsens of the world – they’re great because of domain-specific talent, rather than general intelligence. 

Phlogiston and Albert Einstein’s IQ

Check out this article.

The article declares, without evidence, that Einstein had an IQ of 205-225.

The thinking seems to go like this: Most eminent physicists have IQs of around 150-160. Albert Einstein created a paradigm shift in physics (or perhaps multiple such shifts). So he must have had an IQ around 205-225. We’ll just go ahead and retroactively apply that IQ to this man who’s been dead for 65 years and that’ll be great for supporting the idea that IQ and high-end field-specific performance are perfectly correlated.

As an explanation of intelligence, that’s no more helpful than phlogiston in chemistry.

But here’s the thing: It’s easy to ascribe super-high IQs retroactively to highly accomplished dead people, but I have never heard of IQ predicting an individual’s world-best achievement in a specific field. I have never read an article that says, “this kid has an IQ of 220; he’s nearly certain to create a paradigm-shift in physics in 20 years.” There are no Nate Silvers predicting individual achievement based on IQ. IQ does not predict Nobel Prize winners or Fields Medal winners or the next chess #1. A kid with a 220 IQ may get a Ph.D. at age 17 from CalTech, but that doesn’t mean he’s going to be the next Einstein.

Einstein was Einstein because he was an outsider. Because he was intransigent. Because he was creative. Because he was an iconoclast. Because he had the ability to focus. But there is no evidence that he had an IQ over 200. But according to the Isaacson biography at least, there were other pre-eminent physicists who were stronger at math than he was. Of course he was super smart. But there's no evidence he had a super-high IQ (as in, above 200).

We’ve been using IQ as a measure of intelligence for over 100 years and it has never predicted an Einstein, a Musk, or a Carlsen.[4] Who is the best counter-example to this argument? Terence Tao? Without obvious exception, those who have been recognized for early-age IQ are still better known for their achievements as prodigies than their achievements as adults.

Is it unfair to expect that predictive capacity from IQ? Early-age prediction of world-class achievement does happen. Barcelona went and scooped up Leo Messi from the hinterlands of Argentina at age 12 and he went and became Leo Messi. Lebron James was on the cover of Sports Illustrated when he was in high school.

In some fields, predicting world-best performance happens at an early age. But IQ – whatever its other merits – does not seem to serve as an effective mechanism for predicting world-best performance in specific individualized activities.

Magnus Carlsen’s IQ

When I type in Magnus Carlsen’s name into Google, the first thing that autofills (after chess) is “Magnus Carlsen IQ.”

People seem to want to believe that his IQ score can explain why he is the Mozart of chess.  

We don’t know what his IQ is, but the instinct people have to try to explain his performance in terms of IQ feels very similar to people’s desire to ascribe an IQ of 225 to Einstein. It’s phlogiston.

Magnus Carlsen probably has a very high IQ. He obviously has well above-average intelligence. Maybe his IQ is 130, 150, or 170 (there's a website called ScoopWhoop that claims, without citation, that it's 190). But however high his IQ, doubtless there are many or at least a few chess players in the world who have higher IQs than he has. But he’s the #1 chess player in the world – not his competitors with higher IQs. And I don’t think the explanation for why he’s so great is his “mindset” or “grit” or anything like that.

It’s because IQ is akin to an intellectual decathlon, whereas chess is a single-event competition. If we dug deep into the sub-components of Carlsen’s IQ (or perhaps the sub-components of the sub-components), we’d probably find some sub-component where he measured off the charts. I’m not saying there’s a “chess gene,” but I suspect that there is a trait that could be measured as a sub-component of intelligence that that is more specific than IQ that would be a greater explanatory variable of his abilities than raw IQ.

Leo Messi isn’t the greatest soccer player in the world because he’s the best overall athlete in the world. He’s the best soccer player in the world because of his agility and quickness in incredibly tight spaces. Because of his amazing coordination in his lower extremities. Because of his ability to change direction with the ball before defenders have time to react. These are all natural talents. But they are only particularly valuable because of the arbitrary constraints in soccer.

Leo Messi is a great natural athlete. If we had a measure of AQ, he’d probably be in the 98th or 99th percentile. But that doesn’t begin to explain his otherworldly soccer-playing talents. He probably could have been a passable high-school point guard at a school of 1000 students.  He would have been a well-above-average decathlete (though I doubt he could throw the shot put worth a damn).

But it’s the unique athletic gifts that are particularly well suited to soccer that enabled him to be the best in the world at soccer. So, too, with Magnus Carlsen with chess, Elon Musk with entrepreneurialism, and Albert Einstein with paradigm-shifting physics.

The decathlon won’t predict the next Leo Messi or the next Lebron James. And IQ won’t predict the next Magnus Carlsen, Elon Musk, Picasso, Mozart, or Albert Einstein.

And so we shouldn’t seek it out as an after-the-fact explanation for their success, either.


[1] Of course, high performance in some fields is probably more closely correlated with IQ than others: physics professor > english professor > tech founder > lawyer > actor > bassist in grunge band. [Note: this footnote is total unfounded speculation]

[3] The other part is that people don’t like to be defined by traits that they feel they cannot change or improve.

[4] Let me know if I am missing any famous examples here.

[Link] Putanumonit: A spreadsheet helps debias decisions, like picking a girl to date

10 Jacobian 15 March 2017 03:19AM

WMDs in Iraq and Syria

9 ChristianKl 10 May 2017 09:03PM

Tetlock wrote in Superforcasters that the US intelligence establishment was likely justified to believe that it was likely that Iraq was hiding WMDs. According to Tetlock their sin was that they asserted that it's certain that Iraq had WMD.

When first reading Superforcasters I didn't quite understand the situation. After reading https://theintercept.com/2015/04/10/twelve-years-later-u-s-media-still-cant-get-iraqi-wmd-story-right/ I did.

The core problem was that Saddam lost track of some of his chemical weapons. His military didn't do perfect accounting of them and they looked the same as conventional weapons. It takes an x-ray to tell his chemical weapons apart from the normal ones.

The US intercepted communications where Saddam told his units to ensure that they had no chemical weapons that inspectors could find. Of course, that communication didn't happen in English. That communication seems to have been misinterpreted by the US intelligence community as evidence that Saddam is hiding WMDs.

Nearly nobody understood that Iraq having chemical weapons and hiding them are two different systems because you need to know where your chemical weapons happen to be to hide them. On the same token, nobody publically argues that pure incompetence might be the cause of chemical weapon usage in Syria. We want to see human agency and if a chemical weapon exploded we want to know that someone is guilty of having made the decision to use them.
In a recent facebook discussion about Iraq and the value of IARPA, a person asserted that the US intelligence community only thought Iraq had WMDs because they were subject to political pressure. 
We have to get better at understanding that bad events can happen without people intending them to happen.

After understanding Iraq it's interesting to look at Syria. Maybe the chemical weapons that exploded in Syria didn't explode because Assad's troops or the opposition wanted to use chemical weapons. They might have simply exploded because some idiot did bad accounting and mislabeled a chemical weapon as being a conventional weapon.

The idea that WMDs explode by accident might be too horrible to contemplate. We have to be better at seeing incompetence as a possible explanation when we want to pin the guilt for having made an evil decision on another person.

[Link] Don't Shoot the Messenger

9 Vaniver 19 April 2017 10:14PM

What conservatives and environmentalists agree on

9 PhilGoetz 08 April 2017 12:57AM

Today we had a sudden cold snap here in western Pennsylvania, with the temperature dropping 30 degrees F.  I was walking through a white field that had been green yesterday, looking at daffodils poking up through the snow and feeling irritated that they'd probably die.  It occurred to me that, if we could control the weather, people would probably vote for a smooth transition from winter to summer, and this would wreak some unforeseen environmental catastrophe, because it would suddenly make most survival strategies reliably sub-optimal.

This is typical environmentalist thinking:  Whenever you see something in the environment that you don't like, stop and step back before trying to change it.  Trust nature that there's some reason it is that way.  Interfere as little as possible.

The classic example is forest fires.  Our national park service used to try to stop all forest fires.  This policy changed in the 1960s for several reasons, including the observation that no new Sequoia saplings had sprouted since the beginning of fire suppression in the 19th century.  Fire is dangerous, destructive, and necessary.

It struck me that this cornerstone of environmentalism is also the cornerstone of social conservatism.

continue reading »

[Link] ‘Crucial Considerations and Wise Philanthropy’, by Nick Bostrom

9 casebash 31 March 2017 12:52PM

Announcing: The great bridge

9 Elo 17 March 2017 01:11AM

Original post: http://bearlamp.com.au/announcing-the-great-bridge-between-communities/


In the deep dark lurks of the internet, several proactive lesswrong and diaspora leaders have been meeting each day.  If we could have cloaks and silly hats; we would.

We have been discussing the great diversification, and noticed some major hubs starting to pop up.  The ones that have been working together include:

  • Lesswrong slack
  • SlateStarCodex Discord
  • Reddit/Rational Discord
  • Lesswrong Discord
  • Exegesis (unofficial rationalist tumblr)

The ones that we hope to bring together in the future include (on the willingness of those servers):

  • Lesswrong IRC (led by Gwern)
  • Slate Star Codex IRC
  • AGI slack
  • Transhumanism Discord
  • Artificial Intelligence Discord

How will this work?

About a year ago, the lesswrong slack tried to bridge across to the lesswrong IRC.  That was bad.  From that experience we learnt a lot that can go wrong, and have worked out how to avoid those mistakes.  So here is the general setup.

Each server currently has it's own set of channels, each with their own style of talking and addressing problems, and sharing details and engaging with each other.  We definitely don't want to do anything that will harm those existing cultures.  In light of this, taking the main channel from one server and mashing it into the main channel of another server is going to reincarnate into HELL ON EARTH.  and generally leave both sides with the sentiment that "<the other side> is wrecking up <our> beautiful paradise".  Some servers may have a low volume buzz at all times, other servers may become active for bursts, it's not good to try to marry those things.

Logistics:

Room: Lesswrong-Slack-Open

Bridged to:

  • exegesis#lwslack_bridge
  • Discord-Lesswrong#lw_slack_main
  • R/rational#lw_slack_open
  • SSC#bridge_slack

I am in <exegesis, D/LW, R/R, SSC> what does this mean?

If you want to peek into the lesswrong slack and see what happens in their #open channel.  You can join or unmute your respective channel and listen in, or contribute (two way relay) to their chat.  Obviously if everyone does this at once we end up spamming the other chat and probably after a week we cut the bridge off because it didn't work.  So while it's favourable to increase the community; be mindful of what goes on across the divide and try not to anger our friends.

I am in Lesswrong-Slack, what does this mean?

We have new friends!  Posts in #open will be relayed to all 4 children rooms where others can contribute if they choose.  Mostly they have their own servers to chat on, and if they are not on an info-diet already, then maybe they should be.  We don't anticipate invasion or noise.

Why do they get to see our server and we don't get to see them?

So glad you asked - we do.  There is an identical set up for their server into our bridge channels.  in fact the whole diagram looks something like this:

Server Main channel Slack-Lesswrong Discord-Exegesis Discord-Lesswrong Discord-r/rational Discord-SSC
Slack-Lesswrong Open   lwslack_bridge lw_slack_main lw_slack_open bridge_slack
Discord-Exegesis Main #bridge_rat_tumblr   exegesis_main exegesis_rattumb_main bridge_exegesis
Discord-Lesswrong Main #Bridge_discord_lw lwdiscord_bridge   lw_discord_main bridge_lw_disc
Discord-r/rational General #bridge_r-rational_dis rrdiscord_bridge reddirati_main   bridge_r_rational
Discord-SSC General #bridge_ssc_discord sscdiscord_bridge ssc_main ssc_discord_gen  

Pretty right? No it's not.  But that's in the backend.

For extra clarification, the rows are the channels that are linked.  Which is to say that Discord-SSC, is linked to a child channel in each of the other servers.  The last thing we want to do is impact this existing channels in a negative way.

But what if we don't want to share our open and we just want to see the other side's open?  (/our talk is private, what about confidential and security?)

Oh you mean like the prisoners dilemma?  Where you can defect (not share) and still be rewarded (get to see other servers).  Yea it's a problem.  Tends to be when one group defects, that others also defect.  There is a chance that the bridge doesn't work.  That this all slides, and we do spam each other, and we end up giving up on the whole project.  If it weren't worth taking the risk we wouldn't have tried.

We have not rushed into this bridge thing, we have been talking about it calmly and slowly and patiently for what seems like forever.  We are all excited to be taking a leap, and keen to see it take off.

Yes, security is a valid concern, walled gardens being bridged into is a valid concern, we are trying our best.  We are just as hesitant as you, and being very careful about the process.  We want to get it right.

So if I am in <server1> and I want to talk to <server3> I can just post in the <bridge-to-server2> room and have the message relayed around to server 3 right?

Whilst that is correct, please don't do that.  You wouldn't like people relaying through your main to talk to other people.  Also it's pretty silly, you can just post in your <servers1> main and let other people see it if they want to.

This seems complicated, why not just have one room where everyone can go and hang out?

  1. How do you think we ended up with so many separate rooms
  2. Why don't we all just leave <your-favourite server> and go to <that other server>?  It's not going to happen

Why don't all you kids get off my lawn and stay in your own damn servers?

Thank's grandpa.  No one is coming to invade, we all have our own servers and stuff to do, we don't NEED to be on your lawn, but sometimes it's nice to know we have friends.

<server2> shitposted our server, what do we do now?

This is why we have mods, why we have mute and why we have ban.  It might happen but here's a deal; don't shit on other people and they won't shit on you.  Also if asked nicely to leave people alone, please leave people alone.  Remember anyone can tap out of any discussion at any time.

I need a picture to understand all this.

Great!  Friends on exegesis made one for us.


Who are our new friends:

Lesswrong Slack

Lesswrong slack has been active since 2015, and has a core community. The slack has 50 channels for various conversations on specific topics, the #open channel is for general topics and has all kinds of interesting discoveries shared here.

Discord-Exegesis (private, entry via tumblr)

Exegesis is a discord set up by a tumblr rationalist for all his friends (not just rats). It took off so well and became such a hive in such a short time that it's now a regular hub.

Discord-Lesswrong

Following Exegesis's growth, a discord was set up for lesswrong, it's not as active yet, but has the advantage of a low barrier to entry and it's filled with lesswrongers.

Discord-SSC

Scott posted a link on an open thread to the SSC discord and now it holds activity from users that hail from the SSC comment section. it probably has more conversation about politics than other servers but also has every topic relevant to his subscribers.

Discord-r/rational

reddit rational discord grew from the rationality and rational fiction subreddit, it's quite busy and covers all topics.


As at the publishing of this post; the bridge is not live, but will go live when we flip the switch.


Meta: this took 1 hour to write (actualy time writing) and half way through I had to stop and have a voice conference about it to the channels we were bridging.

Cross posted to lesswrong: http://lesswrong.com/lw/oqz

 

Am I Really an X?

9 Gram_Stone 05 March 2017 12:11AM

As I understand it, there is a phenomenon among transgender people where no matter what they do they can't help but ask themselves the question, "Am I really an [insert self-reported gender category here]?" In the past, a few people have called for a LessWrong-style dissolution of this question. This is how I approach the problem.

There are two caveats which I must address in the beginning.

The first caveat has to do with hypotheses about the etiology of the transgender condition. There are many possible causes of gender identity self-reports, but I don't think it's too controversial to propose that at least some of the transgender self-reports might result from the same mechanism as cisgender self-reports. Again, the idea is that there is some 'self-reporting algorithm', that takes some input that we don't yet know about, and outputs a gender category, and that both cisgender people and transgender people have this. It's not hard to come up with just-so stories about why having such an algorithm and caring about its output might have been adaptive. This is, however, an assumption. In theory, the self-reports from transgender people could have a cause separate from the self-reports of cisgender people, but it's not what I expect.

The second caveat has to do with essentialism. In the past calls for an article like this one, I saw people point out that we reason about gender as if it is an essence, and that any dissolution would have to avoid this mistake. But there's a difference between describing an algorithm that produces a category which feels like an essence, and providing an essentialist explanation. My dissolution will talk about essences because the human mind reasons with them, but my dissolution itself will not be essentialist in nature.

Humans universally make inferences about their typicality with respect to their self-reported gender. Check Google Scholar for 'self-perceived gender typicality' for further reading. So when I refer to a transman, by my model, I mean, "A human whose self-reporting algorithm returns the gender category 'male', but whose self-perceived gender typicality checker returns 'Highly atypical!'"

And the word 'human' at the beginning of that sentence is important. I do not mean "A human that is secretly, essentially a girl," or "A human that is secretly, essentially a boy,"; I just mean a human. I postulate that there are not boy typicality checkers and girl typicality checkers; there are typicality checkers that take an arbitrary gender category as input and return a measure of that human's self-perceived typicality with regard to the category.

So when a transwoman looks in the mirror and feels atypical because of a typicality inference from the width of her hips, I believe that this is not a fundamentally transgender experience, not different in kind, but only in degree, from a ciswoman who listens to herself speak and feels atypical because of a typicality inference from the pitch of her voice.

Fortunately, society's treatment of transgender people has come around to something like this in recent decades; our therapy proceeds by helping transgender people become more typical instances of their self-report algorithm's output.

Many of the typical traits are quite tangible: behavior, personality, appearance. It is easier to make tangible things more typical, because they're right there for you to hold; you aren't confused about them. But I often hear reports of transgender people left with a nagging doubt, a lingering question of "Am I really an X?, which feels far more slippery and about which they confess themselves quite confused.

To get at this question, I sometimes see transgender people try to simulate the subjective experience of a typical instance of the self-report algorithm's output. They ask questions like, "Does it feel the same to be me as it does to be a 'real X'?" And I think this is the heart of the confusion.

For when they simulate the subjective experience of a 'real X', there is a striking dissimilarity between themselves and the simulation, because a 'real X' lacks a pervasive sense of distress originating from self-perceived atypicality.

But what I just described in the previous sentence is itself a typicality inference, which means that this simulation itself causes distress from atypicality, which is used to justify future inferences of self-perceived atypicality!

I expected this to take more than one go-around.

Let's review something Eliezer wrote in Fake Causality:

One of the primary inspirations for Bayesian networks was noticing the problem of double-counting evidence if inference resonates between an effect and a cause.  For example, let's say that I get a bit of unreliable information that the sidewalk is wet.  This should make me think it's more likely to be raining.  But, if it's more likely to be raining, doesn't that make it more likely that the sidewalk is wet?  And wouldn't that make it more likely that the sidewalk is slippery?  But if the sidewalk is slippery, it's probably wet; and then I should again raise my probability that it's raining...

If you didn't have an explicit awareness that you have a general human algorithm that checks the arbitrary self-report against the perceived typicality, but rather you believed that this was some kind of special, transgender-specific self-doubt, then your typicality checker would never be able to mark its own distress signal as 'Typical!', and it would oscillate between judging the subjective experience as atypical, outputting a distress signal in response, judging its own distress signal as atypical, sending a distress signal about that, etc.

And this double-counting is not anything like hair length or voice pitch, or even more slippery stuff like 'being empathetic'; it's very slippery, and no matter how many other ways you would have made yourself more typical, even though those changes would have soothed you, there would have been this separate and additional lingering doubt, a doubt that can only be annihilated by understanding the deep reasons that the tangible interventions worked, and how your mind runs skew to reality.

And that's it. For me at least, this adds up to normality. There is no unbridgeable gap between the point at which you are a non-X and the point at which you become an X. Now you can just go back to making yourself as typical as you want to be, or anything else that you want to be.

5 Project Hufflepuff Suggestions for the Rationality Community

9 lifelonglearner 04 March 2017 02:23AM

<cross-posed on Facebook>


In the spirit of Project Hufflepuff, I’m listing out some ideas for things I would like to see in the rationality community, which seem like perhaps useful things to have. I dunno if all of these are actually good ideas, but it seems better to throw some things out there and iterate.

 

Ideas:


Idea 1) A more coherent summary of all the different ideas that are happening across all the rationalist blogs. I know LessWrong is trying to become more of a Schelling point, but I think a central forum is still suboptimal for what I want. I’d like something that just takes the best ideas everyone’s been brewing and centralizes them in one place so I can quickly browse them all and dive deep if something looks interesting.


Suggestions:

A) A bi-weekly (or some other period) newsletter where rationalists can summarize their best insights of the past weeks in 100 words or less, with links to their content.

B) An actual section of LessWrong that does the above, so people can comment / respond to the ideas.


Thoughts:

This seems straightforward and doable, conditional on commitment from 5-10 people in the community. If other people are also excited, I’m happy to reach out and get this thing started.



Idea 2) A general tool/app for being able to coordinate. I’d be happy to lend some fraction of my time/effort in order to help solve coordination problems. It’s likely other people feel the same way. I’d like a way to both pledge my commitment and stay updated on things that I might be able to plausibly Cooperate on.


Suggestions:

A) An app that is managed by someone, which sends out broadcasts for action every so often. I’m aware that similar things / platforms already exist, so maybe we could just leverage an existing one for this purpose.


Thoughts:

In abstract, this seems good. Wondering what others think / what sorts of coordination problems this would be good for. The main value here is being confident in *actually* getting coordination from the X people who’ve signed up.


Idea 3) More rationality materials that aren’t blogs. The rationality community seems fairly saturated with blogs. Maybe we could do with more webcomics, videos, or something else?


Suggestions:

A) Brainstorm good content from other mediums, benefits / drawbacks, and see why we might want content in other forms.

B) Convince people who already make such mediums to touch on rationalist ideas, sort of like what SMBC does.


Thoughts:

I’d be willing to start up either a webcomic or a video series, conditional on funding. Anyone interested in sponsoring? Happy to have a discussion below.

 

EDIT:

Links to things I've done for additional evidence:

 


Idea 4) More systematic tools to master rationality techniques. To my knowledge, only a small handful of people have really tried to implement Systemization to learning rationality, of whom Malcolm and Brienne are the most visible. I’d like to see some more attempts at Actually Trying to learn techniques.


Suggestions:

A) General meeting place to discuss the learning / practice.

B) Accountability partners + Skype check-ins .

C) List of examples of actually using the techniques + quantified self to get stats.


Thoughts:

I think finding more optimal ways to do this is very important. There is a big step between knowing how techniques work and actually finding ways to do them. I'd be excited to talk more about this Idea.


Idea 5) More online tools that facilitate rationality-things. A lot of rationality techniques seem like they could be operationalized to plausibly provide value.


Suggestions:

A) An online site for Double Cruxing, where people can search for someone to DC with, look at other ongoing DC’s, or propose topics to DC on.

B) Chatbots that integrate things like Murphyjitsu or ask debugging questions.


Thoughts:

I’m working on building a Murphyjitsu chatbot for building up my coding skill. The Double Crux site sounds really cool, and I’d be happy to do some visual mockups if that would help people’s internal picture of how that might work out. I am unsure of my ability to do the actual coding, though.

 

 

Conclusion:

Those are the ideas I currently have. Very excited to hear what other people think of them, and how we might be able to get the awesome ones into place. Also, feel free to comment on the FB post, too, if you want to signal boost.

Notes from the Hufflepuff Unconference (Part 1)

8 Raemon 23 May 2017 09:04PM

April 28th, we ran the Hufflepuff Unconference in Berkeley, at the MIRI/CFAR office common space.

There's room for improvement in how the Unconference could have been run, but it succeeded the core things I wanted to accomplish: 

 - Established common knowledge of what problems people were actually interested in working on
 - We had several extensive discussions of some of those problems, with an eye towards building solutions
 - Several people agreed to work together towards concrete plans and experiments to make the community more friendly, as well as build skills relevant to community growth. (With deadlines and one person acting as project manager to make sure real progress was made)
 - We agreed to have a followup unconference in roughly three months, to discuss how those plans and experiments were going

Rough notes are available here. (Thanks to Miranda, Maia and Holden for takin really thorough notes)

This post will summarize some of the key takeaways, some speeches that were given, and my retrospective thoughts on how to approach things going forward.

But first, I'd like to cover a question that a lot of people have been asking about:

What does this all mean for people outside of the Bay?

The answer depends.

I'd personally like it if the overall rationality community got better at social skills, empathy, and working together, sticking with things that need sticking with (and in general, better at recognizing skills other than metacognition). In practice, individual communities can only change in the ways the people involved actually want to change, and there are other skills worth gaining that may be more important depending on your circumstances.

Does Project Hufflepuff make sense for your community?

If you're worried that your community doesn't have an interest in any of these things, my actual honest answer is that doing something "Project Hufflepuff-esque" probably does not make sense. I did not choose to do this because I thought it was the single-most-important thing in the abstract. I did it because it seemed important and I knew of a critical mass of people who I expected to want to work on it. 

If you're living in a sparsely populated area or haven't put a community together, the first steps do not look like this, they look more like putting yourself out there, posting a meetup on Less Wrong and just *trying things*, any things, to get something moving.

If you have enough of a community to step back and take stock of what kind of community you want and how to strategically get there, I think this sort of project can be worth learning from. Maybe you'll decide to tackle something Project-Hufflepuff-like, maybe you'll find something else to focus on. I think the most important thing is have some kind of vision for something you community can do that is worth working together, leveling up to accomplish.

Community Unconferences as One Possible Tool

Community unconferences are a useful tool to get everyone on the same page and spur them on to start working on projects, and you might consider doing something similar. 

They may not be the right tool for you and your group - I think they're most useful in places where there's enough people in your community that they don't all know each other, but do have enough existing trust to get together and brainstorm ideas. 

If you have a sense that Project Hufflepuff is worthwhile for your community but the above disclaimers point towards my current approach not making sense for you, I'm interested in talking about it with you, but the conversation will look less like "Ray has ideas for you to try" and more like "Ray is interested in helping you figure out what ideas to try, and the solution will probably look very different."

Online Spaces

Since I'm actually very uncertain about a lot of this and see it as an experiment, I don't think it makes sense to push for any of the ideas here to directly change Less Wrong itself (at least, yet). But I do think a lot of these concepts translate to online spaces in some fashion, and I think it'd make sense to try out some concepts inspired by this in various smaller online subcommunities.

Table of Contents:

I. Introduction Speech

 - Why are we here?
 - The Mission: Something To Protect
 - The Invisible Badger, or "What The Hell Is a Hufflepuff?"
 - Meta Meetups Usually Suck. Let's Try Not To.

II. Common Knowledge

 - What Do People Actually Want?
 - Lightning Talks

III. Discussing the Problem (Four breakout sessions)

 - Welcoming Newcomers
 - How to handle people who impose costs on others?
 - Styles of Leadership and Running Events
 - Making Helping Fun (or at least lower barrier-to-entry)

IV. Planning Solutions and Next Actions

V. Final Words

I. Introduction: It Takes A Village to Save a World

(A more polished version of my opening speech from the unconference)

[Epistemic Status: This is largely based on intuition, looking at what our community has done and what other communities seem to be able to do. I'm maybe 85% confident in it, but it is my best guess]

In 2012, I got super into the rationality community in New York. I was surrounded by people passionate about thinking better and using that thinking to tackle ambitious projects. And in 2012 we all decided to take on really hard projects that were pretty likely to fail, because the expected value seemed high, and it seemed like even if we failed we'd learn a lot in the process and grow stronger.

That happened - we learned and grew. We became adults together, founding companies and nonprofits and creating holidays from scratch.

But two years later, our projects were either actively failing, or burning us out. Many of us became depressed and demoralized.

There was nobody who was okay enough to actually provide anyone emotional support. Our core community withered.

I ended up making that the dominant theme of the 2014 NYC Solstice, with a call-to-action to get back to basics and take care each other.

I also went to the Berkeley Solstice that year. And... I dunno. In the back of my mind I was assuming "Berkeley won't have that problem - the Bay area has so many people, I can't even imagine how awesome and thriving a community they must have." (Especially since the Bay kept stealing all the Movers and Shakers of NYC).

The theme of the Bay Solstice turned out to be "Hey guys, so people keep coming to the Bay, running on a dream and a promise of community, but that community is not actually there, there's a tiny number of well-connected people who everyone is trying to get time with, and everyone seems lonely and sad. And we don't even know what to do about this."

In 2015, that theme in the Berkeley Solstice was revisited.

So I think that was the initial seed of what would become Project Hufflepuff - noticing that it's not enough to take on cool projects, that it's not enough to just get a bunch of people together and call it a community. Community is something you actively tend to. Insofar as Maslow's hierarchy is real, it's a foundation you need before ambitious projects can be sustainable.

There are other pieces of the puzzle - different lenses that, I believe, point towards a Central Thing. Some examples:

Group houses, individualism and coordination.

I've seen several group houses where, when people decide it no longer makes sense to live in the house, they... just kinda leave. Even if they've literally signed a lease. And everyone involved (the person leaving and those remain), instinctively act as if it's the remaining people's job to fill the leaver's spot, to make rent.

And the first time, this is kind of okay. But then each subsequent person leaving adds to a stressful undertone of "OMG are we even going to be able to afford to live here?". It eventually becomes depressing, and snowballs into a pit that makes newcomers feel like they don't WANT to move into the house.

Nowadays I've seen some people explicitly building into the roommate agreement a clear expectation of how long you stay and who's responsibility it is to find new roommates and pay rent in the meantime. But it's disappointing to me that this is something we needed, that we weren't instinctively paying to attention to how we were imposing costs on each other in the first place. That when we *violated a written contract*, let alone a handshake agreement, that we did not take upon ourselves (or hold each other accountable), to ensure we could fill our end of the bargain.

Friends, and Networking your way to the center

This community puts pressure on people to improve. It's easier to improve when you're surrounded by ambitious people who help or inspire each other level up. There's a sense that there's some cluster of cool-people-who-are-ambitious-and-smart who've been here for a while, and... it seems like everyone is trying to be friends with those people. 

It also seems like people just don't quite get that friendship is a skill, that adult friendships in City Culture can be hard, and it can require special effort to make them happen.

I'm not entirely sure what's going on here - it doesn't make sense to say anyone's obligated to hang out with any particular person (or obligated NOT to), but if 300 people aren't getting the connection they want it seems like *somewhere people are making a systematic mistake.* 

(Since the Unconference, Maia has tackled this particular issue in more detail)

 

The Mission - Something To Protect

 

As I see it, the Rationality Community has three things going on: Truth. Impact. And "Being People".

In some sense, our core focus is the practice of truthseeking. The thing that makes that truthseeking feel *important* is that it's connected to broader goals of impacting the world. And the thing that makes this actually fun and rewarding enough to stick with is a community that meets our needs, where can both flourish as individuals and find the relationships we want.

I think we have made major strides in each of those areas over the past seven years. But we are nowhere near done.

Different people have different intuitions of which of the three are most important. Some see some of them as instrumental, or terminal. There are people for whom Truthseeking is *the point*, and they'd have been doing that even if there wasn't a community to help them with it, and there are people for whom it's just one tool of many that helps them live their life better or plan important projects.

I've observed a tendency to argue about which of these things is most important, or what tradeoffs are worth making. Inclusiveness verses high standards. Truth vs action. Personal happiness vs high acheivement.

I think that kind of argument is a mistake.

We are falling woefully short on all of these things. 

We need something like 10x our current capacity for seeing, and thinking. 10x our capacity for doing. 10x our capacity for *being healthy people together.*

I say "10x" not because all these things are intrinsically equal. The point is not to make a politically neutral push to make all the things sound nice. I have no idea exactly how far short we're falling on each of these because the targets are so far away I can't even see the end, and we are doing a complicated thing that doesn't have clear instructions and might not even be possible.

The point is that all of these are incredibly important, and if we cannot find a way to improve *all* of these, in a way that is *synergistic* with each other, then we will fail.

There is a thing at the center of our community. Not all of us share the exact same perspective on it. For some of us it's not the most important thing. But it's been at the heart of the community since the beginning and I feel comfortable asserting that it is the thing that shapes our culture the most:

The purpose of our community is to make sure this place is okay:

The world isn't okay right now, on a number of levels. And a lot of us believe there is a strong chance it could become dramatically less okay. I've seen people make credible progress on taking responsibility for pieces of our home. But when all is said and done, none of our current projects really give me the confidence that things are going to turn out all right. 

Our community was brought together on a promise, a dream, and we have not yet actually proven ourselves worthy of that dream. And to make that dream a reality we need a lot of things.

We need to be able to criticize, because without criticism, we cannot improve.

If we cannot, I believe we will fail.

We need to be able to talk about ideas that are controversial, or uncomfortable - otherwise our creativity and insight will be crippled.

If we cannot, I believe we will fail.

We need to be able to do those things without alienating people. We need to be able to criticize without making people feel untrusted and discouraged from even taking action. We need to be able to discuss challenging things while earnestly respecting the notion that *talking about ideas gives those ideas power and has concrete effects on social reality*, and sometimes that can hurt people.

If we cannot figure out how to do that, I believe we will fail.

We need more people who are able and willing to try things that have never been done before. To stick with those things long enough to *get good at them*, to see if they can actually work. We need to help each other do impossible things. And we need to remember to check for and do the *possible*, boring, everyday things that are in fact straightforward and simple and not very inspiring. 

If we cannot manage to do that, I believe we will fail.

We need to be able to talk concretely about what the *highest leverage actions in the world are*. We need to prioritize those things, because the world is huge and broken and we are small. I believe we need to help each other through a long journey, building bigger and bigger levers, building connections with people outside our community who are undertaking the same journey through different perspectives.

And in the process, we need to not make it feel like if *you cannot personally work on those highest leverage things, that you are not important.* 

There's the kind of importance where we recognize that some people have scarce skills and drive, and the kind of importance where we remember that *every* person has intrinsic worth, and you owe *nobody* any special skills or prestigious sounding projects for your life to be worthwhile.

This isn't just a philosophical matter - I think it's damaging to our mental health and our collective capacity. 

We need to recognize that the distribution of skills we tend to reward or punish is NOT just about which ones are actually most valuable - sometimes it is simply founder effects and blind spots.

We cannot be a community for everyone - I believe trying to include anyone with a passing interest in us is a fool's errand. But there are many people who had valuable skills to contribute who have turned away, feeling frustrated and un-valued.

If we cannot find a way to accomplish all of these things at once, I believe we will fail.

The thesis of Project Hufflepuff is that it takes (at least) a village to save a world. 

It takes people doing experimental impossible things. It takes caretakers. It takes people helping out with unglorious tasks. It takes technical and emotional and physical skills. And while it does take some people who specialize in each of those things, I think it also needs many people who are least a little bit good at each of them, to pitch in when needed.

Project Hufflepuff is not the only things our community needs, or the most important. But I believe it is one of the necessary things that our community needs, if we're to get to 10x our current Truthseeking, Impact and Human-ing.

If we're to make sure that our home is okay.

The Invisible Badger

"A lone hufflepuff surrounded by slytherins will surely wither as if being leeched dry by vampires."

- Duncan

[Epistemic Status: My evidence for this is largely based on discussions with a few people for whom the badger seems real and valuable, and who report things being different in other communities, as well as some of my general intuitions about society. I'm 75% sure the badger exists, 90% that's it worth leaning into the idea of the badger to see if it works for you, and maybe 55% sure that it's worth trying to see the badger if you can't already make out it's edges.]


 

If I *had* to pick a clear thing that this conference is about without using Harry Potter jargon, I'd say "Interpersonal dynamics surrounding trust, and how those dynamics apply to each of the Impact/Truth/Human focuses of the rationality community."

I'm not super thrilled with that term because I think I'm grasping more for some kind of gestalt. An overall way of seeing and being that's hard to describe and that doesn't come naturally to the sort of person attracted to this community.

Much like the blind folk and the elephant, who each touched a different part of the animal and came away with a different impression (the trunk seems like a snake, the legs seem like a tree), I've been watching several people in the community try to describe things over the past few years. And maybe those things are separate but I feel like they're secretly a part of the same invisible badger.

Hufflepuff is about hard work, and loyalty, and camaraderie. It's about emotional intelligence. It's about seeing value in day to day things that don't directly tie into epic narratives. 

There's a bunch of skills that go into Hufflepuff. And part of want I want is for people to get better at those skills. But It think a mindset, an approach, that is fairly different from the typical rationalist mindset, that makes those skills easier. It's something that's harder when you're being rigorously utilitarian and building models of the world out of game theory and incentives.

Mindspace is deep and wide, and I don't expect that mindset to work for everyone. I don't think everyone should be a Hufflepuff. But I do think it'd be valuable to the community if more people at least had access to this mindset and more of these skills.

So what I'd like, for tonight, is for people to lean into this idea. Maybe in the end you'll find that this doesn't work for you. But I think many people's first instinct is going to be that this is alien and uncomfortable and I think it's worth trying to push past that.

The reason we're doing this conference together is because the Hufflepuff way doesn't really work if people are trying to do it alone - I think it requires trust and camaraderie and persistence to really work. I don't think we can have that required trust all at once, but I think if there are multiple people trying to make it work, who can incrementally trust each other more, I think we can reach a place where things run more smoothly, where we have stronger emotional connections, and where we trust each other enough to take on more ambitious projects than we could if we're all optimizing as individuals.

Meta-Meetups Suck. Let's Not.

This unconference is pretty meta - we're talking about norms and vague community stuff we want to change.

Let me tell you, meta meetups are the worst. Typically you end up going around in circles complaining and wishing there were more things happening and that people were stepping up and maybe if you're lucky you get a wave of enthusiasm that lasts a month or so and a couple things happen but nothing really *changes*.

So. Let's not do that. Here's what I want to accomplish and which seems achievable:

1) Establish common knowledge of important ideas and behavior patterns. 

Sometimes you DON'T need to develop a whole new skill, you just need to notice that your actions are impacting people in a different way, and maybe that's enough for you to decide to change somethings. Or maybe someone has a concept that makes it a lot easier for you to start gaining a new skill on your own.

2) Establish common knowledge of who's interested in trying which new norms, or which new skills. 

We don't actually *know* what the majority of people want here. I can sit here and tell you what *I* think you should want, but ultimately what matters is what things a critical mass of people want to talk about tonight.

Not everyone has to agree that an idea is good to try it out. But there's a lot of skills or norms that only really make sense when a critical mass of other people are trying them. So, maybe of the 40 people here, 25 people are interested in improving their empathy, and maybe another 20 are interested in actively working on friendship skills, or sticking to commitments. Maybe those people can help reinforce each other.

3) Explore ideas for social and skillbuilding experiments we can try, that might help. 

The failure mode of Ravenclaws is to think about things a lot and then not actually get around to doing them. A failure mode of ambitious Ravenclaws, is to think about things a lot and then do them and then assume that because they're smart, that they've thought of everything, and then not listen to feedback when they get things subtly or majorly wrong.

I'd like us to end by thinking of experiments with new norms, or habits we'd like to cultivate. I want us to frame these as experiments, that we try on a smaller scale and maybe promote more if they seem to be working, while keeping in mind that they may not work for everyone.

4) Commit to actions to take.

Since the default action is for them to peter out and fail, I'd like us to spend time bulletproofing them, brainstorming and coming up with trigger-action plans so that they actually have a chance to succeed.

Tabooing "Hufflepuff"

Having said all that talk about The Hufflepuff Way...

...the fact is, much of the reason I've used those towards is to paint a rough picture to attract the sort of person I wanted to attract to this unconference.

It's important that there's a fuzzy, hard-to-define-but-probably-real concept that we're grasping towards, but it's also important not to be talking past each other. Early on in this project I realized that a few people who I thought were on the same page actually meant fairly different things. Some cared more about empathy and friendship. Some cared more about doing things together, and expected deep friendships to arise naturally from that.

So I'd like us to establish a trigger-action-plan right now - for the rest of this unconference, if someone says "Hufflepuff", y'all should say "What do you mean by that?" and then figure out whatever concrete thing you're actually trying to talk about.

II. Common Knowledge

The first part of the unconference was about sharing our current goals, concerns and background knowledge that seemed useful. Most of the specifics are covered in the notes. But I'll talk here about why I included the things I did and what my takeaways were afterwards on how it worked.

Time to Think

The first thing I did was have people sit and think about what they actually wanted to get out of the conference, and what obstacles they could imagine getting in the way of that. I did this because often, I think our culture (ostensibly about helping us think better) doesn't give us time to think, and instead has people were are quick-witted and conversationally dominant end up doing most of the talking. (I wrote a post a year ago about this, the 12 Second Rule). In this case I gave everyone 5 minutes, which is something I've found helpful at small meetups in NYC.

This had mixed results - some people reported that while they can think well by themselves, in a group setting they find it intimidating and their mind starts wandering instead of getting anything done. They found it much more helpful when I eventually let people-who-preferred-to-talk-to-each-other go into another room to talk through their ideas outloud.

I think there's some benefit to both halves of this and I'm not sure how common which set of preferences are. It's certainly true that it's not common for conferences to give people a full 5 minutes to think so I'd expect it to be someone uncomfortable-feeling regardless of whether it was useful.

But an overall outcome of the unconference was that it was somewhat lower energy than I'd wanted, and opening with 5 minutes of silent thinking seemed to contribute to that, so for the next unconference I run, I'm leaning towards a shorter period of time for private thinking (Somewhere between 12 and 60 seconds), followed by "turn to your neighbors and talk through the ideas you have", followed by "each group shares their concepts with the room."

"What is do you want to improve on? What is something you could use help with?"

I wanted people to feel like active participants rather than passive observers, and I didn't want people to just think "it'd be great if other people did X", but to keep an internal locus of control - what can *I* do to steer this community better? I also didn't want people to be thinking entirely individualistically.

I didn't collect feedback on this specific part and am not sure how valuable others found it (if you were at the conference, I'd be interested if you left any thoughts in the comments). Some anonymized things people described:

  • When I make social mistakes, consider it failure; this is unhelpful

  • Help point out what they need help with

  • Have severe akrasia, would like more “get things done” magic tools

  • Getting to know the bay area rationalist community

  • General bitterness/burned out

  • Reduce insecurity/fear around sharing

  • Avoiding spending most words signaling to have read a particular thing; want to communicate more clearly

  • Creating systems that reinforce unnoticed good behaviour

  • Would like to learn how to try at things

  • Find place in rationalist community

  • Staying connected with the group

  • Paying attention to what they want in the moment, in particular when it’s right to not be persistent

  • Would like to know the “landing points” to the community to meet & greet new people

  • Become more approachable, & be more willing to approach others for help; community cohesiveness

  • Have been lonely most of life; want to find a place in a really good healthy community

  • Re: prosocialness, being too low on Maslow’s hierarchy to help others

  • Abundance mindset & not stressing about how to pay rent

  • Cultivate stance of being able to do helpful things (action stance) but also be able to notice difference between laziness and mental health

  • Don’t know how to respect legit safety needs w/o getting overwhelmed by arbitrary preferences; would like to model people better to give them basic respect w/o having to do arbitrary amount of work

  • Starting conversations with new people

  • More rationalist group homes / baugruppe

  • Being able to provide emotional support rather than just logistics help

  • Reaching out to people at all without putting too much pressure on them

  • Cultivate lifelong friendships that aren’t limited to particular time and place

  • Have a block around asking for help bc doesn’t expect to reciprocate; would like to actually just pay people for help w stuff

  • Want to become more involved in the community

  • Learn how to teach other people “ops skills”

  • Connections to people they can teach and who can teach them

Lightning Talks

Lightning talks are a great way to give people an opportunity to not just share ideas, but get some practice at public presentation (which I've found can be a great gateway tool for overall confidence and ability to get things done in the community). Traditionally they are 5 minutes long. CFAR has found that 3.5 minute lightning talks are better than 5 minute talks, because it cuts out some rambling and tangents.

It turned out we had more people than I'd originally planned time for, so we ended up switching to two minute talks. I actually think this was even better, and my plan for next time is do 1-minute timeslots but allow people to sign up for multiple if they think their talk requires it, so people default to giving something short and sweet.

Rough summaries of the lightning talks can be found in the notes.

III. Discussing the Problem

The next section involved two "breakout session" - two 20 minute periods for people to split into smaller groups and talk through problems in detail. This was done in an somewhat impromptu fashion, with people writing down the talks they wanted to do on the whiteboard and then arranging them so most people could go to a discussion that interested them.

The talks were:

 -  Welcoming Newcomers
 -  How to handle people who impose costs on others?
 -  Styles of Leadership and Running Events
 -  Making Helping Fun (or at least lower barrier-to-entry)
 -  Circling session 

There was a suggested discussion about outreach, which I asked to table for a future unconference. My reason was that outreach discussions tend to get extremely meta and seem to be an attractor (people end up focusing on how to bring more people into the community without actually making sure the community is good, and I wanted the unconference to focus on the latter.)

I spent some time drifting between sessions, and was generally impressed both with the practical focus each discussion had, as well as the way they were organically moderated.

Again, more details in the notes.

IV. Planning Solutions and Next Actions

After about an hour of discussion and mingling, we came back to the central common space to describe key highlights from each session, and begin making concrete plans. (Names are crediting people who suggested an idea and who volunteered to make it happen)

Creating Norms for Your Space (Jane Joyce, Tilia Bell)

The "How to handle people who impose costs on other" conversation ended up focusing on minor but repeated costs. One of the hardest things to moderate as an event host is not people who are actively disruptive, but people who just a little bit awkward or annoying - they'd often be happy to change their behavior if they got feedback, but giving feedback feels uncomfortable and it's hard to do in a tactful way. This presents two problems at once: parties/events/social-spaces end up a more awkward/annoying than they need to be, and often what happens is that rather than giving feedback, the hosts stop inviting people doing those minor things, which means a lot of people still working on their social skills end up living in fear of being excluded.

Solving this fully requires a few different things at once, and I'm not sure I have a clear picture of what it looks like, but one stepping stone people came up with was creating explicit norms for a given space, and a practice of reminding people of those norms in a low-key, nonjudgmental way.

I think will require a lot of deliberate effort and practice on the part of hosts to avoid alternate bad outcomes like "the norms get disproportionately enforced on people the hosts like and applied unfairly to people they aren't close with". But I do think it's a step in the right direction to showcase what kind of space you're creating and what the expectations are.

Different spaces can be tailored for different types of people with different needs or goals. (I'll have more to say about this in an upcoming post - doing this right is really hard, I don't actually know of any groups that have done an especially good job of it.)

I *was* impressed with the degree to which everyone in the conversation seemed to be taking into account a lot of different perspectives at once, and looking for solutions that benefited as many people as possible.

Welcoming Committee (Mandy Souza, Tessa Alexanian)

Oftentimes at events you'll see people who are new, or who don't seem comfortable getting involved with the conversation. Many successful communities do a good job of explicitly welcoming those people. Some people at the unconference decided to put together a formal group for making sure this happens more.

The exact details are still under development, but I think the basic idea is to have a network of people who are interested
he idea is to have a group of people who go to different events, playing the role of the welcomer. I think the idea is sort of a "Uber for welcomers" network (i.e. it both provides a place for people running events to go to ask for help with welcoming, and people who are interested in welcoming to find events that need welcomers)

It also included some ideas for better infrastructure, such as reviving "bayrationality.org" to make it easier for newcomers to figure out what events are going on (possibly including links to the codes of conduct for different spaces as well). In the meanwhile, some simple changes were the introduction of a facebook group for Bay Area Rationalist Social Events.

Softskill-sharing Groups (Mike Plotz and Jonathan Wallis)

The leadership styles discussion led to the concept that in order to have a flourishing community, and to be a successful leader, it's valuable to make yourself legible to others, and others more legible to yourself. Even small improvements in an activity as frequent as communication can have huge effects over time, as we make it easier to see each other as we actually are and to clearly exchange our ideas. 

A number of people wanted to improve in this area together, and so we’re working towards establishing a series of workshops with a focus on practice and individual feedback. A longer post on why this is important is coming up, and there will be information on the structure of the event after our first teacher’s meeting. If you would like to help out or participate, please fill out this poll:

https://goo.gl/forms/MzkcsMvD2bKzXCQN2

Circling Explorations (Qiaochu and others)

Much of the discussion at the Unconference, while focused on community, ultimately was explored through an intellectual lens. By contrast, "Circling" is a practice developed by the Authentic Relating community which is focused explicitly on feelings. The basic premise is (sort of) simple: you sit in a circle in a secluded space, and you talk about how you're feeling in the moment. Exactly how this plays out is a bit hard to explain, but the intended result is to become better both at noticing your own feelings and the people around you.

Opinions were divided as to whether this was something that made sense for "rationalists to do on their own", or whether it made more sense to visit more explicitly Circling-focused communities, but several people expressed interest in trying it again.

Making Helping Fun and More Accessible (Suggested by Oliver Habryka)

Ultimately we want a lot of people who are able and excited to help out with challenging projects - to improve our collective group ambition. But to get there, it'd be really helpful to have "gateway helping" - things people can easily pitch in to do that are fun, rewarding, clearly useful but on the "warm fuzzies" side of helping. Oliver suggested this as a way to get people to start identifying as people-who-help.

There were two main sets of habits that worth cultivating:

1) Making it clear to newcomers that they're encouraged to help out with events, and that this is actually a good way to make friends and get more involved. 

2) For hosts and event planners, look for opportunities to offer people things that they can help with, and make sure to publicly praise those who do help out.

Some of this might dovetail nicely with the Welcoming Committee, both as something people can easily get involved with, and if there ends up being a public facing website to introduce people to the community, using that to connect people with events that could use help).

Volunteering-as-Learning, and Big Event Specific Workshops

Sometimes volunteering just requires showing up. But sometimes it requires special skills, and some events might need people who are willing to practice beforehand or learn-by-doing with a commitment to help at multiple events.

A vague cluster of skills that's in high demand is "predict logistical snafus in advance to head them off, and notice logistical snafus happening in realtime so you can do something about them." Earlier this year there was an Ops Workshop that aimed to teach this sort of skill, which went reasonably but didn't really lead into a concrete use for the skills to help them solidify.

One idea was to do Ops workshops (or other specialized training) in the month before a major event like Solstice or EA Global, giving them an opportunity to practice skills and making that particular event run smoother.

(This specific idea is not currently planned for implementation as it was among the more ambitious ones, although Brent Dill's series of "practice setting up a giant dome" beach parties in preparation for Burning Man are pointing in a similar direction)

Making Sure All This Actually Happens (Sarah Spikes, and hopefully everyone!)

To avoid the trap of dreaming big and not actually getting anything done, Sarah Spikes volunteered as project manager, creating an Asana page. People who were interested in committing to a deadline could opt into getting pestered by her to make sure things things got done. 

V. Parting Words

To wrap up the event, I focused on some final concepts that underlie this whole endeavor. 

The thing we're aiming for looks something like this:

In a couple months (hopefully in July), there'll be a followup unconference. The theme will be "Innovation and Excellence", addressing the twofold question "how do we encourage more people to start cool projects", and "how to do we get to a place where longterm projects ultimately reach a high quality state?"

Both elements feel important to me, and they require somewhat different mindsets (both on the part of the people running the projects, and the part of the community members who respond to them). Starting new things is scary and having too high standards can be really intimidating, yet for longterm projects we may want to hold ourselves to increasingly high standards over time.

My current plan (subject to lots of revision) is for this to become a series of community unconferences that happen roughly every 3 months. The Bay area is large enough with different overlapping social groups that it seems worthwhile to get together every few months and have an open-structured event to see people you don't normally see, share ideas, and get on the same page about important things.

Current thoughts for upcoming unconference topics are:

Innovation and Excellence
Personal Epistemic Hygiene
Group Epistemology

An important piece of each unconference will be revisiting things at the previous one, to see if projects, ideas or experiments we talked about were actually carried out and what we learned from them (most likely with anonymous feedback collected beforehand so people who are less comfortable speaking publicly have a chance to express any concerns). I'd also like to build on topics from previous unconferences so they have more chance to sink in and percolate (for example, have at least one talk or discussion about "empathy and trust as related to epistemic hygiene").

Starting and Finishing Unconferences Together

My hope is to get other people involved sooner rather than later so this becomes a "thing we are doing together" rather than a "thing I am doing." One of my goals with this is also to provide a platform where people who are interested in getting more involved with community leadership can take a step further towards that, no matter where they currently stand (ranging anywhere from "give a 30 second lightning talk" to "run a discussion, or give a keynote talk" to "be the primary organizer for the unconference.")

I also hope this is able to percolate into online culture, and to other in-person communities where a critical mass of people think this'd be useful. That said, I want to caution that I consider this all an experiment, motivated by an intuitive sense that we're missing certain things as a culture. That intuitive sense has yet to be validated in any concrete fashion. I think "willingness to try things" is more important than epistemic caution, but epistemic caution is still really important - I recommend collecting lots of feedback and being willing to shift direction if you're trying anything like the stuff suggested here.

(I'll have an upcoming post on "Ways Project Hufflepuff could go horribly wrong")

Most importantly, I hope this provides a mechanism for us to collectively take ideas more seriously that we're ostensibly supposed to be taking seriously. I hope that this translates into the sort of culture that The Craft and The Community was trying to point us towards, and, ideally, eventually, a concrete sense that our community can play a more consistently useful role at making sure the world turns out okay. 

If you have concerns, criticism, or feedback, I encourage you to comment here if you feel comfortable, or on the Unconference Feedback Form. So far I've been erring on the side of move forward and set things in motion, but I'll be shifting for the time being towards "getting feedback and making sure this thing is steering in the right direction."

-

In addition to the people listed throughout the post, I'd like to give particular thanks to Duncan Sabien for general inspiration and a lot of concrete help, Lahwran for giving the most consistent and useful feedback, and Robert Lecnik for hosting the space. 

View more: Next