Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

Comment author: Habryka 17 September 2017 02:48:45AM 1 point [-]

We are planning to leave the wiki up, and probably restyle it at some point, so it will not be gone. User accounts will no longer be shared though, for the foreseeable future, which I don't think will be too much of an issue.

But I don't yet have a model of how to make the wiki in general work well. The current wiki is definitely useful, but I feel that it's main use has been the creation of sequences and collections of posts, which is now integrated more deeply into the site via the sequences functionality.

Comment author: Wei_Dai 17 September 2017 04:56:53PM 3 points [-]

The wiki is also useful for defining basic concepts used by this community, and linking to them in posts and comments when you think some of your readers might not be familiar with them. It might also be helpful for outreach, for example our wiki page for decision theory shows up in the first page of Google results for "decision theory".

Comment author: IlyaShpitser 17 September 2017 01:08:59AM *  2 points [-]

Google is using a much more complicated algorithm that is constantly tweaked, and is a trade secret -- precisely because as soon as it became profitable to do so, the ecosystem proceeded to game the hell out of PageRank.

Google hasn't been using PageRank-as-in-the-paper for ages. The real secret sauce behind Google is not eigenvalues, it's the fact that it's effectively anti-inductive, because the algorithm isn't open and there is an army of humans looking for attempts to game it, and modifying it as soon as such an attempt is found.

Comment author: Wei_Dai 17 September 2017 02:04:03AM 7 points [-]

Given that, it seems equally valid to say "this will work, for the same reason that PageRank worked", i.e., we can also tweak the reputation algorithm as people try to attack it. We don't have as much resources as Google, but then we also don't face as many attackers (with as strong incentives) as Google does.

I personally do prefer a forum with karma numbers, to help me find quality posts/comments/posters that I would likely miss or have to devote a lot of time and effort to sift through.

Comment author: paulfchristiano 16 September 2017 03:43:55AM 0 points [-]

I think that Facebook's behavior has probably gotten worse over time as part of general move towards cashing in / monetizing.

I don't think I've looked at my feed in a few years.

On the original point: I think at equilibrium services like Facebook maximize total welfare, then take their cut in a socially efficient way (e.g. as payment). I think the only question is how long it takes to get there.

Comment author: Wei_Dai 17 September 2017 01:33:51AM 0 points [-]

On the original point: I think at equilibrium services like Facebook maximize total welfare, then take their cut in a socially efficient way (e.g. as payment). I think the only question is how long it takes to get there.

Why? There are plenty of theoretical models in economics where at equilibrium total welfare does not get maximized. See this post and the standard monopoly model for some examples. The general impression I get from studying economics is that the conditions under which total welfare does get maximized tend to be quite specific and not easy to obtain in practice. Do you agree? In other words, do you generally expect markets to have socially efficient equilibria and expect Facebook to be an instance of that absent a reason to think otherwise, or do you think there's something special about Facebook's situation?

Comment author: John_Maxwell_IV 09 September 2017 07:02:58AM 3 points [-]

So what's the fix here? If people think mailing lists work better than peer review, maybe an organization like OpenPhil should set up a mailing list for academics working on AI safety and award grants based on discussions on the mailing list? Academia has a lot of momentum behind it, and it seems more efficient to redirect that momentum than try to set up something new from scratch.

Comment author: Wei_Dai 09 September 2017 07:21:01PM 0 points [-]

It's probably not as simple as that. Part of why online discussions work as well as they do is probably that there's no money riding on them. If funders start making grant decisions based on mailing list discussions, we might start seeing mailing lists becoming politicized to an uncomfortable and unproductive degree. I think for now the "fix" is just for people to monitor efforts to reform peer review in academia and adopt the ones that work well into the AI safety field, and also maintain a number of AI safety research institutions with diverse cultures instead of e.g. demanding that everyone publish in academic venues as a condition for funding.

Comment author: paulfchristiano 02 December 2016 06:08:40PM *  1 point [-]

How do you expect this to happen?

I think there are two mechanisms:

  • Public image is important to companies like Facebook and Google. I don't think that they will charge for a user-aligned version, but I also don't think there would be much cost to ad revenue from moving in this direction. E.g. I think they might cave on the fake news thing modulo the proposed fixes mostly being terrible ideas. Optimizing user preferences may be worth it in the interests of a positive public image alone.
  • I don't think that Facebook ownership and engineers are entirely profit-focused, they will sometimes do things just because they feel like it makes the world better at modest cost. (I know more people in Google and am less informed about FB.)

Relating the two, if e.g. Google organized its services in this way, if the benefits were broadly and understood, and if Facebook publicly continued to optimize for things that its users don't want optimized, I think it could be bad for the image of Facebook (with customers, and especially with hires).

I'd be quite surprised if any of these happened.

Does this bear on our other disagreements about how optimistic to be about humanity? Is it worth trying to find a precise statement and making a bet?

I'm probably willing to give > 50% on something like: "Within 5 years, there is a Google or Facebook service that conducts detailed surveys of user preferences about what content to display and explicitly optimizes for those preferences." I could probably also make stronger statements re: scope of adoption.

And why isn't it a bad sign that Facebook hasn't already done what you suggested in your post?

I think these mechanisms probably weren't nearly as feasible 5 years ago as they are today, based on gradual shifts in organization and culture at tech companies (especially concerning ML). And public appetite for more responsible optimization has been rapidly increasing. So I don't think non-action so far is a very strong sign.

Also, Facebook seems to sometimes do things like survey users on how much they like content, and include ad hoc adjustments to their optimization in order to produce more-liked content (e.g. downweighting like-baiting posts). In in some sense this is just a formalization of that procedure. I expect in general that formalizing optimizations will become more common over the coming years, due to a combination of increasing usefulness of ML and cultural change to accommodate ML progress.

Comment author: Wei_Dai 09 September 2017 04:58:00PM *  0 points [-]

I'm curious if you occasionally unblock your Facebook newsfeed to check if things have gotten better or worse. I haven't been using Facebook much until recently, but I've noticed a couple of very user-unfriendly "features" that seem to indicate that FB just doesn't care much about its public image. One is suggested posts (e.g., "Popular Across Facebook") that are hard to distinguish from posts from friends, and difficult to ad-block (due to looking just like regular posts in HTML). Another is fake instant message notifications on the mobile app whenever I "friend" someone new, that try to entice me into installing its instant messaging app (only to find out that the "notification" merely says I can now instant message that person). If I don't install the IM app, I get more and more of these fake notifications (2 from one recent "friend" and 4 from another).

Has it always been this bad or even worse in the past? Does it seem to you that FB is becoming more user-aligned, or less?

ETA: I just saw this post near the top of Hacker News, pointing out a bunch of other FB features designed to increase user engagement at the expense of their actual interests. The author seems to think the problem has gotten a lot worse over time.

Comment author: IlyaShpitser 08 September 2017 03:40:39PM *  0 points [-]

You know what I will say, yall should stay in your lane, re: incentives.

Yudkowsky's incentives caused him to write HPMOR (which has precisely zero (0) academic value), and publish basically nothing. So as far as the mainstream is concerned his footprint does not exist. He's collecting a salary at MIRI, presumably. What is that salary buying?

Mainstream academics who collect a salary will say they teach undergraduates, and publish stuff to make grant agencies happy. Some of that stuff is useless, a lot of it is very useful indeed.


Reform attempts for "non-aligned" ecosystems like academia will almost certainly not work because (as you all are well aware) "aligning" is hard.


MIRI has the same problem everyone else has: if it grows it will become a non-aligned ecosystem, if it doesn't grow it will not have any impact.

Comment author: Wei_Dai 09 September 2017 07:36:48AM *  1 point [-]

You know what I will say, yall should stay in your lane, re: incentives.

I don't understand this. Please clarify? (Urban dictionary says "stay in your lane" means mind your own business, which is exactly what we're doing, namely trying to figure out what direction to push our own culture.)

and publish basically nothing

He's publishing mostly on Arbital these days. See this and this for examples. I'm not sure why he doesn't at least post links elsewhere to draw people's attention though. Hopefully that will change after LW 2.0 goes live.

So as far as the mainstream is concerned his footprint does not exist.

I'm not sure what you mean by this either. Certainly the people who work on AI safety at Berkeley, OpenAI, and DeepMind all know about Eliezer and MIRI's approach to AI alignment, even if they don't agree that it's the most promising one. Are you saying that if Eliezer had published in academia, they'd be more inclined to follow that approach, as opposed to the more ML-based approaches that they're currently following?

MIRI has the same problem everyone else has: if it grows it will become a non-aligned ecosystem

I think having "aligned" human institutions is too much to hope for. As I mentioned elsewhere in this thread, perhaps the best we can do is to have different bad incentives / inefficiencies in different institutions so that they're able to reach different sets of low hanging fruit, and not all suffer from the same collective blind spots.

Comment author: cousin_it 07 September 2017 10:01:41AM *  1 point [-]

As long as MIRI is led and funded by people who care about the actual goal rather than citations, I don't see why we would go astray.

Comment author: Wei_Dai 07 September 2017 04:49:47PM 1 point [-]

I can see a couple different ways that it could happen. Funders might have trouble judging actual progress in the absence of academic peer-reviewed publications and citations. Especially as more academics join the AI risk field and produce more papers and citations, funders might be tempted to think that they should re-direct resources towards academia (in part for subconscious status reasons). MIRI may have to switch to more academic norms in order to compete, which would then rub off on LW. (This seems to already be happening to some extent.) Or LW moves towards a more academic culture for internal status-economics reasons, and MIRI leaders may not have much control over that. (In that world, maybe LWers will eventually look down upon MIRI for not being sufficiently academic.)

Comment author: cousin_it 05 September 2017 09:29:48PM *  1 point [-]

Yeah, I agree that the connection to academia shouldn't be the end goal, but it could be one of several factors that help.

Comment author: Wei_Dai 06 September 2017 12:56:08PM *  1 point [-]

We also need to figure out how to avoid the bad incentives of academia. For example, to avoid the problem of publishing papers for the sake of publishing or for the sake of gaining citation counts, we should only reward someone with status if the act of academic publishing leads to positive consequences, for example productive academic research that otherwise wouldn't likely to have occurred (and not just additional citations), or practical deployment of the idea that otherwise wouldn't likely to have occurred. But this will be hard to do in practice, whereas counting papers / citations will be easy.

We have to keep in mind that the people who created the status / monetary economy in academia surely didn't intend to cause the incentive problems that now exist within it, and many people both in academia and out (e.g., policy makers, funders) have probably since noticed those problems and would love to have ways to fix them, but the bad incentives still exist. In some sense, inefficiencies are simply inevitable due to the multi-player nature of the game. The best we can do is perhaps to have a different set of inefficiencies / bad incentives, which allow us to reach a different (and not necessarily larger) set of low-hanging fruit.

I think this suggests that we should be ruthless about avoiding becoming just like academia, and throw out the baby with the bathwater if necessary to avoid it. For example, if we can't figure out a foolproof way of rewarding academic publishing only when it leads to positive consequences, we should assume that trying to reward academic publishing will lead to inefficiencies / bad incentives similar to ones existing in academia, and therefore not try to reward academic publishing at all. Or, alternatively, we need to create the kind of culture where we can say, "oops, rewarding people for academic publishing is making us too much like academia in terms of sharing the same set of inefficiencies / bad incentives, so we shouldn't be rewarding people for academic publishing anymore" but that seems significantly harder to accomplish. Academia is very likely a basin of attraction in the space of culture and institutional design, and we risk irreversibly falling into it just by getting close.

Comment author: cousin_it 05 September 2017 08:05:54PM *  8 points [-]

Some ideas:

  • Open up MIRI's internal discussions about strategy as they happen.
  • Open up MIRI's workshops as they happen, let people participate remotely.
  • Prizes, like Quantified Health or Paul's recent offer of funding for AI alignment research.
  • Post drafts of papers before publishing them.
  • Summarize and discuss ideas from Arbital.
  • Summarize and discuss Paul's work.
  • Guest posts / debates / AMAs with high profile non-LW people. Think someone like Nick Bostrom ideally.
  • Merge IAFF and the MIRI news blog into LW 2.0.
Comment author: Wei_Dai 05 September 2017 08:52:43PM 2 points [-]

These are great ideas, and makes me optimistic that LW 2.0 can succeed. :)

Comment author: cousin_it 05 September 2017 04:35:04PM *  7 points [-]

Thank you for not giving up on this discussion! Many people have mentioned the intellectual benefits of peer review, but I just thought of another argument that might be new to you.

Many of us agree that solving problems together is great fun. But what if it's just rationalization? What if we really want to participate in some status economy, and will come up with smart things to say only if we're paid with status in return? I know it's not true for you, because you came up with UDT on your own. But it's definitely true for me. Posting something like this and getting no response feels very discouraging to me, even if the topic is exciting. And since I'm close to the top of the LW heap, I imagine it's even more true for others.

The question then becomes, how do we set up a status economy that will encourage research? Peer review is one way, because publications and citations are a status badge desired by many people. Participating in a forum like LW when it's "hot" and frequented by high status folks is another way, but unfortunately we don't have that anymore. From that perspective it's easy to see why the massively popular HPMOR didn't attract many new researchers to AI risk, but attracted people to HPMOR speculation and rational fic writing. People do follow their interests sometimes, but mostly they try to find venues to show off.

Of course you could be happy with a system that's optimized for people like you, with few status rewards. But I suspect you'd miss out on many good contributors (think of all the smart people who drifted away from LW in recent years). I'd prefer to have something more like a pyramid or funnel, with popular appeal on one end and intellectual progress on the other. Academic credibility (including peer review) could be a key part of that funnel for us, and a central forum like LW would also help a lot. There are probably other measures that could work in synergy with these.

I wonder if people at MIRI think the same way. In a sense, the funnel idea was there from the beginning, as "raising the sanity waterline". CFAR can also be seen as part of that. But these efforts are mostly aimed at outreach, and I'm not sure they ever consciously tried to build a mechanism for converting status to research. What would it take to build such a mechanism today?

Comment author: Wei_Dai 05 September 2017 08:47:55PM 1 point [-]

From that perspective it's easy to see why the massively popular HPMOR didn't attract many new researchers to AI risk, but attracted people to HPMOR speculation and rational fic writing.

I think this is a nice insight that hadn't occurred to me before.

Participating in a forum like LW when it's "hot" and frequented by high status folks is another way, but unfortunately we don't have that anymore.

From looking around on the LW 2.0 closed beta, it will have some features specifically designed to attract some of the people that left, like trusted authors can have their own areas where they exercise greater moderation power. This will also hopefully prevent new high status folks from leaving later.

I want to echo Dr_Manhattan and suggest that you take a look at LW2 beta and see what more can be done there to support your ideas. They are planning to launch on November 1 with an open beta a few weeks before, so major new features are probably out (at least until later), but things like changes to the karma or moderation system are probably still possible. The people behind LW2 are planning to write a post soon about the karma changes and ask for review/suggestions so you can hold off your ideas until then as well.

Academic credibility (including peer review) could be a key part of that funnel for us

How do you envision this? Like if we get results published in academia, that will draw more people into this community? This makes me a bit worried that if being published in academia is the ultimate marker of status in this community, that'll discourage people who have a distaste for academia (like me when I first joined). May still be a good idea though...

View more: Next