Less Wrong is a community blog devoted to refining the art of human rationality. Please visit our About page for more information.

SatvikBeri comments on On the importance of Less Wrong, or another single conversational locus - Less Wrong

82 Post author: AnnaSalamon 27 November 2016 05:13PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (357)

You are viewing a single comment's thread. Show more comments above.

Comment author: SatvikBeri 27 November 2016 04:59:00PM 7 points [-]

On (4), does anyone have a sense of how much it would cost to improve the code base? Eg would it be approximately $1k, $10k, or $100k (or more)? Wondering if it makes sense to try and raise funds and/or recruit volunteers to do this.

Comment author: Vaniver 27 November 2016 05:17:21PM 6 points [-]

I think a good estimate is close to $10k. Expect to pay about $100/hr for developer time, and something like 100 hours of work to get from where we are to where we want to be doesn't seem like a crazy estimate. Historically, the trouble has been finding people willing to do the work, not the money to fund people willing to do the work.

If you can find volunteers who want to do this, we would love code contributions, and you can point them towards here to see what needs to be worked on.

Comment author: Viliam 27 November 2016 09:50:29PM *  23 points [-]

I think you are underestimating this, and a better estimate is "$100k or more". With an emphasis on the "or more" part.

Historically, the trouble has been finding people willing to do the work, not the money to fund people willing to do the work.

Having "trouble to find people willing to do the work" usually means you are not paying enough to solve the problem. Market price, by definition, is a price at which you can actually buy a product or service, not a price that seems like it should be enough but you just can't find anyone able and/or willing to accept the deal.

The problem with volunteers is that LW codebase needs too much highly specialized knowledge. Python and Ruby just to get a chance, and then study the code which was optimized for perfomance and backwards compatibility, at the expense of legibility and extensibility. (Database-in-the-database antipattern; values precomputed and cached everywhere.) Most of the professional programmers are simply unable to contribute, without spending a lot of time studying something they will never use again. For a person who has the necessary skills, $10k is about their monthly salary (if you include taxes), and one month feels like too short time to understand the mess of the Reddit code, and implement everything that needs to be done. And the next time, if you need another upgrade, and the same person isn't available, you need another person to spend the same time to understand the Reddit code.

I believe in long term it would be better to rewrite the code from scratch, but that's definitely going to take more than one month.

Comment author: Vaniver 27 November 2016 10:28:58PM 3 points [-]

Having "trouble to find people willing to do the work" usually means you are not paying enough to solve the problem.

I had difficulties finding people without mentioning a price; I'm pretty sure the defect was in where and how I was looking for people.

I also agree that it makes more sense to have a small number of programmers make extensive changes, rather than having a large number of people become familiar with how to deal with LW's code.

I believe in long term it would be better to rewrite the code from scratch, but that's definitely going to take more than one month.

I will point out there's no strong opposition to replacing the current LW codebase with something different, so long as we can transfer over all the old posts without breaking any links. The main reason we haven't been approaching it that way is that it's harder to make small moves and test their results; either you switch over, or you don't, and no potential replacement was obviously superior.

Comment author: ananda 29 November 2016 05:31:53PM 18 points [-]

I'm new and came here from Sarah Constantin's blog. I'd like to build a new infrastructure for LW, from scratch. I'm in a somewhat unique position to do so because I'm (1) currently searching for an open source project to do, and (2) taking a few months off before starting my next job, granting the bandwidth to contribute significantly to this project. As it stands right now, I can commit to working full time on this project for the next three months. At that point, I will continue to work on the project part time and it will be robust enough to be used in an alpha or beta state, and attract devs to contribute to further development.

Here is how I envision the basic architecture of this project:

  1. A server that manages all business logic (i.e. posting, moderation, analytics) and interfaces with the frontend (2) and database (3).
  2. A standalone, modular frontend (probably built with React, maybe reusing components provided by Telescope) that is modern, beautiful, and easily extensible/composable from a dev perspective.
  3. A database, possibly NoSql given the nature of the data that needs to be stored (posts, comments, etc). The first concern is security, all others predicated on that.

I will kickstart all three parts and bring them to a good place. After this threshold, I will need help with the frontend - this is not my forte and will be better executed by someone passionate about it.

I'm not asking for any compensation for my work. My incentive is to create a project that is actually immediately useful to someone; open-sourcing it and extending that usability is also nice. I also sympathize with the LW community and the goals laid out in this post.

I considered another approach: reverse-engineer HackerNews and use that as the foundation to be adapted to LW's unique needs. If this approach would be of greater utility to LW, I'd be happy to take it.

Comment author: Vaniver 29 November 2016 08:19:06PM 5 points [-]

Thanks for the offer! Maybe we should talk by email? (this username @ gmail.com)

Comment author: Gram_Stone 29 November 2016 05:41:21PM 2 points [-]

If you don't get a proper response, it may be worthwhile to make this into its own post, if you have the karma. (Open thread is another option.)

Comment author: ChristianKl 11 December 2016 09:24:37PM 0 points [-]

I considered another approach: reverse-engineer HackerNews and use that as the foundation to be adapted to LW's unique needs

Currently HackerNews and LW both run on the Reddit code base. On of the problems is that Reddit didn't design their software to be easily adopted to new projects. That means it's not easily possible to update the code with new versions.

A database, possibly NoSql given the nature of the data that needs to be stored (posts, comments, etc).

A lot of the data will be votes.

Comment author: whpearson 11 December 2016 10:58:57PM *  2 points [-]

Nitpick: Hackernews isn't reddit derived. It is some written in arc. And not open source.

Comment author: Drea 11 December 2016 08:12:35PM 0 points [-]

I see various people volunteering for different roles. I'd be interested in providing design research and user experience support, which would probably only be needed intermittently if we have someone acting as a product manager. It might be nice to have someone in a light-weight graphic design role as well, and that can be freelance.

Like ananda, I'm happy to do this as an open-contribution project rather than paid. I'll reach out to Vaniver via email.

Comment author: arunkhanna00 07 December 2016 05:34:20AM 0 points [-]

I have some front-end experience and would love to help you(I'm a student). Email me at my username @gmail.com

Comment author: Viliam 27 November 2016 10:57:09PM *  9 points [-]

Well, if someone would be willing me to pay for one year of full-time work, I would be happy to rewrite the LW code from scratch. Maybe one year is an overestimate, but maybe not -- there is this thing known as planning fallacy. That would cost somewhat less than $100k. Let's say $100k, and that included a reserves for occassionally paying someone else to help me with some specific thing, if needed.

I am not saying that paying me for this job is a rational thing to do; let's just take this as an approximate estimate of the upper bound. (The lower bound is hoping that one day someone will appear and do it for free. Probably also not a rational thing to do.)

Maybe it was a mistake that I didn't mention this option sooner... but hearing all the talk about "some volunteers doing it for free in their free time" made me believe that this offer would be seen as exaggerated. (Maybe I was wrong. Sorry, can't change the past.)

I certainly couldn't do this in my free time. And trying to fix the existing code would probably take just as much time, the difference being that at the end, instead of new easily maintainable and extensible code, we would have the same old code with a few patches.

And there is also a risk that I am overestimating my abilities here. I never did a project of this scale alone. I mean, I feel quite confident that I could do it in a given time frame, but maybe there would be problems with performance, or some kind of black swan.

I will point out there's no strong opposition to replacing the current LW codebase with something different, so long as we can transfer over all the old posts without breaking any links.

I would probably try to solve it as a separate step. First, make the new website, as good as possible. Second, import the old content, and redirect the links. Only worry about the import when the new site works as expected.

Or maybe don't even import the old stuff, and keep the old website frozen. Just static pages, without ability to edit anything. All we lose is the ability to vote or comment on a years-old content. At the moment of transition, open officially the new website, block the ability to post new articles on the old one, but still allow people to post comments on the old one for the following three months. At the end, all old links will work, read-only.

Comment author: WalterL 01 December 2016 08:39:45PM *  2 points [-]

Not trolling here, genuine question.

How is the LW codebase so awful? What makes it so much more complicated than just a typical blog, + karma? I feel like I must be missing something.

From a UI perspective it is text boxes and buttons. The data structure that you need to track doesn't SEEM too complicated (Users have names, karma totals, passwords and roles? What am I not taking into account?

Comment author: Vaniver 01 December 2016 09:20:27PM 4 points [-]

How is the LW codebase so awful?

Age, mostly. My understanding is Reddit was one of the first of its kind, and so when building it they didn't have a good sense of what they were actually making. One of the benefits of switching to something new is not just that it's using technology people are more likely to be using in their day jobs, but also that the data arrangement is more aligned with how the data is actually used and thought about.

Comment author: jackk 05 December 2016 04:34:02AM 0 points [-]

It's also tied to some pretty old versions of Python and various libraries, and people often need help just getting the development server going.

Comment author: Lumifer 01 December 2016 08:55:34PM *  0 points [-]

It's a modified copy of an early Reddit codebase. Besides it has, um, founder effects X-/ -- for example the backend SQL database is used just as an engine behind a handcrafted key-value store...

Comment author: alyssavance 27 November 2016 05:37:29PM 2 points [-]

If the money is there, why not just pay a freelancer via Gigster or Toptal?

Comment author: Vaniver 27 November 2016 05:55:46PM *  6 points [-]

Historically, the answers have been things like a desire to keep it in the community (given the number of software devs floating around), the hope that volunteer effort would come through, and me not having much experience with sites like those and thus relatively low affordance for that option. But I think if we pay for another major wave of changes, we'll hire a freelancer through one of those sites.

(Right now we're discussing how much we're willing to pay for various changes that could be made, and once I have that list I think it'll be easy to contact freelancers, see if they're cheap enough, and then get done the things that make sense to do.)

[edit] I missed one--until I started doing some coordination work, there wasn't shared knowledge of what sort of changes should actually be bought. The people who felt like they had the authority to design changes didn't feel like they had the authority to spend money, but the people who felt like they had the authority to spend money didn't feel like they had the authority to design changes, and both of them had more important things to be working on.

Comment author: John_Maxwell_IV 28 November 2016 03:23:15AM *  5 points [-]

The people who felt like they had the authority to design changes didn't feel like they had the authority to spend money, but the people who felt like they had the authority to spend money didn't feel like they had the authority to design changes, and both of them had more important things to be working on.

This sort of leadership vacuum seems to be a common problem in the LW community. Feels to me like people can err more on the side of assuming they have the authority to do things.

Comment author: SatvikBeri 28 November 2016 03:50:08AM 5 points [-]

Yeah, a good default is the UNODIR pattern ("I will do X at Y time unless otherwise directed")

Comment author: skeptical_lurker 01 December 2016 12:17:39AM 0 points [-]

I can code in python, but I have no web dev experience - I could work out what algorithms are needed, but I'm not sure I would know how to implement them, at least not off the bat.

Still, I'd be willing to work on it for less then $100 per hour.

Comment author: Vaniver 01 December 2016 06:39:53PM 0 points [-]

Thanks for the offer!

Still, I'd be willing to work on it for less then $100 per hour.

If you're working for $x an hour, do you think you would take fewer that 100/x times as long as someone who is experienced at web dev?

Comment author: skeptical_lurker 01 December 2016 07:27:49PM 0 points [-]

If you're working for $x an hour, do you think you would take fewer that 100/x times as long as someone who is experienced at web dev?

Fair pay would be $x an hour given that it takes me 100/x times as long as someone who is experienced at web dev. However in reality estimates of how long the work will take seem to vary wildly - for instance you and Viliam disagree by an order of magnitude.

The more efficient system might be for me to work with someone who does have some web dev experience, if there is someone else working on this.