JoshuaZ comments on No One Can Exempt You From Rationality's Laws - Less Wrong
You are viewing a comment permalink. View the original post to see all comments and the full post content.
You are viewing a comment permalink. View the original post to see all comments and the full post content.
Comments (51)
Somewhat off-topic and answering an ancient comment, but a useful reminder of how important endeavours can actually be horribly short of resources and much more fragile than people think:
We know this :-) Wikipedia as monopoly provider of the world's encyclopedia is an anti-pattern. But the network effects are very powerful.
This means we are a great big single point of failure. Our single data centre is one hurricane away from disappearing. Even making a good backup of English Wikipedia is a remarkably difficult endeavour because it's SO BIG. A billion and a half words. Can you mentally grasp how big that is? I sure can't.
And the distributed network you outline would be a wonderful thing. But, like most things that it would be nice to do with Wikipedia, it requires coding on MediaWiki. Lots of people have "Why don't you ..." technical ideas - nearly none of them follow them with the requisite code.
The budget for this year includes a pile of cash on technical resources: a second data centre and a lot more coders. We're also developing a pattern where young whizzkids work for WMF for a couple of years at charity pay and go off to make a bundle in industry - and that's fine by us.
Note that the size of 1.5 billion words isn't what really makes it so large. The real issue is the sheer number of revisions which increases the database size by orders of magnitude. The large number of images also contribute.
Yeah, it's the full history dump that basically hasn't worked properly in years.