XFrequentist comments on [LINK] Get paid to train your rationality - Less Wrong

27 Post author: XFrequentist 03 August 2011 03:01PM

You are viewing a comment permalink. View the original post to see all comments and the full post content.

Comments (55)

You are viewing a single comment's thread. Show more comments above.

Comment author: gwern 04 August 2011 05:31:21PM 4 points [-]
Comment author: XFrequentist 04 August 2011 06:00:25PM 1 point [-]

That is awesome!

Comment author: Morendil 04 August 2011 08:47:11PM 2 points [-]

Especially (mischievous mode ON) as I've only implied, not outright stated, that I've applied.

Mischievous mode OFF - that's a problem in arbitrating predictions, btw - the potential for ambiguity inherent in all human languages. If I hadn't in fact applied (I have), how should the prediction that I am "turned down" be judged?

I should use PredictionBook more often but I don't, partly due this kind of thing, also due to the trivial-inconvenience effort of having to come up with my own predictions to assess and the general uselessness for that purpose of the stream of other users' predictions.

Other than Tricycle folks, is anyone here on LW officially (or unofficially) "in charge" of maintaining and enhancing PredictionBook?

Comment author: gwern 04 August 2011 11:09:17PM *  3 points [-]

Other than Tricycle folks, is anyone here on LW officially (or unofficially) "in charge" of maintaining and enhancing PredictionBook?

I have some sort of moderator power; I am de facto in charge of the content house-keeping - editing bad due-dates, making private bad or long-overdue-unjudged predictions, criticizing predictions, etc. I also make and register hundreds of predictions, obviously.

(In addition, I have commit access to the codebase on GitHub, but I don't know Ruby, so I will probably never make use of said commit-bit.)

Comment author: Morendil 04 September 2011 03:53:59PM 1 point [-]

One thing that would probably greatly improve PB for my purposes is a tagging / filtering system, so that you could for instance pick out predictions about consumer devices or predictions about politics; or conversely leave out some uninteresting categories (e.g. predictions about the private lives of particular PB users, which I interpret as pure noise).

Comment author: gwern 04 September 2011 03:57:35PM 0 points [-]

Google is not sufficient, I take it?

Comment author: Morendil 04 September 2011 04:38:01PM 3 points [-]

No; I just tried the query "consumer electronics site:predictionbook.com", and that only returned 1 hit; I know there are more (including one I just made and another I just voted on). It really is the lack of user-supplied meta-information that prevents useful querying, not the lack of a UI for doing so. The UI encourages predictions to be written very tersely, and doesn't supply an extended-info field when you make a prediction.

PB.com is quite possibly the least well executed idea out there that I keep not giving up on. :)

Comment author: gwern 04 September 2011 05:37:13PM *  1 point [-]

Ah, that's what you meant by tags. Yes, that would be nice. On the other hand, I rather doubt that tags would instantly create massive demand for PB's services - other places like Intrade have well-categorized predictions/bets, and none of them have seen traffic explode the moment they implemented that feature.

If you really found tags all that valuable, you could start doing them inside comments. Go over the 969 upcoming predictions and add comments like 'tags: personal, exercise' or 'tags: America, politics'. Later, it'd be even easier to turn them into some real software-supported tags/categories, and in the meantime, you can query using Google. This wouldn't even take very long - at 30 predictions a day, which ought to take 10 minutes max, you'd be done in a month.

(I doubt you will adopt my suggestion and tag even 500 predictions (10%). This seems to be common to suggestions for PB: 'I'd use and really find PB useful if only it were executed better in this way', which of course never happens. It's starting to remind me of cryonics.)

Comment author: Morendil 04 September 2011 07:28:48PM *  1 point [-]

If you really found tags all that valuable, you could start doing them inside comments.

Preliminary report: this isn't going to work, not without drastic contortions in the choice of tags (which IMO kills the effectiveness of the tactic). For instance, from my first set of 30 I tagged a number with the tag "personal", predictions which only concern one user (or two acquainted with each other) and that I don't want to see because I can't effectively assess them. The Google query including "personal" returns close to 30 spurious results: for instance those containing "personal computer" or "personal transportation". (A temporary workaround is to include the term "tags" in the query, but this will cease to work once a greater fraction of predictions have been tagged.)

I doubt you will adopt my suggestion

You are correct about the likely outcome, but I think I've just proven your model of the underlying reasons wrong: I won't do it because it won't work, not because I lack the conscientiousness to do so, or because I'm too selfish to take on an effort that will benefit all users.

Comment author: gwern 04 September 2011 08:05:32PM 0 points [-]

The Google query including "personal" returns close to 30 spurious results: for instance those containing "personal computer" or "personal transportation".

JoshuaZ has (example) been adding brackets to the tags, such as [economics]. You don't mention forcing Google to include the brackets, so it's not surprising it includes those extra results.

Comment author: Morendil 04 September 2011 07:06:22PM 1 point [-]

This seems to be common to suggestions for PB: 'I'd use and really find PB useful if only it were executed better in this way', which of course never happens.

How many times was a new feature implemented as a test of such a hypothesis?

PB.com seems like it would be a great place for things like A/B testing and other tactics in the "Lean startup" repertoire, but what actually seems to be the case is that the site isn't under active development any more; no one is apparently trying to develop traffic or usage by improving the user experience there. (This isn't to slight your own efforts or your obvious enthusiasm; merely my best current hypothesis.)

(I'm finding the comparison with cryonics ironically apt, as a fence-straddling LW user who's not giving up on the entire project despite facing, as a non-US citizen, a battery of obstacles that I suspect also apply in the US, where they're just less obvious and as a result people take it for granted that things will "just work". Though it's more likely that the comparison is entirely besides the point and just a red herring for the purposes of present discussion.)

If you really found tags all that valuable, you could start doing them inside comments.

I'll try that, for a minimum of 30 predictions.

Comment author: gwern 04 September 2011 08:12:03PM 4 points [-]

How many times was a new feature implemented as a test of such a hypothesis?...PB.com seems like it would be a great place for things like A/B testing and other tactics in the "Lean startup" repertoire, but what actually seems to be the case is that the site isn't under active development any more; no one is apparently trying to develop traffic or usage by improving the user experience there.

This is true. Trike is not doing anything but maintenance because their options are to work on PB, LW, or Khan Academy. When I asked for features to be added and argued that work on PB could be justified, Matthew Fallshaw gave me Analytics data to look at. At that point, LW had ~140,000 unique visitors in the previous 30 days. And Khan Academy had a total of 25.2 million video watches. And Trike had no shortage of valuable things it could do on LW or Khan - why should it work on PB? (Practice in Agile methodology? Better done on high-traffic sites where measurements are more trustworthy.)

The final crushing statistic: PB had just 4 visitors who visited more than 10 times that month. Including me.

Comment author: JoshuaZ 04 September 2011 05:54:40PM 0 points [-]

Hmm, I think this is a good idea. When I make a prediction or comment on it I will add tag remarks. It is non-ideal hack but should help a little bit.